eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: In my app I use SQLAlchemy and mysql-connector-python. I would like to perform such query SELECT * FROM :table LIMIT 10 on my mysql database. However my code doesn't work table_name = "tmp1" QUERY = "SELECT * FROM :table LIMIT 10" conn = create_sql_connection() res = conn.execute( QUERY, {'table': table_name} ).fetchall() print(res)  I've read that you cannot use table name as a parameter and I should just use python string format. However I'm really scared that it's absolutely not safe against sql injection. How to solve it ? Is there any utility that would escape my table name variable ? Postgres has a solution - Passing table name as a parameter in psycopg2 - Do you know how to solve it while using mysql ?

Answer: you can pass the user provided string to a Table statement and build queries from this: (here I assume you get the user data from a post request json) table_name_string = request.get_json().get('table') selected_table = db.Table(table_name_string, metadata, autoload=True) query = selected_table.select()  you can also go on an use this query as a raw sql string at this point if you really want to query_string = str(query)  some input validation or restrictions what tables are valid is definitely recommended, but it is sql injection safe, since the table must exist in the metadata, or it will throw an Exception (sqlalchemy.exc.NoSuchTableError:)
Question: I am trying to insert data from a data frame into a Hive table. I have been able to do so successfully using df.write.insertInto("db1.table1", overwrite = True). I am just a little confused about the overwrite = True part -- I tried running it multiple times and it seemed to append, not overwrite. There wasn't too much in the docs, but when should I set overwrite to False vs. True?

Answer: df.insertInto works only if table already exists in hive. df.write.insertInto("db.table1",overwrite=False) will append the data to the existing hive table. df.write.insertInto("db.table1",overwrite=True) will overwrite the data in hive table. Example: df.show() #+----+---+ #|name| id| #+----+---+ #| a| 1| #| b| 2| #+----+---+ #save the table to hive df.write.saveAsTable("default.table1") #from hive #hive> select * from table1; #OK #a 1 #b 2 df.write.insertInto("moch.table1",overwrite=True) #from hive #hive> select * from table1; #OK #a 1 #b 2 #appending data to hive df.write.insertInto("moch.table1",overwrite=False) #from hive #hive> select * from table1; #OK #a 1 #b 2 #a 1 #b 2
Question: Looking at Python-Dev and StackOverflow, Python's ternary operator equivalent is: a if condition else b  Looking at PEP-572 and StackOverflow, I understand what Walrus operator is: :=  Now I'm trying to to combine the "walrus operator's assignment" and "ternary operator's conditional check" into a single statement, something like: other_func(a) if (a := some_func(some_input)) else b  For example, please consider the below snippet: do_something(list_of_roles) if list_of_roles := get_role_list(username) else "Role list is [] empty"  I'm failing to wrap my mind around the syntax. Having tried various combinations, every time the interpreter throws SyntaxError: invalid syntax. My python version is 3.8.3. My question is What is the correct syntax to embed walrus operator within ternary operator?

Answer: Syntactically, you are just missing a pair of parenthesis. do_something(list_of_roles) if (list_of_roles := get_role_list(username)) else "Role list is [] empty"   If you look at the grammar, := is defined as part of a high-level namedexpr_test construct: namedexpr_test: test [':=' test]  while a conditional expression is a kind of test: test: or_test ['if' or_test 'else' test] | lambdef  This means that := cannot be used in a conditional expression unless it occurs inside a nested expression.
Question: I am using cloud storage with App Engine Flex. Out of the blue i start getting this error message after deploy succeeds The error is happening from these lines in my flask app. from google.cloud import storage, datastore client = storage.Client()  File "/home/vmagent/app/main.py", line 104, in _load_db client = storage.Client() File "/env/lib/python3.6/site-packages/google/cloud/storage/client.py", line 110, in __init__ project=project, credentials=credentials, _http=_http File "/env/lib/python3.6/site-packages/google/cloud/client.py", line 250, in __init__ Client.__init__(self, credentials=credentials, client_options=client_options, _http=_http) File "/env/lib/python3.6/site-packages/google/cloud/client.py", line 143, in __init__ scopes = client_options.scopes or self.SCOPE AttributeError: 'ClientOptions' object has no attribute 'scopes'  This is something to do with breaking upgrades made to grpcio and google-api-core and google-cloud-storage packages based on numerous SO threads. However, I cant figure out where this is happening. My requirements.txt is as follows: setuptools>=40.3 grpcio<=1.27.2 google-api-core<1.17.0 Flask gevent>=0.13 gunicorn>=19.7.1 numpy>=1.18.0 numpy-financial scipy>=1.4 pvlib>=0.7 google-cloud-storage==1.28.0 google-cloud-datastore==1.12.0 google-cloud-pubsub pandas==1.0.5  my app.yaml is as follows: service: app-preprod runtime: custom env: flex entrypoint: gunicorn -t 600 -c gunicorn.conf.py -b :$PORT main:app runtime_config: python_version: 3.6 manual_scaling: instances: 1 resources: cpu: 1 memory_gb: 4 beta_settings: cloud_sql_instances: xxxx:europe-west6:component-cost endpoints_api_service: name: apipreprod-dot-xxxx.appspot.com rollout_strategy: managed  Looking at the release histories, some new versions of google-cloud-storage etc were released a few days ago, but i have tried to maintain the same older version number. The ridiculous thing is that with these exact same requirements.txt, i have an identical prod app engine that is working fine --- but that i had not redeployed for a week. Obviously, no problems at all with exactly the same versions of storage and datastore to run the client from my local machine. --EDIT-- Apparently according to https://github.com/googleapis/google-cloud-python/issues/10471 i should just add google-cloud-core==1.3.0 to requirements.txt This seems a workaround --- any better permanent way of ensuring this break doesnt catch me unawares?

Answer: This is due to https://github.com/googleapis/google-cloud-python/issues/10471. I'd recommend upgrading google-cloud-core and google-api-core to the latest versions with the bugfix.
Question: Consider the following custom layer code from a TensorFlow tutorial: class MyDenseLayer(tf.keras.layers.Layer): def __init__(self, num_outputs): super(MyDenseLayer, self).__init__() self.num_outputs = num_outputs def build(self, input_shape): self.kernel = self.add_weight("kernel", shape=[int(input_shape[-1]), self.num_outputs]) def call(self, input): return tf.matmul(input, self.kernel)  How do I apply any pre-defined regularization (say tf.keras.regularizers.L1) or custom regularization on the parameters of the custom layer?

Answer: The add_weight method takes a regularizer argument which you can use to apply regularization on the weight. For example: self.kernel = self.add_weight("kernel", shape=[int(input_shape[-1]), self.num_outputs], regularizer=tf.keras.regularizers.l1_l2())  Alternatively, to have more control like other built-in layers, you can modify the definition of custom layer and add a kernel_regularizer argument to __init__ method: from tensorflow.keras import regularizers class MyDenseLayer(tf.keras.layers.Layer): def __init__(self, num_outputs, kernel_regularizer=None): super(MyDenseLayer, self).__init__() self.num_outputs = num_outputs self.kernel_regularizer = regularizers.get(kernel_regularizer) def build(self, input_shape): self.kernel = self.add_weight("kernel", shape=[int(input_shape[-1]), self.num_outputs], regularizer=self.kernel_regularizer)  With that you can even pass a string like 'l1' or 'l2' to kernel_regularizer argument when constructing the layer, and it would be resolved properly.
Question:  Lambda execution failed with status 200 due to customer function error: Object of type 'Decimal' is not JSON serializable  I went through all the existing solutions in the following link but nothing worked for me. What am I doing wrong?: Python JSON serialize a Decimal object import json import boto3 import decimal client = boto3.resource('dynamodb') table = client.Table('table') def lambda_handler(event, context): method = event["httpMethod"] print(event) if method=="POST": return POST(event) elif method=="DELETE": return DELETE(event) elif method=="GET": return GET(event) #the respons format def send_respons(responseBody, statusCode): response = { "statusCode": statusCode, "headers": { "my_header": "my_value" }, "body": json.dumps(responseBody), "isBase64Encoded": 'false' } return response def GET(event): tab = table.scan()['Items'] ids = [] for item in tab: ids.append({"id":item["id"], "decimalOBJ":decimal.Decimal(item["decimalOBJ"]})) return send_respons(ids, 201) 

Answer: It seems you have two options:  Probably easiest, you can serialize the int/float value of a Decimal object:  """ assume d is your decimal object """ serializable_d = int(d) # or float(d) d_json = json.dumps(d)   You can add simplejson to your requirements.txt, which now has support for serializing Decimals. It's a drop-in replacement for the included json module.   import simplejson as json # instead of import json   The rest of your code will work the same. If you need further assistance, kindly leave a comment.
Question: first of all sorry for my English. I have an Azure Function Linux Consuption Plan using Python and I need to generate an html, transform to pdf using wkhtmltopdf and send it by email.  #generate temporally pdf config = pdfkit.configuration(wkhtmltopdf="binary/wkhtmltopdf") pdfkit.from_string(pdf_content, 'report.pdf',configuration=config, options={}) #read pdf and transform to Bytes with open('report.pdf', 'rb') as f: data = f.read() #encode bytes encoded = base64.b64encode(data).decode() #Send Email EmailSendData.sendEmail(html_content,encoded,spanish_month)  Code is running ok in my local development but when I deploy the function and execute the code I am getting an error saying: Result: Failure Exception: OSError: wkhtmltopdf reported an error: Loading pages (1/6) [> ] 0% [======> ] 10% [==============================> ] 50% [============================================================] 100% QPainter::begin(): Returned false Error: Unable to write to destination  I think that error is reported because for any reason write permission is not available. Can you help me to solve this problem? Thanks in advance.

Answer: The tempfile.gettempdir() method returns a temporary folder, which on Linux is /tmp. Your application can use this directory to store temporary files generated and used by your functions during execution. So use /tmp/report.pdf as the file directory to save temporary file. with open('/tmp/report.pdf', 'rb') as f: data = f.read()  For more details, you could refer to this article.
Question: In fastAPI one can simply write a security dependency at the router level and secure an entire part of the URLs. router.include_router( my_router, prefix="/mypath", dependencies=[Depends(auth.oauth2_scheme)] )  This avoids repeating a lot of code. The only problem is that I would like to protect a part of URLs with a router level dependency that checks the validity of the user token and retrieve the user id for that token. The only way I found, is to add another dependency to all the functions, but this leads to repeating the code that I just saved. Long story short, is there a way to add the dependency at the router level, retrieve and return the user id, and pass the returned value to the handling function? Something like router.py router.include_router( my_router, prefix="/mypath", dependencies=[user_id = Depends(auth.oauth2_scheme)] )  my_router.py my_router = APIRouter() @my_router.get("/my_path") async def get_my_path(**kwargs): user_id = kwargs["user_id"] # Do stuff with the user_id return {} 

Answer: Once the user is authenticated in the dependency function add the user_id to request.state, then on your route you can access it from the request object. async def oauth2_scheme(request: Request): request.state.user_id = "foo" my_router = APIRouter() @my_router .get("/") async def hello(request: Request): print(request.state.user_id) app.include_router( my_router, dependencies=[Depends(oauth2_scheme)] )
Question: New to Python and come from a statically typed language background. I want type hints for https://lxml.de just for ease of development (mypy flagging issues and suggesting methods would be nice!) To my knowledge, this is a python 2.0 module and doesn’t have types. Currently I’ve used https://mypy.readthedocs.io/en/stable/stubgen.html to create stub type definitions and filling in “any”-types I’m using with more information, but it’s really hacky. Are there any safer ways to get type hints?

Answer: There is an official stubs package for lxml now called lxml-stubs: $ pip install lxml-stubs  Note, however, that the stubs are still in development and are not 100% complete yet (although very much usable from my experience). These stubs were once part of typeshed, then curated by Jelle Zijlstra after removal and now are developed as part of the lxml project. If you want the development version of the stubs, install via $ pip install git+https://github.com/lxml/lxml-stubs.git  (the project's readme installation command is missing the git+ prefix in URL's scheme and won't work).
Question: The pandas.plot.kde() function is handy for plotting the estimated density function of a continuous random variable. It will take data x as input, and display the probabilities p(x) of the binned input as its output. How can I extract the values of probabilities it computes? Instead of just plotting the probabilities of bandwidthed samples, I would like an array or pandas series that contains the probability values it internally computed. If this can't be done with pandas kde, let me know of any equivalent in scipy or other

Answer: there are several ways to do that. You can either compute it yourself or get it from the plot.  As pointed out in the comment by @RichieV following this post, you can extract the data from the plot using  data.plot.kde().get_lines()[0].get_xydata()   Use seaborn and then the same as in 1):  You can use seaborn to estimate the kernel density and then matplotlib to extract the values (as in this post). You can either use distplot or kdeplot: import seaborn as sns # kde plot x,y = sns.kdeplot(data).get_lines()[0].get_data() # distplot x,y = sns.distplot(data, hist=False).get_lines()[0].get_data()   You can use the underlying methods of scipy.stats.gaussian_kde to estimate the kernel density which is used by pandas:  import scipy.stats density = scipy.stats.gaussian_kde(data)  and then you can use this to evaluate it on a set of points: x = np.linspace(0,80,200) y = density(xs)
Question: I am trying to create a plot similar to the one below taken from this paper, essentially a 3d plot with two distinct y-axes. Following guidance in this blog, I created a minimal example. Modules from mpl_toolkits import mplot3d import numpy as np %matplotlib inline import numpy as np import matplotlib.pyplot as plt  Create some data def f(x, y): return np.sin(np.sqrt(x ** 2 + y ** 2)) x = np.linspace(-6, 6, 30) y = np.linspace(-6, 6, 30) X, Y = np.meshgrid(x, y) Z = f(X, Y) Z2 = Z*100+100  Plotting This creates a nice 3d plot, but obviously with only one y-axis. I could not find any advice online on how to get there for python, albeit some for matlab. fig = plt.figure() ax = plt.axes(projection='3d') ax.plot_surface(X, Y, Z2, rstride=1, cstride=1, cmap='viridis', edgecolor='none') ax.set_title('surface'); ax.set_xlabel('x') ax.set_ylabel('y') ax.set_zlabel('z');  Code gives:  Reference graph: 

Answer: This isn't easy. One possible workaround approach is as follows:  Based on your shared reference figure, I think you mean actually that you are looking to implement a second z-axis, not y-axis. The axes object for the 3d plot remains singular and shared (out of necessity / apparent matplotlib 3d plot limitations), but the data is plotted as though on the same continuous scale of values yet the axis ticks and labels are custom overwritten to reflect different scales of values.  E.g., import numpy as np import matplotlib.pyplot as plt def f(x, y): return np.sin(np.sqrt(x ** 2 + y ** 2)) def g(x, y): return -np.cos(np.sqrt(x ** 2 + y ** 2)) x = np.linspace(-6, 6, 30) y = np.linspace(-6, 6, 30) X, Y = np.meshgrid(x, y) Z = f(X, Y) Z_new = g(X, Y) offset = 5 Z_new_offset = Z_new + Z.max() + offset fig = plt.figure(figsize=(16, 12)) ax = fig.add_subplot(111, projection="3d") surf1 = ax.plot_surface( X, Y, Z, rstride=1, cstride=1, cmap="viridis", edgecolor="none", alpha=0.7 ) surf2 = ax.plot_surface( X, Y, Z_new_offset, rstride=1, cstride=1, cmap="plasma", edgecolor="none", alpha=0.7, ) z_ticks_original = np.linspace(Z.min(), Z.max(), 5) # Add custom tick labels and tick marks for the new plot on the left z_ticks_new = np.linspace(Z_new_offset.min(), Z_new_offset.max(), 5) for z_tick in z_ticks_new: ax.text( X.min() - 0.5, Y.min() - 2.5, z_tick + 0.25, f"{z_tick - (offset+1):.1f}", color="k", verticalalignment="center", ) ax.plot( [X.min() - 0.5, X.min()], [Y.min() - 0.5, Y.min()], [z_tick, z_tick], color="k", ) ax.set_zticks(np.block([z_ticks_original, z_ticks_new])) fig.canvas.draw() labels = [] for lab, tick in zip(ax.get_zticklabels(), ax.get_zticks()): if float(tick) >= 1.0: lab.set_text("") labels += [lab] ax.set_zticklabels(labels) # Draw the left Z-axis line ax.plot( [X.min() - 0.5] * 2, [Y.min() - 0.5] * 2, [z_ticks_new.min(), z_ticks_new.max()], color="k", ) ax.set_xlabel("X") ax.set_ylabel("Y") ax.set_zlabel("Z") cbar1 = fig.colorbar( surf1, ax=ax, pad=-0.075, orientation="vertical", shrink=0.5 ) cbar1.set_label("Z Values (primary z-axis)") cbar2 = fig.colorbar( surf2, ax=ax, pad=0.12, orientation="vertical", shrink=0.5, ticks=z_ticks_original, ) cbar2.set_label("Z Values (secondary z-axis)") plt.show()  produces:
Question: I'm using the PyPDF2 library for extracting text, images, page width and heights, annotations, and other attributes from pdf documents. However, the library has many bugs and issues and seems not to be maintained for a long time already. (edit: PyPDF2 is maintained again)  Is there a more vivid fork that is being maintained and developed? Is there a good alternative?  From what I know, reportlab is more suitable for creating brand new pdf's (or maybe I'm just not experienced enough with reportlab).

Answer: Update: pypdf (pypi) is maintained again - and I am the maintainer (of pypdf and PyPDF2) :-) I've just released a new version with several bugfixes. Looking at the top PyPI packages, PyPDF2 is also the most used one (and pypdf==3.1.0 is almost the same as PyPDF2==3.0.0, the community just needs a bit of time to switch to pypdf)  Three potential alternatives which are maintained (just like pypdf):  pymupdf: uses mupdf (only free for open source due to mypdf license) pikepdf: Uses qpdf pdfminer.six: A pure Python project.  I would not use:  PyPDF2: I am the maintainer. In December 2022, I made the last release. I want the community to switch to pypdf (where I'm also the maintainer) PyPDF3 (pypi): Has less activity and probably less features than PyPDF2. PyPDF4 (pypi): Last release on PyPI in 2018
Question: I want to send a DM to the user, who invited/added the bot to his server.  I noticed that it's displayed in the audit log. Can I fetch that and get the user or is there a easier way to achieve that? Example: bot = commands.Bot() @bot.event async def on_guild(guild, inviter): await inviter.send("Thanks for adding the bot to your server!") 

Answer: With discord.py 2.0 you can get the BotIntegration of a server and with that the user who invited the bot. Example from discord.ext import commands bot = commands.Bot() @bot.event async def on_guild_join(guild): # get all server integrations integrations = await guild.integrations() for integration in integrations: if isinstance(integration, discord.BotIntegration): if integration.application.user.name == bot.user.name: bot_inviter = integration.user # returns a discord.User object # send message to the inviter to say thank you await bot_inviter.send("Thank you for inviting my bot!!") break  Note: guild.integrations() requires the Manage Server (manage_guild) permission.  References:  Guild.integrations discord.BotIntegration
Question: I'm trying to install python 3.9 in a conda enviroment. I tried creating a new conda env using the following command, conda create --name myenv python=3.9  But I got an error saying package not found because python 3.9 is not yet released So, I manually created a folder in envs folder and tried to list all envs. But I couldn't get the manually created new environment. So, how do I install python 3.9 in a conda env with all functionalities like pip working?

Answer: To create python 3.11 conda environment use the following command conda create -n py311 python=3.11  py311 - environment name  Update 3 To create python 3.10 conda environment use the following command conda create -n py310 python=3.10  py310 - environment name  Update 2 You can now directly create python 3.9 environment using the following command conda create -n py39 python=3.9  py39 - environment name  Update 1 Python 3.9 is now available in conda-forge. To download the tar file - https://anaconda.org/conda-forge/python/3.9.0/download/linux-64/python-3.9.0-h852b56e_0_cpython.tar.bz2 Anaconda Page - https://anaconda.org/conda-forge/python  As pointed out in the comments, python 3.9 is not yet there on any channels. So, it cannot be install yet via conda. Instead, you can download the python 3.9 executable and install it. Once the installation is done, a new executable will be created for python 3.9 and pip 3.9 will be created. Python: python3.7 python3.7-config python3.7m python3.7m-config python3.9 python3.9-config  pip pip pip3 pip3.7 pip3.8 pip3.9 pipreqs  In order to install ipython for python 3.9, pip3.9 install ipython
Question: I am running a fast api and when i was developing i had the following piece of code in my app.py file code in app.py: import uvicorn if __name__=="__main__": uvicorn.run("app.app:app",host='0.0.0.0', port=4557, reload=True, debug=True, workers=3)  so i was about to run CMD ["python3","app.py"] in my Dockerfile. on the fastapi example they did something like this : CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "80"]  I want to know what is the difference between these two methods as i think both of them will work.

Answer: Update (on 2022-12-31) As an update from @Marcelo Trylesinski, from uvicorn v 0.19.0, the --debug flag was removed (Ref #1640).  No, there is no difference. The commadline run method (uvicorn app.main:app) and executing the app.py using python command (python app.py) are the same. Both methods are calling the uvicorn.main.run(...) function under the hood. In other words, the uvicorn command is a shortcut to the uvicorn.run(...) function. So, in your case the function call uvicorn.run("app.app:app",host='0.0.0.0', port=4557, reload=True, debug=True, workers=3)  can be done by uvicorn commandline as, uvicorn app.app:app --host 0.0.0.0 --port 4557 --reload --debug --workers 3   Side Notes The --debug option is hidden from the command line options help page, but it can be found in the source code. Thus, it feels running the app using uvicorn command can be considered as a production thing.