eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have the following code where I initialize a consumer listening to a queue. consumer = MyConsumer() consumer.declare_queue(queue_name="my-jobs") consumer.declare_exchange(exchange_name="my-jobs") consumer.bind_queue( exchange_name="my-jobs", queue_name="my-jobs", routing_key="jobs" ) consumer.consume_messages(queue="my-jobs", callback=consumer.consume)  The problem is that the consume method is defined as follows: async def consume(self, channel, method, properties, body):  Inside the consume method, we need to await async functions, but this produces an error "coroutine is not awaited" for the consume function. Is there a way to use async function as a callback in pika?

Answer: I annotated my callback with @sync where sync is: def sync(f): @functools.wraps(f) def wrapper(*args, **kwargs): return asyncio.get_event_loop().run_until_complete(f(*args, **kwargs)) return wrapper  (found it here for celery, but it worked with pika too)
Question: I can't figure how i could type annotate my sqlalchemy models code, what kind of type should i use for my model fields. class Email(Model): __tablename__ = 'emails' name: Column[str] = Column(String, nullable=False) sender: Column[str] = Column( String, default=default_sender, nullable=False ) subject: Column[str] = Column(String, nullable=False) html: Column[str] = Column(String, nullable=False) template_id: Column[UUID] = Column( postgresql.UUID(as_uuid=True), ForeignKey('templates.id'), index=True, )  Column type as well as Mapped satisfies the type linter of my editor, but it doesn't look like the right one as opposed to simple types like str/int/uuid Should i use Optional[Column[str]] or Column[Optional[str]]? What to use for relationship? class Foo(Model): ... bar: Bar = relationship( 'Bar', order_by="desc(Bar.created_at)", lazy='dynamic', )  The result of accessing the relationship varies depending on the attribute lazy.

Answer: For SQLAlchemy 2.0 it would be something like: import uuid from sqlalchemy.orm import Mapped, mapped_column from sqlalchemy.dialects.postgresql import UUID class Email(Model): __tablename__ = 'emails' name: Mapped[str] = mapped_column(String, nullable=False) sender: Mapped[str] = mapped_column( String, default=default_sender, nullable=False ) subject: Mapped[str] = mapped_column(String, nullable=False) html: Mapped[str] = mapped_column(String, nullable=False) template_id: Mapped[uuid.UUID] = mapped_column( UUID(as_uuid=True), ForeignKey('templates.id'), index=True, )
Question: I found NonDynamicallyQuantizableLinear while reading torch.nn.modules.activation.py in MultiheadAttention class And I have a few questions about it  What is the difference between Linear and NonDynamicallyQuantizableLinear? Why is NonDynamicallyQuantizableLinear used for MultiheadAttention? 

Answer: It is introduced as a temporary cleanup measure to fail immediately when attempting directly quantize torch.nn.MultiheadAttention. This improves upon the baseline behavior of failing silently. See https://github.com/pytorch/pytorch/issues/58969
Question: How can I set a custom sort order for the API methods in FastAPI Swagger autodocs? This question shows how to do it in Java. My previous question asked how to sort by "Method", which is a supported sorting method. I would really like to take this a step further, so that I can determine which order the methods appear. Right now DELETE appears at the top, but I want API methods to be in the order: GET, POST, PUT, DELETE. I know it is possible to implement a custom sort in JavaScript and give that function to operationsSorter, but you can't include it from the swagger_ui_parameters property that is available in the Python bindings. Is there some way to accomplish this in Python? from fastapi import FastAPI app = FastAPI(swagger_ui_parameters={"operationsSorter": "method"}) @app.get("/") def list_all_components(): pass @app.get("/{component_id}") def get_component(component_id: int): pass @app.post("/") def create_component(): pass @app.put("/{component_id}") def update_component(component_id: int): pass @app.delete("/{component_id}") def delete_component(component_id: int): pass  

Answer: You can use tags to group your endpoints. To do that, pass the parameter tags with a list of str (commonly just one str) to your endpoints. Use the same tag name for endpoints that use the same HTTP method, so that you can group your endpoints that way. For example, use Get as the tag name for GET operations (Note: Get is only an example here, you can use any tag name you wish). Doing just the above would most likely define the endpoints in the order you wish (i.e., GET, POST, PUT, DELETE). However, to ensure this—or, even in case you would like to define a different order for your methods/tags—you could add metadata for the different tags used to group your endpoints. You could do that using the parameter openapi_tags, which takes a list containing one dictionary for each tag. Each dictionary should at least contain the name, which should be the same tag name used in the tags parameter. The order of each tag metadata dictionary also defines the order shown in the docs UI. Working Example from fastapi import FastAPI tags_metadata = [ {"name": "Get"}, {"name": "Post"}, {"name": "Put"}, {"name": "Delete"} ] app = FastAPI(openapi_tags=tags_metadata) @app.get("/", tags=["Get"]) def list_all_components(): pass @app.get("/{component_id}", tags=["Get"]) def get_component(component_id: int): pass @app.post("/", tags=["Post"]) def create_component(): pass @app.put("/{component_id}", tags=["Put"]) def update_component(component_id: int): pass @app.delete("/{component_id}", tags=["Delete"]) def delete_component(component_id: int): pass
Question: I am executing a notebook on my laptop and I get the following error. XXX lineno: 17, opcode: 120 --------------------------------------------------------------------------- SystemError Traceback (most recent call last) Input In [3], in <cell line: 3>() 1 gym = Gym(0, 0, 0, 0).from_dill(BACKUP) 2 ticker = gym.api.returnTicker() ----> 3 gym.wallet = gym.get_wallet() 4 plot_donut_gym_wallet(gym) 5 plot_donut_gym_wallet_makers(gym) File <ipython-input-3-1c4842a503bf>:17, in get_wallet(self) SystemError: unknown opcode  As you can see, the error happens during a function call. The function itself is not the problem, if I define and run ithe function inside a cell it simply works. But importing the function from its own module leads to this error. I have looked around for hints, all the forum I have read are pointing to some problem with having multiple Python installations. However, I tried using new environments, both using venv and conda and I get the same error. The same code on other machines works, so it appear to be something related to my particular installation, but I can't figure out how to fix it. I tried reinstalling conda, making new envs, upgrading python. Iteted this with Python 3.8, 3.9 and 3.10, I always get the same error. Any help is very welcome. Since the function is from a class previously serialized using dill, this dill-related issue may be relevant https://github.com/uqfoundation/dill/issues/438

Answer: Code objects (from functions and class methods) serialized with dill are not guaranteed to work across different Python versions, because the list of valid opcodes change from version to version. In cases where there is an opcode unknown to the interpreter deserializing ("unpickling") the code object, the functions and classes are successfully recreated, but fail to execute as in your example. The code in your example may have been created in Python 2, or Python 3.11 beta... It's impossible to know just by looking. You should ask whoever created the pickle file.  Edit: If you don't have the source code of the functions and methods, you may be able to reconstruct their source using a decompiler like decompyle3 (or uncompyle6 for other Python versions) and re-generate these faulty code objects with another Python version. An example in Python3.7: >>> def f(x, *, verbose=False): ... y = x*x ... if verbose: ... print(y) ... return y ... >>> import inspect >>> f_sign = inspect.signature(f) <Signature (x, *, verbose=False)> >>> >>> import os >>> import decompyle3 >>> with open(os.devnull, 'w') as null: ... f_code = decompyle3.main.decompile(f.__code__, out=null) ... >>> f_code <decompyle3.semantics.pysource.SourceWalker object at 0x7f8932f14990> >>> >>> from textwrap import indent >>> print("def {}{}:\n{}".format(f.__name__, f_sign, indent(f_code.text, " "))) def f(x, *, verbose=False): y = x * x if verbose: print('x**2 =', y) return y
Question: This question has been bothering me for two weeks and I've searched online and asked people but couldn't get an answer. Python by default build the library libpythonMAJOR.MINOR.a and statically links it into the interpreter. Also it has an --enable-shared flag, which will build a share library libpythonMAJOR.MINOR.so.1.0, and dynamically link it to the interpreter. Based on my poor CS knowledge, the first thought came into my mind when I saw "shared library", is that, "the shared bulid one must save a lot of memory compared to the static build one!". Then I had this assumption: # share build 34K Jun 29 11:32 python3.9 21M Jun 29 11:32 libpython3.9.so.1.0  10 shared python processes, mem usage = 0.034M * 10 + 21M ≈ 21M # static build 22M Jun 27 23:45 python3.9  10 static python processes, mem usage = 10*22M = 220M shared python wins! Later I ran a toy test on my machine and found that's wrong. test.py import time i = 0 while i < 20: time.sleep(1) i += 1 print('done')  mem_test.sh #! /bin/bash for i in {1..1000} do ./python3.9 test.py & done  For share python to run I set export LD_LIBRARY_PATH=/home/tian/py3.9.13_share/lib . I ran mem_test.sh separately (one by one) with 2 pythons and simply monitored the total mem usage via htop in another console. It turns out that both eat almost the same amount of memory. Later on people taught me there's something call "paging on demand": Is an entire static program loaded into memory when launched? How does an executable get loaded into RAM, does the whole file get loaded into RAM even when the whole file won't be needed, or does it get loaded in "chunks"? so my previous calculation of static python mem usage is completely wrong. Now I am confused. Shared build python doesn't use less memory via a share library runtime? Question: What's the benefit of a shared build python vs a static build python? Or the shared built python indeed save some memory by the mechanism of using a share library, but my test is too trival to reveal? P.S. Checking some python official Dockerfiles, e.g. this one you would see they all set --enable-shared. Also there's related issue on pyenv https://github.com/pyenv/pyenv/issues/2294 , it seems that neither they figure that out.

Answer: It turns out to be that others are talking about the scenario "Embedding Python in Another Application" (https://docs.python.org/3/extending/embedding.html). If that's the case, then "saving disk space" and other mentioned reasons make sense. Because embedding python in another application, either you need to statically link libpythonMAJOR.MINOR.a or dynamically link libpythonMAJOR.MINOR.so.1.0. So my current conclusion is that whether python is shared built or statically built only affects the "Embedding Python in Another Application" scenario. For normal use cases, e.g. running the python interpeter, it doesn't make much diffferences. Update: Disk usage comparsion, see comments in makfile: https://stackoverflow.com/a/73099136/5983841
Question: I have two GeoDataFrames, left and right, with many polygons in them. Now I am trying to find the total intersection area of each polygon in left, with all polygons in right. I've managed to get the indices of the intersecting polygons in right for each polygon in left using gpd.sjoin, so I compute the intersection area using: area = left.iloc[i].geometry.intersection(right.iloc[idx].geometry).area  Where i and idx are the indices of the intersecting polygons in the two GDFs (let's assume the left poly only intersects with 1 poly in right). The problem is that the area value I get does not seem correct in any way, and I don't know what units it has. The CRS of both GeoDataFrames is EPSG:4326 so the standard WSG84 projection, and the polygon coordinates are defined in degrees latitude and longitude. Does anyone know what units the computed area then has? Or does this not work and do I need to convert them to a different projection before computing the area? Thanks for the help!

Answer: I fixed it by using the EPSG:6933 projection instead, which is an area preserving map projection and returns the area in square metres (EPSG:4326 does not preserve areas, so is not suitable for area calculations). I could just change my GDF to this projection using gdf.to_crs(espg=6933)  And then compute the area in the same way as above.
Question: I'm new to ML and I've been working with an imbalanced data set where the count of negative samples is twice that of the positive samples. In-order to address these i set scikit-learn Random forest class_weight = 'balanced', which gave me an ROC-AUC score of 0.904 and the recall for class- 1 was 0.86, now when i tried to further improve the AUC Score by assigning weight, there wasn't any major difference with the results, i.e Class_weight = {0: 0.5, 1: 2.75}, assuming this would penalize for every wrong classification of 1 but it didn't seem to work as expected. randomForestClf = RandomForestClassifier(random_state = 42, class_weight = {0: 0.5, 1:2.75})  Tried different values but has no major impact as Recall of 1 remains the same or reduces (0.85) and auc value is quite insignificant (0.90122). It only seems to work when one of the label is set 0. Further tried to set the sample weights too. But that didn't seem to work either. # Sample weights class_weights = [0.5, 2] weights = np.ones(y_train.shape[0], dtype = 'float') for i, val in enumerate(y_train): weights[i] = class_weights[val]  Below is the reference to a similar question but the solutions provided didn't work for me. sklearn RandomForestClassifier's class_weights seems to have no effect Is there anything that i'm missing out? Thanks!

Answer: The reason is that you grow the trees out fully, which leads to every leaf node being pure. That will happen regardless of the class weights (though the structure of the tree leading up to those pure nodes will change). The predicted probabilities of each tree will be (almost) all 0 or 1, and so the overall probability estimates are just driven by disagreements between the trees. If you set e.g. max_depth=10 (or whatever tree complexity parameter you like), now many/most of the leaf nodes will not be pure. Setting larger positive-class weights will produce leaf values that are biased toward the positive class (but still aren't just 0 and 1), and so the probability estimates will be skewed higher across the board, leading to a higher recall (at the expense of precision, presumably). The ROC curve is relatively unaffected by class balance and the skewed-higher probabilities arising from the larger weights, and so shouldn't be heavily affected by changing weights, for a fixed max_depth.
Question: In the following code: a = [["2022"], ["2023"]] b = [["blue", "red"], ["green", "yellow"]] c = [["1", "2", "3"], ["4", "5", "6", "7"], ["8", "9", "10", "11"], ["12", "13"]]  I would like a function that outputs this, but for any number of variables: [ ["2022", "blue", "1"], ["2022", "blue", "2"], ["2022", "blue", "3"], ["2022", "red", "4"], ["2022", "red", "5"], ["2022", "red", "6"], ["2022", "red", "7"], ["2023", "green", "8"], ["2023", "green", "9"], ["2023", "green", "10"], ["2023", "green", "11"], ["2023", "yellow", "12"], ["2023", "yellow", "13"], ]  I have searched for a function to do this with itertools or zip, but haven't found anything yet. To clarify, my use case for this was to iterate through values of a nested/multi-level dropdown menu (the first dropdown returns options, and each option returns a different dropdown, and so on).

Answer: First, you join the first argument, to a list of lists with only one element each. Then for each sublist and its index i in the next argument, you pick the i-th list of the previous iteration res[i] and add to aux len(sublist) lists each of one is the res[i] with one item from sublist. from itertools import chain def f(*args): res = list(chain.from_iterable([[item] for item in l] for l in args[0])) for arg in args[1:]: aux = [] for i, sublist in enumerate(arg): aux += [res[i] + [opt] for opt in sublist] res = aux return res  In addition if you want to verify that the arguments passed to the function are correct, you can use this: def check(*args): size = sum(len(l) for l in args[0]) for arg in args[1:]: if len(arg) != size: return False size = sum(len(l) for l in arg) return True
Question: I am new to Python and writing a lambda function. I installed urllib3 using pip but still getting this following error.  I tried restarting vscode/ uninstall and reinstall but still getting the error. this is the result when I run pip show urllib3  what am i missing here?

Answer: Maybe there is more than one python environment on your machine, And the location where you installed the package is inconsistent with the python interpreter you are using now. You can use CTRL + SHIFT + P to open the command palette and search Python: Select Interpreter (or click on the interpreter version displayed in the lower right corner).  Select the environment interpreter where you have the urllib3 package installed.
Question: I am a beginner at machine learning. I try to use LSTM algorism but when I write from keras.models import Sequential it shows error as below: ImportError: dlopen(/Users/wangzifan/opt/anaconda3/lib/python3.9/site-packages/tensorflow/python/_pywrap_tfe.so, 2): Library not loaded: @rpath/_pywrap_tensorflow_internal.so Referenced from: /Users/wangzifan/opt/anaconda3/lib/python3.9/site-packages/tensorflow/python/_pywrap_tfe.so Reason: image not found  How can I fix this? Thank you so much! full error message: Traceback (most recent call last): File "/Users/wangzifan/Desktop/machine/LSTM.py", line 39, in <module> from keras.models import Sequential File "/Users/wangzifan/opt/anaconda3/lib/python3.9/site-packages/keras/__init__.py", line 21, in <module> from tensorflow.python import tf2 File "/Users/wangzifan/opt/anaconda3/lib/python3.9/site-packages/tensorflow/__init__.py", line 37, in <module> from tensorflow.python.tools import module_util as _module_util File "/Users/wangzifan/opt/anaconda3/lib/python3.9/site-packages/tensorflow/python/__init__.py", line 37, in <module> from tensorflow.python.eager import context File "/Users/wangzifan/opt/anaconda3/lib/python3.9/site-packages/tensorflow/python/eager/context.py", line 33, in <module> from tensorflow.python import pywrap_tfe File "/Users/wangzifan/opt/anaconda3/lib/python3.9/site-packages/tensorflow/python/pywrap_tfe.py", line 25, in <module> from tensorflow.python._pywrap_tfe import * ImportError: dlopen(/Users/wangzifan/opt/anaconda3/lib/python3.9/site-packages/tensorflow/python/_pywrap_tfe.so, 2): Library not loaded: @rpath/_pywrap_tensorflow_internal.so Referenced from: /Users/wangzifan/opt/anaconda3/lib/python3.9/site-packages/tensorflow/python/_pywrap_tfe.so Reason: image not found 

Answer: Problem solved. install tensorflow again with sudo pip3 install tensorflow  and change the import to from tensorflow.python.keras.models import Sequential
Question: So I'm trying to do a bar plot on a data where x is the username (string ) and each x is long enough to overlap each other, so I have to rotate the x label. No problem there. However, when exporting the plot results, the x label on the exported image is cropped. I tried using plt.tight_layout() and worked, but it change the look of the plot. The code is similar to this  from matplotlib import pyplot as plt x= ['abc', 'ronaldo', 'melon_killer_123456'] y= [1, 2, 3] plt.bar(x, y) plt.xticks(rotation = 90) plt.savefig('a.png') plt.show()  Exported image:  I want it to look like this (Got this by using jupyter notebook and manually save output image):  So how to do that?

Answer: You can play around with the rcParams size settings and the plt.subplots_adjust settings until you get your desired image. import matplotlib.pyplot as plt x= ['abc', 'ronaldo', 'melon_killer_123456'] y= [1, 2, 3] plt.rcParams["figure.figsize"] = (5,10) plt.bar(x, y) plt.xticks(rotation = 90) plt.subplots_adjust(top=0.925, bottom=0.20, left=0.07, right=0.90, hspace=0.01, wspace=0.01) plt.savefig('a.png') plt.show()
Question: I am working on a Linux environment and have created my virtual environment using the pyenv tool. I have set the local virtual environment in my working folder the the one I want with pyenv from command line like this for example : pyenv local my_venv_name which in my case my_venv_name=3.9.9 When I opened VSCode in that folder the integrated terminal is indeed opening that virtual environment BUT the code in the VSCode seems to not be able to find some installed dependencies that I can see that I have installed in that particular environment. I can do pip freeze in the terminal and I see the packages but VSCode does not see them. How can I set the correct virtual environment to the VSCode editor?

Answer: Ok so you have to select the correct python interpreter because pyenv could be using multiple python version in different environments. I found two ways to change it:  Open the command palette either from the gear icon bottom left corner or by typing Ctrl + Shift + P. Then type select python interpreter and select the one that corresponds to the virtual environment you want  You can find the same option by simply clicking and selecting the interpreter from the bottom right corner of the VSCode as shown in the image below:
Question: Python includes a simple to use profiler: >> import cProfile >> import re >> cProfile.run('re.compile("foo|bar")') 197 function calls (192 primitive calls) in 0.002 seconds Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function) 1 0.000 0.000 0.001 0.001 <string>:1(<module>) 1 0.000 0.000 0.001 0.001 re.py:212(compile) ...  How can do the exact same thing, but sorted by tottime instead of standardname?

Answer: Use the sort=... argument of cProfile.run: >>> import cProfile >>> import time >>> cProfile.run('time.sleep(1); time.monotonic()', sort='tottime') Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) 1 1.001 1.001 1.001 1.001 {built-in method time.sleep} 1 0.000 0.000 1.001 1.001 {built-in method builtins.exec} 1 0.000 0.000 1.001 1.001 <string>:1(<module>) 1 0.000 0.000 0.000 0.000 {built-in method time.monotonic} 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
Question: I have the following Pandas dataframe df that looks as follows: import pandas as pd df = pd.DataFrame({'state' : ['NY', 'CA', 'FL', 'NJ', 'TX', 'CT', 'MA', 'WA', 'IL', 'GA'], 'user_id' : [10000, 3200, 1600, 1200, 800, 600, 400, 350, 270, 260] }) state user_id 0 NY 10000 1 CA 3200 2 FL 1600 3 NJ 1200 4 TX 800 5 CT 600 6 MA 400 7 WA 350 8 IL 270 9 GA 260  I'd like to be able to create a Plotly choropleth that includes data labels over each of the states. To do so, I use add_scattergeo: fig = px.choropleth(df, locations = 'state', locationmode = "USA-states", scope = "usa", color = 'user_id', color_continuous_scale = "blues", ) fig.add_scattergeo( locations = df['state'], text = df['user_id'], mode = 'text', ) fig.show()   But, using add_scattergeo does not apply the desired labels. What's the best way to add data labels to a Plotly choropleth? Thanks!

Answer: You need to also add locationmode="USA-states" to add_scattergeo: fig = px.choropleth( df, locations='state', locationmode="USA-states", scope="usa", color='user_id', color_continuous_scale="blues", ) fig.add_scattergeo( locations=df['state'], locationmode="USA-states", text=df['user_id'], mode='text', )  Output: