eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I developed some API endpoints using FastAPI. These endpoints are allowed to run BackgroundTasks. Unfortunately, I do not know how to handle unpredictable issues from theses tasks. An example of my API is shown below: # main.py from fastapi import FastAPI import uvicorn app = FastAPI() def test_func(a, b): raise ... @app.post("/test", status_code=201) async def test(request: Request, background_task: BackgroundTasks): background_task.add_task(test_func, a, b) return { "message": "The test task was successfully sent.", } if __name__ == "__main__": uvicorn.run( app=app, host="0.0.0.0", port=8000 ) # python3 main.py to run # fastapi == 0.78.0 # uvicorn == 0.16.0  Can you help me to handle any type of exception from such a background task? Should I add any exception_middleware from Starlette, in order to achieve this?

Answer:  Can you help me to handle any type of exception from such a background task?  Background tasks, as the name suggests, are tasks that are going to run in the background after returning a response. Hence, you can't raise an Exception and expect the client to receive some kind of response. If you are just looking to catch any Exception occuring inside the background task, you can simply use the try-except block to catch any Exception and handle it as desired. For example: def test_func(a, b): try: # some background task logic here... raise <some_exception> except Exception as e: print('Something went wrong') # use `print(e.detail)` to print out the Exception's details  If you would like to log any exceptions being raised in the task (instead of just printing them out), you could use Python's logging module—have a look at this answer, as well as this answer and this answer on how to do that. You may also find helpful information on FastAPI/Starlette's custom/global exception handlers at this post and this post, as well as here, here and here. Finally, this answer will help you understand in detail the difference between def and async def endpoints (as well as background task functions) in FastAPI, and find solutions for tasks blocking the event loop (if you ever come across this issue).
Question: I created a new environment using conda and wanted to add it to jupyter-lab. I got a warning about frozen modules? (shown below) $ ipython kernel install --user --name=testi2 0.00s - Debugger warning: It seems that frozen modules are being used, which may 0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off 0.00s - to python to disable frozen modules. 0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation. Installed kernelspec testi2 in /home/michael/.local/share/jupyter/kernels/testi2  All I had installed were ipykernel, ipython, ipywidgets, jupyterlab_widgets, ipympl Python Version 3.11.0, Conda version 22.11.0 And I used conda install nodejs -c conda-forge --repodata-fn=repodata.json to get the latest version of nodejs I also tried re-installing ipykernel to a previous version (6.20.1 -> 6.19.2)

Answer: This is just a warning that the debugger cannot debug frozen modules.  In Python 3.11, the core modules essential for Python startup are “frozen”. ... This reduces the steps in module execution process ... Interpreter startup is now 10-15% faster in Python 3.11. This has a big impact for short-running programs using Python.  — What’s New In Python 3.11 § Faster Startup  it's not possible for the debugger to debug frozen modules as the filename is definitely required to hit breakpoints  — https://github.com/fabioz/PyDev.Debugger/issues/213#issuecomment-1058247166 E.g. os.path.realpath.__code__.co_filename is now '<frozen posixpath>' in Python 3.11.  The possible resolutions are mentioned with the warning.  If you need to debug those modules, pass -Xfrozen_modules=off to python: # ipython kernel install --user --name=testi2 python -Xfrozen_modules=off -m ipykernel install --user --name=testi2  # jupyter-lab python -Xfrozen_modules=off -m jupyterlab   If you just want to suppress the warning, set PYDEVD_DISABLE_FILE_VALIDATION=1: PYDEVD_DISABLE_FILE_VALIDATION=1 ipython kernel install --user --name=testi2  PYDEVD_DISABLE_FILE_VALIDATION=1 jupyter-lab
Question: I have a DBT project, and a python script will be grabbing data from the postgresql to produce output. However, part of the python script will need to make the DBT run. I haven't found the library that will let me cause a DBT run from an external script, but I'm pretty sure it exists. How do I do this? ETA: The correct answer may be to download the DBT CLI and then use python system calls to use that.... I was hoping for a library, but I'll take what I can get.

Answer: Update: v1.5 has arrived! With v1.5 of dbt, we get a stable and officially supported Python API for invoking dbt operations; this API has functional parity with the CLI. From the docs: from dbt.cli.main import dbtRunner, dbtRunnerResult # initialize dbt = dbtRunner() # create CLI args as a list of strings cli_args = ["run", "--select", "tag:my_tag"] # run the command res: dbtRunnerResult = dbt.invoke(cli_args) # inspect the results for r in res.result: print(f"{r.node.name}: {r.status}")  There are some caveats about the stability of artifacts returned by dbt.invoke; read the docs for more details. Original Answer (As of Jan 2023) There is not a public Python API for dbt, yet. It is expected in v1.5, which should be out in a couple months. Right now, your safest option is to use the CLI. If you don't want to use subprocess, the CLI uses Click now, and Click provides a runner that you can use to invoke Click commands. It's usually used for testing, but I think it would work for your use case, too. The CLI command is here. That would look something like: from click.testing import CliRunner from dbt.cli.main import run dbt_runner = CliRunner() dbt_runner.invoke(run, args="-s my_model")  You could also invoke dbt the way they do in the test suite, using run_dbt.
Question: I am attempting to use Flask's rate limiting library to rate limit an API based on the seconds. So I have used this exact same format to limit requests to an API on an Apache Server. However I am now using an NGINX. I do not thinks this makes a difference but when I run this code: import api app = Flask(__name__, instance_relative_config=True) limiter = Limiter(app, default_limits=["5/second"], key_func=lambda: get_remote_address) limiter.limit("5/second", key_func=lambda: request.args.get('token') if 'token' in request.args else get_remote_address)(api.bp) app.register_blueprint(api.bp)  Again I have ran this exact same code on another server, but now it is giving this error:  limiter = Limiter(app, "5/second", key_func=lambda: request.args.get('token') if 'token' in request.args else get_remote_address) TypeError: Limiter.__init__() got multiple values for argument 'key_func'  Any help would be great. I am using Flask-Limiter in python and running gevent on gunicorn server for NGINX.

Answer: Your Limiter class instantiation is incorrect. Below is the correct one- limiter = Limiter(get_remote_address, app=app, default_limits=["200 per day", "50 per hour"])
Question: I am trying to install locally Stable Diffusion. I follow the presented steps but when I get to the last one "run webui-use file" it opens the terminal and it's saying "Press any key to continue...". If I do so the terminal instantly closes. I went to the SB folder, right-clicked open in the terminal and used ./webui-user to run the file. The terminal does not longer close but nothing is happening and I get those two errors: Couldn't install torch, No matching distribution found for torch==1.12.1+cu113 I've researched online and I've tried installing the torch version from the error, also I tried pip install --user pipenv==2022.1.8 but I get the same errors.

Answer: I had a similar error. What solved the issue was, I installed the 32 bit version of python. I uninstalled and installed the x86-64 bit version. And everything installed just fine. I have version 3.8 installed as i have found out it is the most stable version. I downloaded the 64 bit installer from here python 3.8
Question: I am trying to understad the keypoint output of the yolov7, but I didn't find enough information about that. I have the following output: array([ 0, 0, 430.44, 476.19, 243.75, 840, 0.94348, 402.75, 128.5, 0.99902, 417.5, 114.25, 0.99658, 385.5, 115, 0.99609, 437.75, 125.5, 0.89209, 366.75, 128, 0.66406, 471, 229.62, 0.97754, 346.75, 224.88, 0.97705, 526, 322.75, 0.95654, 388.5, 340.75, 0.95898, 424.5, 314.75, 0.94873, 483.5, 335.5, 0.9502, 465.5, 457.75, 0.99219, 381.5, 456.25, 0.99219, 451.5, 649, 0.98584, 379.25, 649.5, 0.98633, 446.5, 818, 0.92285, 366, 829.5, 0.9248])  the paper https://arxiv.org/pdf/2204.06806.pdf tells "So, in total there are 51 elements for 17 keypoints associated with an anchor. " but the length is 58. there are 18 numbers that probably are confidences of a keypoint: array([ 0.94348, 0.99902,, 0.99658, 0.99609, 0.89209, 0.66406, 0.97754, 0.97705, 0.95654, 0.95898, 0.94873, 0.9502, 0.99219, 0.99219, 0.98584, 0.98633, 0.92285, 0.9248])  But the paper tells that are 17 keypoints. In this repo https://github.com/retkowsky/Human_pose_estimation_with_YoloV7/blob/main/Human_pose_estimation_YoloV7.ipynb tells that the keypoints are the following:  but that shape doesn't match the prediction:  Is the first image right about the keypoints? and what are the first four digits?  0, 0, 430.44, 476.19  Thanks EDIT This is not a complet answer but editing the plot function I can get the following information Given the following output keypoint: array([[ 0, 0, 312.31, 486, 291.75, 916.5, 0.94974, 304.5, 118.75, 0.99902, 320.75, 102.25, 0.99756, 287.75, 103.25, 0.99658, 345, 112, 0.96338, 268.25, 115.25, 0.69531, 394, 226.25, 0.98145, 228.25, 230.12, 0.98389, 428.5, 358.5, 0.95898, 192.88, 364.75, 0.96533, 407, 464.25, 0.95166, 215.75, 464.25, 0.9585, 363.75, 491, 0.99219, 257.75, 491.5, 0.99268, 361.5, 680, 0.9834, 250.88, 679, 0.98438, 361, 861.5, 0.91064, 247, 863, 0.91504]])  from this position ouput[7:] you can get the points of each keypoint, with the following sort as you can see in the image  array([ 304.5, 118.75, 0.99902, 320.75, 102.25, 0.99756, 287.75, 103.25, 0.99658, 345, 112, 0.96338, 268.25, 115.25, 0.69531, 394, 226.25, 0.98145, 228.25, 230.12, 0.98389, 428.5, 358.5, 0.95898, 192.88, 364.75, 0.96533, 407, 464.25, 0.95166, 215.75, 464.25, 0.9585, 363.75, 491, 0.99219, 257.75, 491.5, 0.99268, 361.5, 680, 0.9834, 250.88, 679, 0.98438, 361, 861.5, 0.91064, 247, 863, 0.91504])  but I am not sure about what are the rest of the values: 0, 0, 312.31, 486, 291.75, 916.5, 0.94974,

Answer: I assume you have passed your output through output_to_keypoint function in utils.plots. Based on the comment left by the authors of that function, the first 7 values should be (in order):  batch_id class_id x coordinate of the center of the bounding box y coordinate of the center of the bounding box w - width of the bounding box h - height of the bounding box conf - confidence in the bounding box
Question: After Mark Shannon's optimisation of Python objects, is a plain object different from an object with slots? I understand that after this optimisation in a normal use case, objects have no dictionary. Have the new Python objects made the use of slots totally unnecessary?

Answer: No, __slots__ still produces more compact objects. With __slots__ for attributes, an object's memory layout only needs one pointer per slot. With the new lazy __dict__ creation, an object needs to store a PyDictValues object and a pointer to that PyDictValues object. The PyDictValues object contains room for a number of pointers based on the "usable size" of the shared keys object when the PyDictValues object is created, which is usually more pointers than what you would have gotten with __slots__. It also holds some extra metadata and padding, stored before those pointers: a "prefix size" representing the size of this metadata, a "used size" representing the number of values stored, and an array of bytes tracking insertion order. The padding is used to ensure this extra metadata doesn't interfere with the alignment of the pointers. (None of this is reflected in the PyDictValues struct definition, since C struct definitions aren't expressive enough for this - it mostly has to be handled manually.) So, without __slots__, you've got an extra PyDictValues * in the object directly, usually more room allocated for attribute pointers than necessary, and a bunch of extra metadata in the PyDictValues itself, relative to using __slots__. Plus, with __slots__, if you don't explicitly declare a __weakref__ slot, you don't get one, saving some memory at the cost of not being able to create weak references to your objects. With no __slots__, Python will automatically add a __weakref__ slot to your objects.
Question: I have 3 rules and their names are somewhat long. When using ruleorder, the line goes over my desired 80 character limit. Is it possible break up the ruleorder into multiple lines in such a way that the behaviour is exactly the same as if I wrote it all in one line? Example: ruleorder: long_rule_1 > long_rule_2 > long_rule_3  I would like to reformat it into something like this: ruleorder: ( long_rule_1 > long_rule_2 > long_rule_3 ) 

Answer: After looking at ways to do this, I believe the best way is pretty simple: ruleorder: long_rule_1 > long_rule_2 > long_rule_3  The other answers are good too, but this is the one that I'm using
Question: The Python Tutorial chapter 6 (Modules) says:  The directory containing the script being run is placed at the beginning of the search path, ahead of the standard library path. This means that scripts in that directory will be loaded instead of modules of the same name in the library directory. This is an error unless the replacement is intended.  Why is this an error? In other terms, what is the meaning of the sentence "This is an error unless the replacement is intended"?

Answer:  After initialization, Python programs can modify sys.path. The directory containing the script being run is placed at the beginning of the search path, ahead of the standard library path. This means that scripts in that directory will be loaded instead of modules of the same name in the library directory. This is an error unless the replacement is intended. See section Standard Modules for more information.  Assuming you have a package (= module) installed called fooo. This is the package you want to use for e script. Now, if you import it using fooo, everything works as intended. Now, if there's a module in the directory mentioned above, with the same name (fooo), this one will replace your intended package. The error is, that you now got imported the wrong package. Resource: https://docs.python.org/3/tutorial/modules.html#the-module-search-path
Question: I'm trying to find a way to pass a string (coming from outside the python world!) that can be interpreted as **kwargs once it gets to the Python side. I have been trying to use this pyparsing example, but the string thats being passed in this example is too specific, and I've never heard of pyparsing until now. I'm trying to make it more, human friendly and robust to small differences in spacing etc. For example, I would like to pass the following. input_str = "a = [1,2], b= False, c =('abc', 'efg'),d=1" desired_kwargs = {a : [1,2], b:False, c:('abc','efg'), d:1}  When I try this code though, no love. from pyparsing import * # Names for symbols _quote = Suppress('"') _eq = Suppress('=') # Parsing grammar definition data = ( delimitedList( # Zero or more comma-separated items Group( # Group the contained unsuppressed tokens in a list Regex(u'[^=,)\s]+') + # Grab everything up to an equal, comma, endparen or whitespace as a token Optional( # Optionally... _eq + # match an = _quote + # a quote Regex(u'[^"]*') + # Grab everything up to another quote as a token _quote) # a quote ) # EndGroup - will have one or two items. )) # EndList def process(s): items = data.parseString(s).asList() args = [i[0] for i in items if len(i) == 1] kwargs = {i[0]:i[1] for i in items if len(i) == 2} return args,kwargs def hello_world(named_arg, named_arg_2 = 1, **kwargs): print(process(kwargs)) hello_world(1, 2, "my_kwargs_are_gross = True, some_bool=False, a_list=[1,2,3]") #output: "{my_kwargs_are_gross : True, some_bool:False, a_list:[1,2,3]}"  Requirements:  The '{' and '}' will be appended on the code side. Only standard types / standard iterables (list, tuple, etc) will be used in the kwargs-string. No special characters that I can think of... The kwargs-string will be like they are entered into a function on the python side, ie, 'x=1, y=2'. Not as a string of a dictionary. I think its a safe assumption that the first step in the string parse will be to remove all whitespace. 

Answer: One option could be to use the ast module to parse some wrapping of the string that turns it into a valid Python expression. Then you can even use ast.literal_eval if you’re okay with everything it can produce: >>> import ast >>> kwargs = "a = [1,2], b= False, c =('abc', 'efg'),d=1" >>> expr = ast.parse(f"dict({kwargs}\n)", mode="eval") >>> {kw.arg: ast.literal_eval(kw.value) for kw in expr.body.keywords} {'a': [1, 2], 'b': False, 'c': ('abc', 'efg'), 'd': 1}
Question: I've been using dataframe_image for a while and have had great results so far. Last week, out of a sudden, all my code containing the method dfi.export() stopped working with this error as an output raise SyntaxError("not a PNG file") File <string> SyntaxError: not a PNG file  I can export the images passing the argument table_conversion='matplotlib' but they do not come out styled... This is my code: now = str(datetime.now()) filename = ("Extracciones-"+now[0:10]+".png") df_styled = DATAFINAL.reset_index(drop=True).style.apply(highlight_rows, axis=1) dfi.export(df_styled, filename,max_rows=-1) IMAGEN = Image.open(filename) IMAGEN.show()  Any clues on why this just suddenly stopped working? Or any ideas to export dataframes as images (not using html)? These were the outputs i used to get: fully styled dataframe images and this is the only thing I can get right now Thank you in advance

Answer: dataframe_image has a dependency on Chrome, and a recent Chrome update (possibly v109 on 2013-01-10) broke dataframe_image. v0.1.5 was released on 2023-01-14 to fix this. pip install --upgrade dataframe_image pip show dataframe_image  The version should now be v0.1.5 or later, which should resolve the problem.  Some users have reported still having the error even after upgrading. This could be due to upgrading the package in the wrong directory (due to multiple installations of python, pip, virtual envs, etc). The reliable way to check the actual version of dataframe_image that the code is using, is to add this debugging code to the top of your python code: import pandas as pd import dataframe_image as dfi from importlib.metadata import version print(version('dataframe_image')) df = pd.DataFrame({'x':[1,2]}) dfi.export(df, 'out.png') exit()  Also check chrome://version/ in your Chrome browser.
Question: When reading a toml file in normal read ("r") mode, I get an error import tomli with open("path_to_file/conf.toml", "r") as f: # have to use "rb" ! toml_dict = tomli.load(f)   TypeError: File must be opened in binary mode, e.g. use open('foo.toml', 'rb')  Same happens when writing a toml file. Why? tomli github readme says  The file must be opened in binary mode (with the "rb" flag). Binary mode will enforce decoding the file as UTF-8 with universal newlines disabled, both of which are required to correctly parse TOML.  I thought the age of typewriters was over, so why is the "universal newline" not allowed? toml spec says "Newline means LF (0x0A) or CRLF (0x0D 0x0A)" (poor Mac users) - that also doesn't clarify the reason to me... so, what am I missing?

Answer: To wrap this up, the problem/behavior described in the question is actually a specific case of a more general problem: how to enforce a specific decoding when reading a text file with Python's open built-in. Or rephrased: ensure the file has a specific encoding. tomli requires the user to handle the file IO, so the user could also use an arbitrary encoding in open(path-to-file, "r", encoding=...). However, the toml specification requires the input to be UTF-8. tomli implements this requirement by forcing the user to use binary mode "b" when reading the file, then does the decoding based on the read bytes (src).
Question: I need your help for the following task: I have the following dataframe: test = {'Col1':[2,5], 'Col2':[5,7], 'Col_List':[['One','Two','Three','Four','Five'], ['Two', 'Four']], 'One':[0,0], 'Two':[0,0], 'Three':[0,0], 'Four':[0,0], 'Five':[0,0],} df=pd.DataFrame.from_dict(test) df  which looks like:     Col1 Col2 Col_List One Two Three Four Five     2 5 [One, Two, Three, Four, Five] 0 0 0 0 0   5 7 [Two, Four] 0 0 0 0 0     I need to inspect the list in Col_List and set, depending on which item is in the list, the value of column Col1 in the specific column (One, Two, Three, Four or Five). Now I would like to have the following result:     Col1 Col2 Col_List One Two Three Four Five     2 5 [One, Two, Three, Four, Five] 2 2 2 2 2   5 7 [Two, Four] 0 5 0 5 0    

Answer: exploded = df.explode("Col_List") df.update(pd.get_dummies(exploded["Col_List"]) .mul(exploded["Col1"], axis="rows") .groupby(level=0).sum())   explode lists' elements to their own rows get 1-hot representation of "One", "Two" etc. multiply it with the (exploded) "Col1" values  1/0 values will act as a selector   then undo the explosion: groupby & sum lastly update the original frame's "One", "Two"... columns with this  to get >>> df Col1 Col2 Col_List One Two Three Four Five 0 2 5 [One, Two, Three, Four, Five] 2 2 2 2 2 1 5 7 [Two, Four] 0 5 0 5 0
Question: I have a dataframe that looks like :  df = pd.DataFrame({'qty': [10,7,2,1], 'status 1': [5,2,2,0], 'status 2': [3,2,0,1], 'status 3': [2,3,0,0] })  Each row has a qty of items. These items have one status (1,2 or 3). So qty = sum of values of status 1,2,3. I would like to :  Duplicate each row by the "qty" column  Then edit 3 status (or update a new column), to get just 1 status.   The output should look like this:  Edit: the order is not important, but I will need to keep other columns of my initial df. My (incomplete) solution so far - I found a way to duplicate the rows using this : df2= df2.loc[df2.index.repeat(df2['qty'])].reset_index(drop=True)  But I can't find a way to fill the status. Do I need to use a for loop approach to fill the status? Should I do this transform in 1 or 2 steps? Something like: for each initial row, the n first rows take the first status, where n is the value of status 2.... The output could maybe looks like :  Edit1 : Thank you for your answers ! Last question : now I'm trying to integrate this to my actual df. What is the best approach to apply these methods to my df who contains many other column ? I will answer my last question :  Split df in 2: dfstatus and dfwithoutstatus, keeping the qty column in both  Apply one of your method on the dfstatus  Apply my method on the dfwithoutstatus (a simple duplication)  Merge on index   Thank you all for your answers. Best

Answer: Here is a possible solution: import numpy as np import pandas as pd E = pd.DataFrame(np.eye(df.shape[1] - 1, dtype=int)) result = pd.DataFrame( df['qty'].reindex(df.index.repeat(df['qty'])).reset_index(drop=True), ) result[df.columns[1:]] = pd.concat( [E.reindex(E.index.repeat(df.iloc[i, 1:])) for i in range(len(df))], ).reset_index( drop=True, )  Here is the result: >>> result qty status 1 status 2 status 3 0 10 1 0 0 1 10 1 0 0 2 10 1 0 0 3 10 1 0 0 4 10 1 0 0 5 10 0 1 0 6 10 0 1 0 7 10 0 1 0 8 10 0 0 1 9 10 0 0 1 10 7 1 0 0 11 7 1 0 0 12 7 0 1 0 13 7 0 1 0 14 7 0 0 1 15 7 0 0 1 16 7 0 0 1 17 2 1 0 0 18 2 1 0 0 19 1 0 1 0