eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'm building an async backend for an analytics system using FastAPI. The thing is it has to: a) listen for API calls and be available at all times; b) periodically perform a data-gathering task (parsing data and saving it into the DB). I wrote this function to act as a daemon:  async def start_metering_daemon(self) -> None: """sets a never ending task for metering""" while True: delay: int = self._get_delay() # delay in seconds until next execution await asyncio.sleep(delay) await self.gather_meterings() # perfom data gathering  What I'm trying to achieve is so that when app starts it also adds this daemon function into the main event loop and execute it when it has time. However, I haven't been able to find a suitable solution which is adequate to the scale of the task (adding Celery and similar stuff is an overkill). I have tried following ways to achieve this but none of them worked: @app.on_event("startup") async def startup_event() -> None: """tasks to do at server startup""" await Gatherer().start_metering_daemon()  Result: server can't start up since the thread is blocked @app.on_event("startup") async def startup_event() -> None: """tasks to do at server startup""" fastapi.BackgroundTasks().add_task(Gatherer().start_metering_daemon)  Result: task is never executed as observed in logs @app.on_event("startup") async def startup_event() -> None: """tasks to do at server startup""" fastapi.BackgroundTasks().add_task(asyncio.run, Gatherer().start_metering_daemon())  Result: same as previous one @app.on_event("startup") async def startup_event() -> None: """tasks to do at server startup""" threading.Thread(target=asyncio.run, args=(Gatherer().start_metering_daemon(),)).start()  Result: this one works but a) makes no sence; b) spawns N identical threads for N Uvicorn workers which all write same data N times into the DB. I am out of solutions by now. I am pretty sure there must be a solution to my problem since is looks pretty trivial to me but I couldn't find one. If you want more context here is the repo of the project I reffer to.

Answer: try @app.on_event("startup") async def startup_event() -> None: """tasks to do at server startup""" asyncio.create_task(Gatherer().start_metering_daemon())
Question: The comment are causing errors. I have a contents.json file which looks like: { "Fridge": [ ["apples"], ["chips","cake","10"] // This comment here is causing error ], "car": [ ["engine","tires","fuel"], ] }  My python script is like this import json jsonfile = open('contents.json','r') jsondata = jsonfile.read() objec = json.loads(jsondata) list_o = objec['Fridge'] for i in (list_o): print(i)  In my list_o, i am trying to load Fridge from contents.jsonfile, when JSON file has that comment, it gives me an error, when the JSON file doesn't have the comment, the script runs properly. I understand that comments is not proper JSON format, but is there any way to ignore comments of JSON file?

Answer: Read the file per line and remove the comment part. import json jsondata = "" with open('contents.json', 'r') as jsonfile: for line in jsonfile: jsondata += line.split("//")[0] objec = json.loads(jsondata) list_o = objec['Fridge'] for i in (list_o): print(i)  ['apples'] ['chips', 'cake', '10']  Update You can also easily just use a library such as commentjson. Just replace : objec = json.loads(jsondata)  To import commentjson # python3 -m pip install commentjson objec = commentjson.loads(jsondata)
Question: If I do something like this some_obj = {"a": 1, "b": 2, "c": 3} first, *rest = some_obj  I'll get a list, but I want it in 2 dictionaries: first = {"a": 1} and rest = {"b": 2, "c": 3}. As I understand, I can make a function, but I wonder if I can make it in one line, like in javascript with spread operator.

Answer: I don't know if there is a reliable way to achieve this in one line, But here is one method. First unpack the keys and values(.items()). Using some_obj only iterate through the keys.  >>> some_obj = {"a":1, "b":2, "c": 3} >>> first, *rest = some_obj.items()  But this will return a tuple, >>> first ('a', 1) >>> rest [('b', 2), ('c', 3)]  But you can again convert back to dict with just a dict call. >>> dict([first]) {'a': 1} >>> dict(rest) {'b': 2, 'c': 3}
Question: Is there a shorthand in Python for (e.g.) print(f'type(var) = {type(var)}'), without having to state the object in the text and the {.}? The short answer may be "no", but I had to ask! E.g. in SAS one may use &= to output a macro variable and its value to the log... %let macrovar = foobar; %put &=macrovar;  which returns in the log: MACROVAR = foobar This is my first question, and I found it difficult to search for an answer, so apologies if it's been asked and answered.

Answer: Indeed there is. As of python 3.8, you can simply type f'{type(var)=}', and you will get the output you desire: >>> x = {} >>> f'{x=}' 'x={}' >>> f'{type(x)=}' "type(x)=<class 'dict'>"  Further reading:  The "What's New In Python 3.8" page The documentation for f-strings The discussion on BPO that led to this feature being implemented.
Question: I have a list like: lst = [1,2,3,4,5,6,7,8,9,10]  and I want to get the combination of all splits for a given n bucket without changing the order of the list. Output exp for n=3: [ [1],[2],[3,4,5,6,7,8,9,10], [1],[2,3],[4,5,6,7,8,9,10], [1],[2,3,4],[5,6,7,8,9,10], . . . [1,2,3,4,5,6,7,8],[9],[10], ]  Python is the language I use but if you can direct me to an algorithm that would nice as well. I see this problem is usually apllied on strings. But couldn't figure it out on the list. P.S. this is my first question. Any feedback is appreciated on how to improve the question.

Answer: Try: from itertools import product def generate(n, l): for c in product(range(1, l), repeat=n - 1): s = sum(c) if s > l - 1: continue yield *c, l - s lst = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] n = 3 for groups in generate(n, len(lst)): l, out = lst, [] for g in groups: out.append(l[:g]) l = l[g:] print(out)  Prints: [[1], [2], [3, 4, 5, 6, 7, 8, 9, 10]] [[1], [2, 3], [4, 5, 6, 7, 8, 9, 10]] [[1], [2, 3, 4], [5, 6, 7, 8, 9, 10]] [[1], [2, 3, 4, 5], [6, 7, 8, 9, 10]] [[1], [2, 3, 4, 5, 6], [7, 8, 9, 10]] [[1], [2, 3, 4, 5, 6, 7], [8, 9, 10]] [[1], [2, 3, 4, 5, 6, 7, 8], [9, 10]] [[1], [2, 3, 4, 5, 6, 7, 8, 9], [10]] [[1, 2], [3], [4, 5, 6, 7, 8, 9, 10]] [[1, 2], [3, 4], [5, 6, 7, 8, 9, 10]] [[1, 2], [3, 4, 5], [6, 7, 8, 9, 10]] [[1, 2], [3, 4, 5, 6], [7, 8, 9, 10]] [[1, 2], [3, 4, 5, 6, 7], [8, 9, 10]] [[1, 2], [3, 4, 5, 6, 7, 8], [9, 10]] [[1, 2], [3, 4, 5, 6, 7, 8, 9], [10]] [[1, 2, 3], [4], [5, 6, 7, 8, 9, 10]] [[1, 2, 3], [4, 5], [6, 7, 8, 9, 10]] [[1, 2, 3], [4, 5, 6], [7, 8, 9, 10]] [[1, 2, 3], [4, 5, 6, 7], [8, 9, 10]] [[1, 2, 3], [4, 5, 6, 7, 8], [9, 10]] [[1, 2, 3], [4, 5, 6, 7, 8, 9], [10]] [[1, 2, 3, 4], [5], [6, 7, 8, 9, 10]] [[1, 2, 3, 4], [5, 6], [7, 8, 9, 10]] [[1, 2, 3, 4], [5, 6, 7], [8, 9, 10]] [[1, 2, 3, 4], [5, 6, 7, 8], [9, 10]] [[1, 2, 3, 4], [5, 6, 7, 8, 9], [10]] [[1, 2, 3, 4, 5], [6], [7, 8, 9, 10]] [[1, 2, 3, 4, 5], [6, 7], [8, 9, 10]] [[1, 2, 3, 4, 5], [6, 7, 8], [9, 10]] [[1, 2, 3, 4, 5], [6, 7, 8, 9], [10]] [[1, 2, 3, 4, 5, 6], [7], [8, 9, 10]] [[1, 2, 3, 4, 5, 6], [7, 8], [9, 10]] [[1, 2, 3, 4, 5, 6], [7, 8, 9], [10]] [[1, 2, 3, 4, 5, 6, 7], [8], [9, 10]] [[1, 2, 3, 4, 5, 6, 7], [8, 9], [10]] [[1, 2, 3, 4, 5, 6, 7, 8], [9], [10]]
Question: I want to draw a seaborn.heatmap and annotate only some rows/columns. Example where all cells have annotation: import seaborn as sns import matplotlib.pyplot as plt import numpy as np n1 = 5 n2 = 10 M = np.random.random((n1, n2)) fig, ax = plt.subplots() sns.heatmap(ax = ax, data = M, annot = True) plt.show()   Following these examples (paragraph Adding Value Annotations), it is possible to pass to seaborn.heatmap an array with annotations for each cell as annot parameter:  annot : bool or rectangular dataset, optional If True, write the data value in each cell. If an array-like with the same shape as data, then use this to annotate the heatmap instead of the data. Note that DataFrames will match on position, not index.  If I try to generate an array of str and pass it as annot parameter to seaborn.heatmap I get the following error: Traceback (most recent call last): File "C:/.../myfile.py", line 16, in <module> sns.heatmap(ax = ax, data = M, annot = A) File "C:\venv\lib\site-packages\seaborn\_decorators.py", line 46, in inner_f return f(**kwargs) File "C:\venv\lib\site-packages\seaborn\matrix.py", line 558, in heatmap plotter.plot(ax, cbar_ax, kwargs) File "C:\venv\lib\site-packages\seaborn\matrix.py", line 353, in plot self._annotate_heatmap(ax, mesh) File "C:\venv\lib\site-packages\seaborn\matrix.py", line 262, in _annotate_heatmap annotation = ("{:" + self.fmt + "}").format(val) ValueError: Unknown format code 'g' for object of type 'numpy.str_'  Code which generates the ValueError (in this case I try to remove annotations of the 4th columns as an example): import seaborn as sns import matplotlib.pyplot as plt import numpy as np n1 = 5 n2 = 10 M = np.random.random((n1, n2)) A = np.array([[f'{M[i, j]:.2f}' for j in range(n2)] for i in range(n1)]) A[:, 3] = '' fig, ax = plt.subplots(figsize = (6, 3)) sns.heatmap(ax = ax, data = M, annot = A) plt.show()  What is the cause of this error? How can I generate a seaborn.heatmap and annotate only selected rows/columns?

Answer: It is a formatting issue. Here the fmt = '' is required if you are using non-numeric labels (defaults to: fmt='.2g') which consider only for numeric values and throw an error for labels with text format. import seaborn as sns import matplotlib.pyplot as plt import numpy as np n1 = 5 n2 = 10 M = np.random.random((n1, n2)) A = np.array([[f'{M[i, j]:.2f}' for j in range(n2)] for i in range(n1)]) A[:, 3] = '' fig, ax = plt.subplots(figsize = (6, 3)) sns.heatmap(ax = ax, data = M, annot = A, fmt='') plt.show()
Question: this is the way when I try to get the Kafka version rpm -qa | grep "^kafka_" kafka_2_6_5_0_292-1.0.0.2.6.5.0-292.noarch  Kafka version is 1.0 , so I did the following in order to cut the Kafka version  rpm -qa | grep "^kafka_" | sed s'/-/ /g' | awk '{print $2}' | cut -c 1-3 1.0 <----- results  above cli seems to be not so elegant and long syntax can we do it better , maybe with Perl or Python one liner command ?

Answer: Refactoring your code rpm -qa | grep "^kafka_" | sed s'/-/ /g' | awk '{print $2}' | cut -c 1-3  1st step: use AWK's FS (Field Seperator) instead preprocessing in sed rpm -qa | grep "^kafka_" | awk 'BEGIN{FS="-"}{print $2}' | cut -c 1-3  2nd step: register {print $2} action to lines matching description rather than filtering it with grep rpm -qa | awk 'BEGIN{FS="-"}/^kafka_/{print $2}' | cut -c 1-3  3rd step: use AWK's substr function in place of cut -c rpm -qa | awk 'BEGIN{FS="-"}/^kafka_/{print substr($2,1,3)}'  Disclaimer: my answer assumes you want behavior exactly like your original code, even if possibly unexpected i.e. it does get first 3 characters of version parts, regardless of how many digits are in 2nd part so for example for 1.15.0.2.6.5.0-292 it does yield 1.1
Question: I have a function that needs to return multiple values: def max_dd(ser): ... compute i,j,dd return i,j,dd  if I have code like this that calls this function passing in a series:  date1, date2, dd = df.rolling(window).apply(max_dd)  however, I get an error: pandas.core.base.DataError: No numeric types to aggregate  If I return a single value from max_dd, everything is fine. How do I return multiple values from a function that has been "apply"?

Answer: Rolling apply can only produce single numeric values. There is no support for multiple returns or even nonnumeric returns (like something as simple as a string) from rolling apply. Any answer to this question will be a work around. That said, a viable workaround is to take advantage of the fact that rolling objects are iterable (as of pandas 1.1.0). What’s new in 1.1.0 (July 28, 2020)  Made pandas.core.window.rolling.Rolling and pandas.core.window.expanding.Expanding iterable（GH11704)  Meaning that it is possible to take advantage of the faster grouping and indexing operations of the rolling function, but obtain more flexible behaviour with python: def some_fn(df_): """ When iterating over a rolling window it disregards the min_periods argument of rolling and will produce DataFrames for all windows The input is also of type DataFrame not Series You are completely responsible for doing all operations here, including ignoring values if the input is not of the correct shape or format :param df_: A DataFrame produced by rolling :return: a column joined, and the max value within the window """ return ','.join(df_['a']), df_['a'].max() window = 5 results = pd.DataFrame([some_fn(df_) for df_ in df.rolling(window)])  Sample DataFrame and output: df = pd.DataFrame({'a': list('abdesfkm')})  df:  a 0 a 1 b 2 d 3 e 4 s 5 f 6 k 7 m  result:  0 1 0 a a 1 a,b b 2 a,b,d d 3 a,b,d,e e 4 a,b,d,e,s s 5 b,d,e,s,f s 6 d,e,s,f,k s 7 e,s,f,k,m s
Question: I would like to use getattr() to access Python's built-in functions. Is that possible? For example: getattr(???, 'abs')  I know I can just simply do: >>> abs <built-in function abs>  But I want to use getattr, because the keyword names are strings.

Answer: The builtins module: You could try importing builtins module: >>> import builtins >>> getattr(builtins, 'abs') <built-in function abs> >>>  As mentioned in the documentation:  This module provides direct access to all ‘built-in’ identifiers of Python; for example, builtins.open is the full name for the built-in function open(). See Built-in Functions and Built-in Constants for documentation.  So the above mentions that builtins.open is the open function. So abs is the same builtins.abs is the same thing as abs. But for gettatr, getattr(builtins, 'abs') is also the same as builtins.abs. The original __builtins__ (NOT RECOMMENDED): You could try getting from the __builtins__: >>> getattr(__builtins__, 'abs') <built-in function abs> >>>  As mentioned in the documentation:  CPython implementation detail: Users should not touch __builtins__; it is strictly an implementation detail. Users wanting to override values in the builtins namespace should import the builtins module and modify its attributes appropriately. The builtins namespace associated with the execution of a code block is actually found by looking up the name __builtins__ in its global namespace; this should be a dictionary or a module (in the latter case the module’s dictionary is used). By default, when in the main module, __builtins__ is the built-in module builtins; when in any other module, __builtins__ is an alias for the dictionary of the builtins module itself.  As you can see, it's not recommended, also usually the __builtins__ is a dict, rather than a module. If you write your code in modules, __builtins__ would return dictionary aliases of the builtins module, which would give something like: {..., 'abs': <built-in function abs>, ...}. More on getattr: Just to have more idea about getattr, as mentioned in the documentation:  Return the value of the named attribute of object. name must be a string. If the string is the name of one of the object’s attributes, the result is the value of that attribute. For example, getattr(x, 'foobar') is equivalent to x.foobar. If the named attribute does not exist, default is returned if provided, otherwise AttributeError is raised.  So: >>> import builtins >>> getattr(builtins, 'abs') <built-in function abs> >>>  Is the same as: >>> import builtins >>> builtins.abs <built-in function abs> >>>  So you might be wondering since: >>> abs <built-in function abs>  Gives the same thing, why we can't just do: getattr(abs)  The reason we can't do that is that that getattr is suppose to be calling methods/functions/classes of a classes/modules. The reason using getattr(builtins, 'abs') works is because builtins is a module and abs is a class/method, it stores all the built-in Python keywords as methods in that module. All the keywords are showed on this page of the documentation.
Question: Some of the packages I use don't type hint their code, so when I use them, Pylance keeps telling me that the functions I use have partially unknown types, which is a problem I can't fix. Is there a way to disable such errors?

Answer: If you're absolutely certain of the type you're getting from the external library and you're sure it's not documented through typeshed either, you can always cast it to signal to the type checker it's to be treated as that type. from typing import cast from elsewhere import Ham spam = some_untyped_return() ham = cast(Ham, spam)
Question: I have encountered this issue when I use Pylance and syntax highlighting is enabled for python in the VSCode with default or the visual studio theme. self and cls parameter are LightSkyBlue color like other parameters  It should be like this: 

Answer: Added the color code inside the settings.json file for the dark themes I use. // correct color self and cls python "editor.semanticTokenColorCustomizations": { "[Default Dark+]": { "rules": { "selfParameter": "#569CD6", "clsParameter": "#569CD6" }, }, "[Visual Studio Dark]": { "rules": { "selfParameter": "#569CD6", "clsParameter": "#569CD6" }, }, "[Default Dark Modern]": { "rules": { "selfParameter": "#569CD6", "clsParameter": "#569CD6" }, } },  Add theme settings based on what theme you are using. for example, I just changed my theme to [Default Dark Modern] and then added the color settings. These two issues on pylance and vscode github repository helped:  https://github.com/microsoft/pylance-release/issues/323 https://github.com/microsoft/vscode/issues/118946
Question: I have this simple program that should display a pie chart, but whenever I run the program, it opens a page on Chrome and just keeps loading without any display, and sometimes it refuses to connect. How do I solve this? P.S.: I would like to use it offline, and I'm running it using cmd on windows10 import pandas as pd import numpy as np from datetime import datetime import plotly.express as px def graph(dataframe): figure0 = px.pie(dataframe, values=dataframe['POPULATION'], names=dataframe['CONTINENT']) figure0.show() df = pd.DataFrame({'POPULATION': [60, 17, 9, 13, 1], 'CONTINENT': ['Asia', 'Africa', 'Europe', 'Americas', 'Oceania']}) graph(df) 

Answer: Disclaimer: I extracted this answer from the OPs question. Answers should not be contained in the question itself.  Answer provided by g_odim_3: So instead of figure0.show(), I used figure0.write_html('first_figure.html', auto_open=True) and it worked: import pandas as pd import numpy as np from datetime import datetime import plotly.express as px def graph(dataframe): figure0 = px.pie(dataframe, values=dataframe['POPULATION'], names=dataframe['CONTINENT'], title='Global Population') # figure0.show() figure0.write_html('first_figure.html', auto_open=True) df = pd.DataFrame({'POPULATION':[60, 17, 9, 13, 1], 'CONTINENT':['Asia', 'Africa', 'Europe', 'Americas', 'Oceania']}) graph(df)
Question: I installed and added Python3.9 and Pip to the PATH through the installer. python --version # Python 3.9.7 pip --version # pip 21.2.4 from C:\Users\{MyUserName}\AppData\Local\Programs\Python\Python39\lib\site-packages\pip (python 3.9)  I installed pipenv with pip install pipenv and pipenv --version outputs pipenv, version 2021.5.29. Although, if I try to install any package with pipenv, or just enter the pipenv shell and then run python --version, I always get No Python at 'C:\Python39\python.exe'. Python sys path is C:\Users\{MyUserName}\AppData\Local\Programs\Python\Python39, so why does pipenv look into another folder? And how can I fix this? I'm running all these commands in git bash.

Answer: For anyone running into this error, run the following to delete the virtual environment (built with the previous/future version of Python):  cd $project_folder pipenv --rm  Then rerun this to build your pipenv virtual environment with your new version of Python: pipenv install
Question: I'm trying to type hint a numpy ndarray like this: RGB = numpy.dtype[numpy.uint8] ThreeD = tuple[int, int, int] def load_images(paths: list[str]) -> tuple[list[numpy.ndarray[ThreeD, RGB]], list[str]]: ...  but at the first line when I run this, I got the following error: RGB = numpy.dtype[numpy.uint8] TypeError: 'numpy._DTypeMeta' object is not subscriptable  How do I type hint a ndarray correctly?

Answer: It turns out that strongly type a numpy array is not straightforward at all. I spent a couple of hours to figure out how to do it properly. A simple method that do not add yet another dependency to your project is to use a trick described here. Just wrap numpy types with with ': import numpy import numpy.typing as npt from typing import cast, Type, Sequence import typing RGB: typing.TypeAlias = 'numpy.dtype[numpy.uint8]' ThreeD: typing.TypeAlias = tuple[int, int, int] NDArrayRGB: typing.TypeAlias = 'numpy.ndarray[ThreeD, RGB]' def load_images(paths: list[str]) -> tuple[list[NDArrayRGB], list[str]]: ...  The trick is to use single-quotes to avoid the infamous TypeError: 'numpy._DTypeMeta' object is not subscriptable when Python tries to interpret the [] in the expression. This trick is well handled for instance by VSCode Pylance type-checker:  Notice that the colors for types are respected and that the execution gives no error. Note about nptyping As suggested by @ddejohn, one can use nptyping. Just install the package: pip install nptyping. However, as of now (16 June 2022), there is no Tuple type defined in nptyping so you won't be able to prefectly type you code that way. I have open a new issue so maybe in the future it will work. edits Turns out there is a different way to express a tuple as a nptyping.Shape as answered by ramonhagenaars, which is also elegant: from nptyping import NDArray, Shape, UInt8 # A 1-dimensional array (i.e. 1 RGB color). RGBArray1D = NDArray[Shape["[r, g, b]"], UInt8] # A 2-dimensional array (i.e. an array of RGB colors). RGBArrayND = NDArray[Shape["*, [r, g, b]"], UInt8] def load_images_trick(paths: list[str]) -> tuple[list[RGBArrayND], list[str]]: ...  However, this solution is not well supported by VSCode Pylance, an I get an error suggestion for Shape: Expected class type but received "Literal" "Literal" is not a class "Literal" is not a classPylancereportGeneralTypeIssues Pylance(reportGeneralTypeIssues)