eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: suppose if I have a web server like this: from fastapi import FastAPI import uvicorn import asyncio app = FastAPI() def blocking_function(): import time time.sleep(5) return 42 @app.get("/") async def root(): loop = asyncio.get_running_loop() result = await loop.run_in_executor(None, blocking_function) return result @app.get("/ok") async def ok(): return {"ok": 1} if __name__ == "__main__": uvicorn.run(app, host="0.0.0.0", workers=1)  As I understand, the code will spawn another thread in the default ThreadExecutorPool and then execute the blocking function in the thread pool. On the other side, thinking about how the GIL works, the CPython interpreter will only execute a thread for 100 ticks and then it will switch to another thread to be fair and give other threads a chance to progress. In this case, what if the Python interpreter decides to switch to the threads where the blocking_function is executing? Will it block the who interpreter to wait for whatever remaining on the time.sleep(5)? The reason I am asking this is that I have observed sometimes my application will block on the blocking_function, however I am not entirely sure what's in play here as my blocking_function is quite special -- it talks to a COM API object through the win32com library. I am trying to rule out that this is some GIL pitfalls I am falling into.

Answer: Both time.sleep (as explained in this question) and the win32com library (according to this mailing list post) release the GIL when they are called, so they will not prevent other threads from making progress while they are blocking. To answer the "high-level" question - "can run_in_executor ever (directly or indirectly) block the event-loop?" - the answer would only be "yes" if you used a ThreadPoolExecutor, and the code you executed in run_in_executor did blocking work that didn't release the GIL. While that wouldn't completely block the event loop, it would mean both your event loop thread and the executor thread could not run in parallel, since both would need to acquire the GIL to make progress.
Question: N, M = 1000, 4000000 a = np.random.uniform(0, 1, (N, M)) k = np.random.randint(0, N, (N, M)) out = np.zeros((N, M)) for i in range(N): for j in range(M): out[k[i, j], j] += a[i, j]  I work with very long for-loops; %%timeit on above with pass replacing the operation yields 1min 19s ± 663 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  this is unacceptable in context (C++ took 6.5 sec). There's no reason for above to be done with Python objects; arrays have well-defined types. Implementing this in C/C++ as an extension is an overkill on both developer and user ends; I'm just passing arrays to loop and do arithmetic on. Is there a way to tell Numpy "move this logic to C", or another library that can handle nested loops involving only arrays? I seek it for the general case, not workarounds for this specific example (but if you have one I can open a separate Q&A).

Answer: This is basically the idea behind Numba. Not as fast as C, but it can get close... It uses a jit compiler to compile python code to machine and it's compatible with most Numpy functions. (In the docs you find all the details) import numpy as np from numba import njit @njit def f(N, M): a = np.random.uniform(0, 1, (N, M)) k = np.random.randint(0, N, (N, M)) out = np.zeros((N, M)) for i in range(N): for j in range(M): out[k[i, j], j] += a[i, j] return out def f_python(N, M): a = np.random.uniform(0, 1, (N, M)) k = np.random.randint(0, N, (N, M)) out = np.zeros((N, M)) for i in range(N): for j in range(M): out[k[i, j], j] += a[i, j] return out  Pure Python: %%timeit N, M = 100, 4000 f_python(M, N)   338 ms ± 12.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  With Numba: %%timeit N, M = 100, 4000 f(M, N)   12 ms ± 534 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Question: I would like my bot to send message into chat like this: await ctx.send("This country is not supported, you can ask me to add it here")  But to make "here" into clickable link, In HTML I would do it like this, right? <a href="https://www.youtube.com/" > This country is not supported, you can ask me to add it here </a>  How can I do it in python?

Answer: As the other answer explained, you can't add hyperlinks in normal messages, but you can in Embeds. I don't see why you wouldn't want to use an Embed for an error message, especially considering it adds more functionality, so you should consider using that. embed = discord.Embed() embed.description = "This country is not supported, you can ask me to add it [here](your_link_goes_here)." await ctx.send(embed=embed)  Feel free to mess around with the Embed & add some fields, a title, a colour, and whatever else you might want to do to make it look better. More info in the relevant API docs.
Question: Suppose an array a.shape == (N, M) and a vector v.shape == (N,). The goal is to compute argmin of abs of v subtracted from every element of a - that is, out = np.zeros(N, M) for i in range(N): for j in range(M): out[i, j] = np.argmin(np.abs(a[i, j] - v))  I have a vectorized implementation via np.matlib.repmat, and it's much faster, but takes O(M*N^2) memory, unacceptable in practice. Computation's still done on CPU so best bet seems to be implementing the for-loop in C as an extension, but maybe Numpy already has this logic implemented. Does it? Any use-ready Numpy functions implementing above efficiently?

Answer: Inspired by this post, we can leverage np.searchsorted - def find_closest(a, v): sidx = v.argsort() v_s = v[sidx] idx = np.searchsorted(v_s, a) idx[idx==len(v)] = len(v)-1 idx0 = (idx-1).clip(min=0) m = np.abs(a-v_s[idx]) >= np.abs(v_s[idx0]-a) m[idx==0] = 0 idx[m] -= 1 out = sidx[idx] return out  Some more perf. boost with numexpr on large datasets : import numexpr as ne def find_closest_v2(a, v): sidx = v.argsort() v_s = v[sidx] idx = np.searchsorted(v_s, a) idx[idx==len(v)] = len(v)-1 idx0 = (idx-1).clip(min=0) p1 = v_s[idx] p2 = v_s[idx0] m = ne.evaluate('(idx!=0) & (abs(a-p1) >= abs(p2-a))', {'p1':p1, 'p2':p2, 'idx':idx}) idx[m] -= 1 out = sidx[idx] return out  Timings Setup : N,M = 500,100000 a = np.random.rand(N,M) v = np.random.rand(N) In [22]: %timeit find_closest_v2(a, v) 4.35 s ± 21.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) In [23]: %timeit find_closest(a, v) 4.69 s ± 173 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
Question: I have a list like this: list_1 = [0, 1, 1, 0, 0, 0, 1, 1, 1, 1, 0, 1]  How can I calculate the size of blocks of values of 1 and 0 in this list? The resulting list will look like : list_2 = [1, 2, 2, 3, 3, 3, 4, 4, 4, 4, 1, 1] 

Answer: Try with cumsum with diff then transform count s = pd.Series(list_1) s.groupby(s.diff().ne(0).cumsum()).transform('count') Out[91]: 0 1 1 2 2 2 3 3 4 3 5 3 6 4 7 4 8 4 9 4 10 1 11 1 dtype: int64
Question: I am stuck here, but I it's a two part question. Looking at the output of .describe(include = 'all'), not all columns are showing; how do I get all columns to show? This is a common problem that I have all of the time with Spyder, how to have all columns to show in Console. Any help is appreciated. import matplotlib.pyplot as plt import pandas as pd import scipy.stats as stats import seaborn as sns mydata = pd.read_csv("E:\ho11.csv") mydata.head() print(mydata.describe(include="all", exclude = None)) mydata.info()  OUTPUT:  code output 

Answer: Solution You could use either of the following methods: Method-1: source pd.options.display.max_columns = None  Method-2: source pd.set_option('display.max_columns', None) # to reset this pd.reset_option('display.max_columns')  Method-3: source # assuming df is your dataframe pd.set_option('display.max_columns', df.columns.size) # to reset this pd.reset_option('display.max_columns')  Method-4: source # assuming df is your dataframe pd.set_option('max_columns', df.columns.size) # to reset this pd.reset_option('max_columns')  To not wrap the output into multiple lines do this source pd.set_option('display.expand_frame_repr', False)  References I will recommend you to explore the following resources for more details and examples.  How to show all of columns name on pandas dataframe?  How do I expand the output display to see more columns of a pandas DataFrame?  How to show all columns / rows of a Pandas Dataframe?
Question: I want to use this resource below for my cdk app, I using Python for CDK: 'arn:aws:s3:::${LoggingBucket}/AWSLogs/${AWSAccoutID}/*'  Therefore I need to substitute the value of LoggingBucket and AWSAccountID. Here is what I tried: bucket = s3.Bucket(self, "my-bucket", bucket_name = 'my-bucket') core.Fn.sub('arn:aws:s3:::${LoggingBucket}/AWSLogs/${AWSAccoutID}/*',[bucket.bucket_name, core.Environment.account])  But I get this error for line of core.Fn.sub : AttributeError: type object 'property' has no attribute '__jsii_type__' Subprocess exited with error 1  Then I tried this as well: mappings = { 'LoggingBucket': bucket.bucket_name, 'AWSAccountID': core.Environment.account } core.Fn.sub('arn:aws:s3:::${LoggingBucket}/AWSLogs/${AWSAccoutID}/*',mappings)   $ cdk synth  I still getting the same error as above. Question: Please give me a solution on how to use the !sub function in cloudformation in CDK. Let me know what I doing wrong as well. Thank you.

Answer: Since you are using Python (or other programming language) there is no need to use the instrinsic functions that Cloudformation provides. I suggest a more elegant and easy way to format the arn: arn= f'arn:aws:s3:::{bucket.bucket_name}/AWSLogs/{core.Environment.account}/*'
Question: The documentation suggests raising an HTTPException with client errors, which is great. But how can I show those specific errors in the documentation following HTTPException's model? Meaning a dict with the "detail" key. The following does not work because HTTPException is not a Pydantic model. @app.get( '/test', responses={ 409 : { 'model' : HTTPException, 'description': 'This endpoint always raises an error' } } ) def raises_error(): raise HTTPException(409, detail='Error raised') 

Answer: Yes it is not a valid Pydantic type however since you can create your own models, it is easy to create a Model for it. from fastapi import FastAPI from fastapi.exceptions import HTTPException from pydantic import BaseModel class Dummy(BaseModel): name: str class HTTPError(BaseModel): detail: str class Config: schema_extra = { "example": {"detail": "HTTPException raised."}, } app = FastAPI() @app.get( "/test", responses={ 200: {"model": Dummy}, 409: { "model": HTTPError, "description": "This endpoint always raises an error", }, }, ) def raises_error(): raise HTTPException(409, detail="Error raised")  I believe this is what you are expecting
Question: I would like to convert two arrays (x and y) into a frequency n x n matrix (n = 5), indicating each cell the number of point that contains. It consists on resampling both variables into five intervals and count the existing number of points per cell. I have tried using pandas pivot_table but don't know the way of referencing to each axis coordinate. X and Y arrays are two dependent variables that contain values between 0 and 100. I would really appreciate some one's aid. Thank you very much in advance. This is an example of the code: import pandas as pd import numpy as np import matplotlib.pyplot as plt # Arrays example. They are always float type and ranging 0-100. (n_size array = 15) x = 100 * np.random.random(15) y = 100 * np.random.random(15) # Df created for trying to pivot and counting values per cell df = pd.DataFrame({'X':x,'Y':y}) # Plot the example data: df.plot(x = 'X',y = 'Y', style = 'o')  This is what I have:  This is the objetive matrix, saved as a df: 

Answer: If you do not explicitly need to use pandas (which you don't, if it's just about a frequency matrix), consider using numpy.histogram2d: # Sample data x = 100*np.random.random(15) y = 100*np.random.random(15)  Construct your bins (since your x and y bins are the same, one set is enough) bins = np.linspace(0, 100, 5+1) # bins = array([ 0., 20., 40., 60., 80., 100.])  Now use the histogram function: binned, binx, biny = np.histogram2d(x, y, bins = [bins, bins]) # To get the result you desire, transpose objmat = binned.T  Note: x-values are binned along the first dimension(axis 0), which visually means 'vertical'. Hence the transpose. Plotting: fig, ax = plt.subplots() ax.grid() ax.set_xlim(0, 100) ax.set_ylim(0, 100) ax.scatter(x, y) for i in range(objmat.shape[0]): for j in range(objmat.shape[1]): c = int(objmat[::-1][j,i]) ax.text((bins[i]+bins[i+1])/2, (bins[j]+bins[j+1])/2, str(c), fontdict={'fontsize' : 16, 'ha' : 'center', 'va' : 'center'})  Result:
Question: I'm trying to install the MS Python extension (ms-python.python-2020.7.96456.vsix) on a VS Code (1.40.2) install and I'm receiving the following error. "Unable to install extension 'ms-python-python' as it is not compatible with VS Code '1.40.2'". How do I go about finding out what version would be compatible? I'm in an environment where I can't connect to the internet, so the vsix must be used in an offline mode.

Answer: Thanks rioV8! After looking into the package.json I've found that the "engines" field is what details the minimum version of VS Code required. Per code.visualstudio.com (https://code.visualstudio.com/api/working-with-extensions/publishing-extension)  Visual Studio Code compatibility When authoring an extension, you will need to describe what is the extension's compatibility to Visual Studio Code itself. This can be done via the engines.vscode field inside package.json: { "engines": { "vscode": "^1.8.0" } } A value of 1.8.0 means that your extension is compatible only with VS Code 1.8.0. A value of ^1.8.0 means that your extension is compatible with VS Code 1.8.0 and onwards, including 1.8.1, 1.9.0, etc. You can use the engines.vscode field to make sure the extension only gets installed for clients that contain the API you depend on. This mechanism plays well with the Stable release as well as the Insiders one. For example, imagine that the latest Stable version of VS Code is 1.8.0 and that during 1.9.0's development a new API is introduced and thus made available in the Insider release through version 1.9.0-insider. If you want to publish an extension version that benefits from this API, you should indicate a version dependency of ^1.9.0. Your new extension version will be installed only on VS Code greater than or equal to 1.9.0, which means all current Insider customers will get it, while the Stable ones will only get the update when Stable reaches 1.9.0.
Question: I want to export a Pandas DataFrame to LaTeX with . as a thousand seperator and , as a decimal seperator and two decimal digits. E.g. 4.511,34 import numpy as np import pandas as pd df = pd.DataFrame( np.array([[4511.34242, 4842.47565]]), columns=['col_1', 'col_2'] ) df.to_latex('table.tex', float_format="{:0.2f}".format)  Ho can I achieve this? If I change the . to an , in the code I receive ValueError: Invalid format specifier. Thank you!

Answer: I would format with _ as the thousands seperator and . as the decimal seperator and then replace those with str.replace. df.applymap(lambda x: str.format("{:0_.2f}", x).replace('.', ',').replace('_', '.')).to_latex('table.tex')  Gives the following latex: \begin{tabular}{lll} \toprule {} & col\_1 & col\_2 \\ \midrule 0 & 4.511,34 & 4.842,48 \\ \bottomrule \end{tabular}
Question: I am trying to find version of pandas: def check_library_version(): print("Checking library version") subprocess.run(f'bash -c "conda activate {ENV_NAME};"', shell=True) import pandas pandas.__version__  Desired output: 1.1.3 Output:  Checking library version CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'. To initialize your shell, run $ conda init <SHELL_NAME>  Currently supported shells are:  bash fish tcsh xonsh zsh powershell  See 'conda init --help' for more information and options. IMPORTANT: You may need to close and restart your shell after running 'conda init'.  To clarify, I don't seek to update the environment of the currently running script; I just want to briefly activate that environment and find out which Pandas version is installed there.

Answer: This doesn't make any sense at all; the Conda environment you activated is terminated when the subprocess terminates. You should (conda init and) conda activate your virtual environment before you run any Python code. If you just want to activate, run a simple Python script as a subprocess of your current Python, and then proceed with the current script outside of the virtual environment, try something like subprocess.run(f"""conda init bash conda activate {ENV_NAME} python -c 'import pandas; print(pandas.__version__)'""", shell=True, executable='/bin/bash', check=True)  This just prints the output to the user; if your Python program wants to receive it, you need to add the correct flags; check = subprocess.run(...whatever..., text=True, capture_output=True) pandas_version = check.stdout  (It is unfortunate that there is no conda init sh; I don't think anything in the above depends on executable='/bin/bash' otherwise. Perhaps there is a way to run this in POSIX sh and drop the Bash requirement.)
Question: I have the following table:  year pop1 pop2 0 0 100000 100000 1 1 999000 850000 2 2 860000 700000 3 3 770000 650000  I want to find for each pop (pop1 ,pop2) the year the pop was closest to a given number, for example, the year the pop was the closest to 830000. Is there any way to find the nearest value inside column based on given value? I have seen this post (How do I find the closest values in a Pandas series to an input number?_ but seems like here the result is above and below and I wat to get in the end only one number. *I don't have code example because I don't find any argument to use to get the nearest

Answer: Convert column year to index, then subtract value, get absolute values and last index (here year) by nearest value - here minimal by DataFrame.idxmin: val = 830000 s = df.set_index('year').sub(val).abs().idxmin() print (s) pop1 2 pop2 1 dtype: int64
Question: I tried to set variable types in my functions. There is no problem when I tried to use normal variable type. For example, def myString(name:str) -> str: return "hello " + name  However, I got problem in list. Many examples in internet said use List, but it got error. Now I use list, and there is no error. Is it ok to use this? Another problem that I found someone can use def myListString() -> list[str]: return ["ABC", "CDE"]  I found error. TypeError: 'type' object is not subscriptable  How should I correct this? Another problem that I found is I cannot declare myClass in the myClass. For example, class Point: def __init__(self, x:int, y:int): self.x:int = x self.y:int = y def isSamePoint(self, p:Point) -> bool: return ((self.x==p.x) and (self.y==p.y)) p0 = Point(10, 5) p1 = Point(5, 5) p0.isSamePoint(p1)  I found error, def isSamePoint(self, p:Point): NameError: name 'Point' is not defined  Please help me solve the problem.

Answer:  TypeError: 'type' object is not subscriptable  Python 3.9 allows for list[str]. Earlier you had to import List from typing and do -> List[str].  NameError: name 'Point' is not defined  If you want to declare the type of "self" you can either put that in a string def isSamePoint(self, p: "Point") -> bool: or create an interface. >>> class A: pass >>> class B(A): pass >>> b = B() >>> isinstance(b, A) True  so def isSamePoint(self, p: A) would do the trick. Also, if you want to check if isSamePoint you might want to consider your own __eq__.
Question: I've created a class and trying to assign one of its values to something that expects a string, however it is saying it is getting a Tuple[str] instead, and I don't see how? from azure.identity import ClientSecretCredential class ServicePrincipal: """ Service Principal class is used to authorise the service """ def __init__(self): self.tenant_id = "123-xyz", self.client_id = "123-abc", self.client_secret = "123-lmn", def credentials(self): """ Returns a ClientServiceCredential object using service principal details :return: """ # ISSUE IS HERE return ClientSecretCredential( tenant_id=self.tenant_id, # <---- Getting Tuple[str] client_id=self.client_id, # <---- Getting Tuple[str] client_secret=self.client_secret, # <---- Getting Tuple[str] )  if I copy paste the string directly into the parameter its fine. So the self.value is causing an issue somehow?

Answer: You should remove the commas here: def __init__(self): self.tenant_id = "123-xyz", # remove the comma self.client_id = "123-abc", # remove the comma self.client_secret = "123-lmn", # remove the comma  Comma make the variable be a Tuple
Question: I'm trying to run a SQL statement to be rendered by Airflow, but I'm also trying to include a variable inside the statement which is passed in from Python. The SQL statement is just a where clause, and after the WHERE, I'm trying to add a datetime, minus a few seconds: f" ' {{ ts - macros.timedelta(seconds={lower_delay} + 1) }} ' "  so I want what's in the double curly braces to be computed and rendered in Airflow, but I want to pass in this variable called lower_delay before it does that. I've tried varying combinations with zero, one, or two additional curly braces around lower_delay as well as the entire string, but I seem to get a different error each time. What is the proper way to pass in this lower_delay variable (it's just a number) so that it ends up rendering correctly?

Answer: Jinja templating requires two curly braces, when you use f-strings or str.format it will replace two braces with one while rendering:  Format strings contain “replacement fields” surrounded by curly braces {}. Anything that is not contained in braces is considered literal text, which is copied unchanged to the output. If you need to include a brace character in the literal text, it can be escaped by doubling: {{ and }}.  So the following should work: f" ' {{{{ ts - macros.timedelta(seconds={lower_delay} + 1) }}}} ' "
Question: Like many command line tools, mine accepts optional filenames. Argparse seems to support this via nargs='*', which is working for me as expected: import argparse parser = argparse.ArgumentParser() parser.add_argument( 'files', help='file(s) to parse instead of stdin', nargs='*') parser.parse_args()  However, the help output is bizarre: $ ./help.py -h usage: help.py [-h] [files [files ...]]  How can I avoid the nested optional and repeated parameter name? The repetition adds no information beyond [files ...], which is the traditional way optional parameter lists are indicated on Unix: $ grep --help usage: grep [-abcDEFGHhIiJLlmnOoqRSsUVvwxZ] [-A num] [-B num] [-C[num]] [-e pattern] [-f file] [--binary-files=value] [--color=when] [--context[=num]] [--directories=action] [--label] [--line-buffered] [--null] [pattern] [file ...] $ ls --help Usage: exa [options] [files...] $ vim --help Usage: nvim [options] [file ...] Edit file(s)  Any help is appreciated. I'm trying argparse because using it seems to be a Python best practice, but this help output is a dealbreaker for me.

Answer: This was fixed in Python 3.9, see https://bugs.python.org/issue38438 and commit a0ed99bc that fixed it. Your code produces the usage message you expect if run on 3.9: Python 3.9.0 (default, Oct 12 2020, 02:44:01) [GCC 9.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import argparse >>> parser = argparse.ArgumentParser() >>> parser.add_argument('files', help='file(s) to parse instead of stdin', nargs='*') _StoreAction(option_strings=[], dest='files', nargs='*', const=None, default=None, type=None, choices=None, help='file(s) to parse instead of stdin', metavar=None) >>> parser.print_help() usage: [-h] [files ...]