eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: How can I get the name of a route/endpoint using FastAPI/Starlette? I have access to the Request object and I need this information in one of my middlewares. For example, if I hit services/1, I should then be able to get the abc name. Is this possible in FastAPI? @app.get("/services/{service}", name="abc") async def list_services() -> dict: do something  Update 1: Output of request.scope {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.3'}, 'http_version': '1.1', 'server': ('127.0.0.1', 8001), 'client': ('127.0.0.1', 56670), 'scheme': 'http', 'root_path': '', 'headers': [(b'user-agent', b'PostmanRuntime/7.29.2'), (b'accept', b'*/*'), (b'postman-token', b'f2da2d0f-e721-44c8-b14f-e19750ea8a68'), (b'host', b'localhost:8001'), (b'accept-encoding', b'gzip, deflate, br'), (b'connection', b'keep-alive')], 'method': 'GET', 'path': '/health', 'raw_path': b'/health', 'query_string': b'', 'app': <fastapi.applications.FastAPI object at 0x1036d5790>}  Update 2: Providing middleware code where request.scope["route"] is breaking. from fastapi import FastAPI,Request app = FastAPI() @app.middleware("http") async def logging_middleware(request: Request, call_next): print(request.scope['route'].name) response = await call_next(request) return response @app.get('/', name='abc') def get_name(request: Request): return request.scope['route'].name 

Answer: Option 1 You can get the name value inside an endpoint as follows: from fastapi import FastAPI,Request app = FastAPI() @app.get('/', name='abc') def get_name(request: Request): return request.scope['route'].name  Option 2 Inside a middleware, make sure to get the route's name after calling call_next(request), otherwise you would be faced with KeyError: 'route', as the route key/object would not yet exist in the scope dictionary. Example: @app.middleware("http") async def some_middleware(request: Request, call_next): response = await call_next(request) print(request.scope['route'].name) return response  Option 3 Instead of a middleware, you could create a custom APIRoute class, which would allow you to get the route's name before processing the request and getting the response (if that's a requirement for your app). You could add an endpoint that you would like to be handled by that APIRoute class using @<name_of_router_instance> instead of @app (e.g., @router.get('/', name='abc')). More than one endpoint could be added in the same way. Example: from fastapi import APIRouter, FastAPI, Request, Response from typing import Callable from fastapi.routing import APIRoute class CheckNameRoute(APIRoute): def get_route_handler(self) -> Callable: original_route_handler = super().get_route_handler() async def custom_route_handler(request: Request) -> Response: print(request.scope['route'].name) response = await original_route_handler(request) return response return custom_route_handler app = FastAPI() router = APIRouter(route_class=CheckNameRoute) @router.get('/', name='abc') def get_name(request: Request): return request.scope['route'].name app.include_router(router)   To get the route path of an endpoint instead, please see this answer.
Question: Let's say I have df = pl.DataFrame({ "date": pl.Series(["2022-01-01", "2022-01-02"]).cast(pl.Date) })  How do I localize that to a specific timezone and make it a datetime? I tried: df.select(pl.col('date').cast(pl.Datetime(time_zone='America/New_York')))  but that gives me shape: (2, 1) ┌────────────────────────────────┐ │ date │ │ --- │ │ datetime[μs, America/New_York] │ ╞════════════════════════════════╡ │ 2021-12-31 19:00:00 EST │ │ 2022-01-01 19:00:00 EST │ └────────────────────────────────┘  so it looks like it's starting from the presumption that the naïve datetimes are UTC and then applying the conversion. I set os.environ['TZ']='America/New_York' but I got the same result. I looked through the polars config options in the API guide to see if there's something else to set but couldn't find anything about default timezone.

Answer: As of polars 0.16.3, you can do: df.select( pl.col('date').cast(pl.Datetime).dt.replace_time_zone("America/New_York") )
Question: I have a python polars dataframe as- df_pol = pl.DataFrame({'test_names':[['Mallesham','','Bhavik','Jagarini','Jose','Fernando'], ['','','','ABC','','XYZ']]})  I would like to get a count of elements from each list in test_names field not considering the empty values. df_pol.with_columns(pl.col('test_names').list.len().alias('tot_names'))   Here it is considering empty strings into count, this is why we can see 6 names in list-2. actually it has only two names. required output as: 

Answer: You can use list.eval to run any polars expression on the list's elements. In an list.eval expression, you can pl.element() to refer to the lists element and then apply an expression. Next we simply use a filter expression to prune the values we don't need. df = pl.DataFrame({ "test_names":[ ["Mallesham","","Bhavik","Jagarini","Jose","Fernando"], ["","","","ABC","","XYZ"] ] }) df.with_columns( pl.col("test_names").list.eval(pl.element().filter(pl.element() != "")) )  shape: (2, 1) ┌─────────────────────────────────────┐ │ test_names │ │ --- │ │ list[str] │ ╞═════════════════════════════════════╡ │ ["Mallesham", "Bhavik", ... "Fer... │ │ ["ABC", "XYZ"] │ └─────────────────────────────────────┘
Question: I have a multipart partitioned parquet on s3. Each partition contains multiple parquet files. The below code narrows in on a single partition which may contain somewhere around 30 parquet files. When I use scan_parquet on a s3 address that includes *.parquet wildcard, it only looks at the first file in the partition. I verified this with the count of customers. It has the count from just the first file in the partition. Is there a way that it can scan across files? import polars as pl s3_loc = "s3://some_bucket/some_parquet/some_partion=123/*.parquet" df = pl.scan_parquet(s3_loc) cus_count = df.select(pl.count('customers')).collect()  If I leave off the *.parquet from the s3 address then I get the following error. exceptions.ArrowErrorException: ExternalFormat("File out of specification: A parquet file must containt a header and footer with at least 12 bytes")

Answer: New Answer polars can natively load files from AWS, Azure, GCP, or plain old http and no longer uses fsspec (very much, if at all). Instead, it uses the object_store under the hood. The syntax to use it is. pl.scan_parquet( "s3://some_bucket/some_parquet/some_partion=123/*.parquet", storage_options= dict_of_credentials)  The dict_of_credentials might be a bit different than what you normally feed to initiate an s3fs.S3FileSystem(). Additionally, you can exclude storage_options if you've got your credentials as env variables but the env variables that work for s3fs might not be exactly what object_store is looking for. For instance, this shows what env variables it will look for. Old Answer using pyarrow.dataset It looks like from the user guide on multiple files that to do so requires a loop creating many lazy dfs that you then combine together. Another approach is to use the scan_ds function which takes a pyarrow dataset object. import polars as pl import s3fs import pyarrow.dataset as ds fs = s3fs.S3FileSystem() # you can also make a file system with anything fsspec supports # S3FileSystem is just a wrapper for fsspec s3_loc = "s3://some_bucket/some_parquet/some_partion=123" myds = ds.dataset(s3_loc, filesystem=fs) lazy_df = pl.scan_pyarrow_dataset(myds) cus_count = lazy_df.select(pl.count('customers')).collect()
Question: The application I deploy is FastAPI with Uvicorn under K8s. While trying to understand how I want to Dockerize the application I understood I want to implement Uvicorn without Gunicorn and to add a system of scale up/down by the load of the requests the application is getting. I did a lot of load testing and discovered that with the default of 1 Uvicorn worker I'm getting 3.5 RPS, while changing the workers to 8 I can get easly 22 RPS (didn't check for more since its great results for me). Now what I was expecting regarding the resources is that the CPU that I will have to provide will be with a limit of 8 (I assume every worker works on one process and thread), but I saw only increase in the memory usage, but barley in the CPU. maybe its because the app don't use that much CPU but indeed its possible for it to use more than 1 CPU? so far it didn't used more than one CPU. How do Uvicorn workers work? How should I calculate how many workers I need for the app? I didn't find any useful information. Again, my goal is to keep a slim machine of 1 cpu, with Autoscaling system.  

Answer: When using uvicorn and applying the --workers argument greater than 1, then uvicorn will spawn subprocesses internally using multiprocessing. You have to remember that uvicorn is asynchronous and that HTTP servers generally are bottle necked by network latency instead of computation. So, it could be that your workloads aren't particularly CPU bound and are IO bound. Without knowing more about the type of work being done by the server on each request, the best way to determine how many workers you will need will be through empirical experimentation. In other words, just test it until you hit a limit. Though the FastAPI documentation does include some guidance for your use case:  If you have a cluster of machines with Kubernetes, Docker Swarm Mode, Nomad, or another similar complex system to manage distributed containers on multiple machines, then you will probably want to handle replication at the cluster level instead of using a process manager (like Gunicorn with workers) in each container. One of those distributed container management systems like Kubernetes normally has some integrated way of handling replication of containers while still supporting load balancing for the incoming requests. All at the cluster level. In those cases, you would probably want to build a Docker image from scratch as explained above, installing your dependencies, and running a single Uvicorn process instead of running something like Gunicorn with Uvicorn workers. - FastAPI Docs  Emphasis mine.
Question: I am trying to run the following pandas code to create a df by reading an excel. However I receive the error below. (I pip-installed the openpyxl but I get the same error.) import pandas as pd import numpy as np df = pd.read_excel("test.xlsx") return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1030, in _gcd_import File "<frozen importlib._bootstrap>", line 1007, in _find_and_load File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked ModuleNotFoundError: No module named 'openpyxl' ImportError: Missing optional dependency 'openpyxl'. Use pip or conda to install openpyxl. 

Answer: Jupyter (with anaconda) is using a specific python environment independent from the local python installation in your computer. First make sure your are installing the packages in the correct interpreter if you want to install it into anaconda (jUPYTER NOTEBOOK) try: conda activate pip install openpyxl  otherwise just make sure you don't have diferent versions of python 3 or just install it to the specific version you are using. For example if I'm running the code with python 3.9 I'll run the code below: python3.9 -m pip install openpyxl
Question: I am trying to implement SAC with a custom environment in Stable Baselines3 and I keep getting the error in the title. The error occurs with any off policy algorithm not just SAC. Traceback: File "<MY PROJECT PATH>\src\main.py", line 70, in <module> main() File "<MY PROJECT PATH>\src\main.py", line 66, in main model.learn(total_timesteps=timesteps, reset_num_timesteps=False, tb_log_name=f"sac_{num_cars}_cars") File "<MY PROJECT PATH>\venv\lib\site-packages\stable_baselines3\sac\sac.py", line 309, in learn return super().learn( File "<MY PROJECT PATH>\venv\lib\site-packages\stable_baselines3\common\off_policy_algorithm.py", line 375, in learn self.train(batch_size=self.batch_size, gradient_steps=gradient_steps) File "<MY PROJECT PATH>\venv\lib\site-packages\stable_baselines3\sac\sac.py", line 256, in train current_q_values = self.critic(replay_data.observations, replay_data.actions) File "<MY PROJECT PATH>\venv\lib\site-packages\torch\nn\modules\module.py", line 1190, in _call_impl return forward_call(*input, **kwargs) File "<MY PROJECT PATH>\venv\lib\site-packages\stable_baselines3\common\policies.py", line 885, in forward return tuple(q_net(qvalue_input) for q_net in self.q_networks) File "<MY PROJECT PATH>\venv\lib\site-packages\stable_baselines3\common\policies.py", line 885, in <genexpr> return tuple(q_net(qvalue_input) for q_net in self.q_networks) File "<MY PROJECT PATH>\venv\lib\site-packages\torch\nn\modules\module.py", line 1190, in _call_impl return forward_call(*input, **kwargs) File "<MY PROJECT PATH>\venv\lib\site-packages\torch\nn\modules\container.py", line 204, in forward input = module(input) File "<MY PROJECT PATH>\venv\lib\site-packages\torch\nn\modules\module.py", line 1190, in _call_impl return forward_call(*input, **kwargs) File "<MY PROJECT PATH>\venv\lib\site-packages\torch\nn\modules\linear.py", line 114, in forward return F.linear(input, self.weight, self.bias) RuntimeError: mat1 and mat2 must have the same dtype  Action and observation spaces: self.action_space = Box(low=-1., high=1., shape=(2,), dtype=np.float) self.observation_space = Box( np.array( [-np.inf] * (9 * 40) + [-np.inf] * 3 + [-np.inf] * 3 + [-np.inf] * 3 + [0.] + [0.] + [0.] + [-1.] + [0.] * 4 + [0.] * 4 + [0.] * 4, dtype=np.float ), np.array( [np.inf] * (9 * 40) + [np.inf] * 3 + [np.inf] * 3 + [np.inf] * 3 + [np.inf] + [1.] + [1.] + [1.] + [1.] * 4 + [np.inf] * 4 + [np.inf] * 4, dtype=np.float ), dtype=np.float )  Observations are returned in the step and reset methods as a numpy array of floats. Is there something I'm missing which is causing this error? If I use one of the environments that come with gym such as pendulum it works fine which is why I think I have a problem with my custom environment. Thanks in advance for any help and please let me know if more information is required.

Answer: Change the inputs to float32 , default the loader set the type as float64. inputs = inputs.to(torch.float32)
Question: I would like to ignore the default values after calling asdict() @dataclass class A: a: str b: bool = True  so if I call a = A("1") result = asdict(a, ignore_default=True) assert {"a": "1"} == result # the "b": True should be deleted 

Answer: The dataclasses module doesn't appear to have support for detecting default values in asdict(), however the dataclass-wizard library does -- via skip_defaults argument. Example: from dataclasses import dataclass from dataclass_wizard import asdict @dataclass class A: a: str b: bool = True a = A("1") result = asdict(a, skip_defaults=True) assert {"a": "1"} == result # the "b": True should be deleted  Further, results show it is close to 2x faster than an approach with dataclasses.adict(). I've added benchmark code I used for testing below. from dataclasses import dataclass, asdict as asdict_orig, MISSING from timeit import timeit from dataclass_wizard import asdict @dataclass class A: a: str b: bool = True def asdict_factory(cls): def factory(obj: list[tuple]) -> dict: d = {} for k, v in obj: field_value = cls.__dataclass_fields__[k].default if field_value is MISSING or field_value != v: d[k] = v return d return factory a = A("1") A_fact = asdict_factory(A) print('dataclass_wizard.asdict(): ', timeit('asdict(a, skip_defaults=True)', globals=globals())) print('dataclasses.asdict(): ', timeit('asdict_orig(a, dict_factory=A_fact)', globals=globals())) result1 = asdict(a, skip_defaults=True) result2 = asdict_orig(a, dict_factory=A_fact) assert {"a": "1"} == result1 == result2 a2 = A("1", True) a3 = A("1", False) assert asdict(a2, skip_defaults=True) == asdict_orig(a2, dict_factory=A_fact) assert asdict(a3, skip_defaults=True) == asdict_orig(a3, dict_factory=A_fact)   Disclaimer: I am the creator and maintainer of this library.
Question: I need to locate the mouse click location on an image in a google colab notebook. I tried the following script but nothing happened. The following code should work in Jupyter notebooks but it doesn't work on google colab: import matplotlib matplotlib.use('TKAgg') import matplotlib.pyplot as plt import matplotlib.image as mpimg f1 = 'sample.tif' fig = plt.figure(figsize=(20,30)) img = mpimg.imread(f1) def onclick(event): ix, iy = event.xdata, event.ydata print(ix, iy) cid = fig.canvas.mpl_connect('button_press_event', onclick) imgplot = plt.imshow(img) plt.show() 

Answer: You need to use an interactive IPython backend, e.g. ipympl:  Installation in Colab:  !pip install ipympl from google.colab import output output.enable_custom_widget_manager()   setup matplotlib to use it:  %matplotlib ipympl   test it:  import matplotlib import matplotlib.pyplot as plt fig, ax = plt.subplots() def onclick(event): ix, iy = event.xdata, event.ydata print(ix, iy) cid = fig.canvas.mpl_connect('button_press_event', onclick)  Output:
Question: I have created this basic stored procedure to query a Snowflake table based on a customer id: CREATE OR REPLACE PROCEDURE SP_Snowpark_Python_Revenue_2(site_id STRING) RETURNS STRING LANGUAGE PYTHON RUNTIME_VERSION = '3.8' PACKAGES = ('snowflake-snowpark-python') HANDLER = 'run' AS $$ from snowflake.snowpark.functions import * def run(session, site_id): df_rev_tmp = session.table("revenue").select(col("site_id"), col("subscription_id"), col("country_name"), col("product_name")) df_rev_final = df_rev_tmp.filter(col("site_id") == site_id) return "SUCCESS" $$;  It works fine but I would like my sproc to return a JSON object for the whole result set. I modified it thusly: CREATE OR REPLACE PROCEDURE SP_Snowpark_Python_Revenue_3(site_id STRING) RETURNS STRING LANGUAGE PYTHON RUNTIME_VERSION = '3.8' PACKAGES = ('snowflake-snowpark-python') HANDLER = 'run' AS $$ from snowflake.snowpark.functions import * def run(session, site_id): df_rev = session.table("revenue").select(col("site_id"), col("subscription_id"), col("country_name"), col("product_name")) df_rev_tmp = df_rev.filter(col("site_id") == site_id) df_rev_final = df_rev_tmp.to_pandas() df_rev_json = df_rev_final.to_json(orient = 'columns') return df_rev_json $$;  It compiles without errors but fails at runtime with this error: CALL SP_Snowpark_Python_Revenue_3('dfgerr6223')..... 255002: Optional dependency: 'pyarrow' is not installed...  What am I missing here?

Answer: You need to ask for pyarrow as a package: PACKAGES = ('snowflake-snowpark-python', 'pyarrow')  But to get these packages, someone in your org will need to approve the Anaconda terms of service, or you'll get the following error: SQL compilation error: Anaconda terms must be accepted by ORGADMIN to use Anaconda 3rd party packages. Please follow the instructions at https://…  Someone with ORGADMIN role can follow these steps:  https://docs.snowflake.com/en/sql-reference/stored-procedures-python.html#getting-started
Question: I am working on a small program in which the user enters text and I would like to check how many times the given words occur in the given input. # Read user input print("Input your code: \n") user_input = sys.stdin.read() print(user_input)  For example, the text that I input in a program is: a=1 b=3 if (a == 1): print("A is a number 1") elif(b == 3): print ("B is 3") else: print("A isn't 1 and B isn't 3")  The words to find out are specified in an array. wordsToFind = ["if", "elif", "else", "for", "while"]  And basically I would like to print how many "if", "elif" and "else" has occurred in a input. How can I count occurrences of words like "if", "elif", "else", "for", "while" in a given string by user input?

Answer: I think the best option is to use the tokenize built-in module of python: # Let's say this is tokens.py import sys from collections import Counter from io import BytesIO from tokenize import tokenize # Get input from stdin code_text = sys.stdin.read() # Tokenize the input as python code tokens = tokenize(BytesIO(code_text.encode("utf-8")).readline) # Filter the ones in wordsToFind wordsToFind = ["if", "elif", "else", "for", "while"] words = [token.string for token in tokens if token.string in wordsToFind] # Count the occurrences counter = Counter(words) print(counter)  Test Let's say you have a test.py: a=1 b=3 if (a == 1): print("A is a number 1") elif(b == 3): print ("B is 3") else: print("A isn't 1 and B isn't 3")  and then you run: cat test.py | python tokens.py  Output: Counter({'if': 1, 'elif': 1, 'else': 1})  Advantages  Only correct python (syntactically) will be parsed  You only will be counting the python keywords (not every if occurrence in the code text, for example, you can have an line like a = "if inside str" That if should not be counted I think
Question: Using Regex in Python (library re (only)), I want to create a function that gives me the position of all leading 0s in a string. For example, if the string was: My house has 01 garden and 003 rooms. I would want me the function to return 13, 27 and 28. I tried for example: import re string = "My house has 01 garden and 003 rooms." pattern = "(0+)[1-9]\d*" print(re.findall(pattern,string))  Obviously, the output gives me the matches but no position...

Answer: You can do the following: import re text = "My house has 01 garden and 003 rooms." pattern = re.compile(r"\b0+") def leading_zeros_index(s: str) -> list: return [i for m in pattern.finditer(s) for i in range(m.start(), m.end())] print(leading_zeros_index(text))  output: [13, 27, 28]  Basically you use .finditer() in order to get the match object, then you create a range() object from match object's .start() and .end(). I used \b0+ as the pattern. There is no need to check the other characters come after zeros. \b is word boundary, here means, zeros should be at the start of the words.
Question: I wrote the following code but when the rows are large it is slow import numpy as np array = np.array([ [1,2],[1,2],[2,3], [1,2],[2,3],[5,2]]) d={} for l in array: t = tuple(l) if t in d: d[t]+=1 else: d[t]=1 print(d)  result: `{(1, 2): 3, (2, 3): 2, (5, 2): 1}`  Is there a faster way to do this?

Answer: Use np.unique elements, counts = np.unique(array, axis=0, return_counts=True)  In your case, elements will be [[1, 2], [2, 3], [5, 2]]  and counts will be [3, 2, 1]
Question: I have a crosstab table with 4 rows and multiple columns, containing numeral values (representing the number of dataset elements on the crossing of two factors). I want to sort the order of columns in the crosstab by the sum of values in each column. e.g. I have: ct = pd.crosstab(df_flt_reg['experience'], df_flt_reg['region']) | a| b| c| d| e| 0 | 1| 0| 7| 3| 6| 1 | 2| 4| 1| 5| 4| 2 | 3| 5| 0| 7| 2| 3 | 1| 3| 1| 9| 1| (sum)| 7| 12| 9| 24| 13| # row doesn't exist, written here to make clear the logic  What do I want:  | d| e| b| c| a| 0 | 3| 6| 0| 7| 1| 1 | 5| 4| 4| 1| 2| 2 | 7| 2| 5| 0| 3| 3 | 9| 1| 3| 1| 1| (sum)| 24| 13| 12| 9| 7|  I succeded only in sorting the columns by their names (in alphabet order), but that's not what I need. I tried to sum those values separately, made a list of properly ordered indexes and then addressed them to crosstab.sort_values() via "by" parameter, but it was sorted in alphabet order again. Also I tried to create a new row "sum", but succeded to create only a new column -_- So I humbly asking for the community's help.

Answer: Calculate the sum and sort the values. Once you have the sorted series get the index and reorder your columns with it. sorted_df = ct[ct.sum().sort_values(ascending=False).index] d e b c a 0 3 6 0 7 1 1 5 4 4 1 2 2 7 2 5 0 3 3 9 1 3 1 1