eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Goal: store a dict() or {} as the value for a key-value pair, to set() onto Redis. Code import redis r = redis.Redis() value = 180 my_dict = dict(bar=value) r.set('foo', my_dict)  redis.exceptions.DataError: Invalid input of type: 'dict'. Convert to a bytes, string, int or float first. 

Answer: You cannot pass a dictionary object as a value in the set() operation to Redis. However, we can use either pickle or json to get the Bytes of an object. Whichever you already have imported would be optimal, imho.  Pickle Serialize to pickle (with pickle.dumps) pre-set() import pickle my_dict = {'a': 1, 'b': 2} dict_bytes = pickle.dumps(my_dict) r.set('my_key', dict_bytes)  Deserialize the object (dict) (with pickle.loads) post-get(): dict_bytes = r.get('my_key') my_dict = pickle.loads(dict_bytes)   JSON string Serialize to JSON string (with json.dumps) pre-set() import json my_dict = {'a': 1, 'b': 2} dict_str = json.dumps(my_dict) r.set('my_key', dict_str)  Deserialize the object (dict) (with json.loads) post-get(): dict_str = r.get('my_key') my_dict = json.loads(dict_str)
Question: I have been using polars but it seems like it lacks qcut functionality as pandas do. I am not sure about the reason but is it possible to achieve the same effect as pandas qcut using current available polars functionalities? The following shows an example about what I can do with pandas qcut. import pandas as pd data = pd.Series([11, 1, 2, 2, 3, 4, 5, 1, 2, 3, 4, 5]) pd.qcut(data, [0, 0.2, 0.4, 0.6, 0.8, 1], labels=['q1', 'q2', 'q3', 'q4', 'q5'])  The results are as follows: 0 q5 1 q1 2 q1 3 q1 4 q3 5 q4 6 q5 7 q1 8 q1 9 q3 10 q4 11 q5 dtype: category  So, I am curious how can I get the same result by using polars? Thanks for your help.

Answer: Update: Series.qcut was added in polars version 0.16.15 data = pl.Series([11, 1, 2, 2, 3, 4, 5, 1, 2, 3, 4, 5]) data.qcut([0.2, 0.4, 0.6, 0.8], labels=['q1', 'q2', 'q3', 'q4', 'q5'], maintain_order=True)  shape: (12, 3) ┌──────┬─────────────┬──────────┐ │ ┆ break_point ┆ category │ │ --- ┆ --- ┆ --- │ │ f64 ┆ f64 ┆ cat │ ╞══════╪═════════════╪══════════╡ │ 11.0 ┆ inf ┆ q5 │ │ 1.0 ┆ 2.0 ┆ q1 │ │ 2.0 ┆ 2.0 ┆ q1 │ │ 2.0 ┆ 2.0 ┆ q1 │ │ … ┆ … ┆ … │ │ 2.0 ┆ 2.0 ┆ q1 │ │ 3.0 ┆ 3.6 ┆ q3 │ │ 4.0 ┆ 4.8 ┆ q4 │ │ 5.0 ┆ inf ┆ q5 │ └──────┴─────────────┴──────────┘   Old answer: From what I can tell .qcut() uses the linear quantile of the bin values? If so, you could implement that part "manually": import polars as pl data = pl.Series([11, 1, 2, 2, 3, 4, 5, 1, 2, 3, 4, 5]) bins = [0.2, 0.4, 0.6, 0.8] labels = ["q1", "q2", "q3", "q4", "q5"] pl.cut(data, bins=[data.quantile(val, "linear") for val in bins], labels=labels)  shape: (12, 3) ┌──────┬─────────────┬──────────┐ │ | break_point | category │ │ --- | --- | --- │ │ f64 | f64 | cat │ ╞══════╪═════════════╪══════════╡ │ 1.0 | 2.0 | q1 │ │ 1.0 | 2.0 | q1 │ │ 2.0 | 2.0 | q1 │ │ 2.0 | 2.0 | q1 │ │ 2.0 | 2.0 | q1 │ │ 3.0 | 3.6 | q3 │ │ 3.0 | 3.6 | q3 │ │ 4.0 | 4.8 | q4 │ │ 4.0 | 4.8 | q4 │ │ 5.0 | inf | q5 │ │ 5.0 | inf | q5 │ │ 11.0 | inf | q5 │ └──────┴─────────────┴──────────┘
Question: after updating pip and setuptools==66.0, pip stopped working and responds to all attempts to call it like this:  Traceback (most recent call last): File "/usr/bin/pip", line 6, in <module> from pkg_resources import load_entry_point File "/usr/local/lib/python3.8/dist-packages/pkg_resources/__init__.py", line 3249, in <module> def _initialize_master_working_set(): File "/usr/local/lib/python3.8/dist-packages/pkg_resources/__init__.py", line 3223, in _call_aside f(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/pkg_resources/__init__.py", line 3261, in _initialize_master_working_set working_set = WorkingSet._build_master() File "/usr/local/lib/python3.8/dist-packages/pkg_resources/__init__.py", line 619, in _build_master return cls._build_from_requirements(__requires__) File "/usr/local/lib/python3.8/dist-packages/pkg_resources/__init__.py", line 632, in _build_from_requirements dists = ws.resolve(reqs, Environment()) File "/usr/local/lib/python3.8/dist-packages/pkg_resources/__init__.py", line 1044, in __init__ self.scan(search_path) File "/usr/local/lib/python3.8/dist-packages/pkg_resources/__init__.py", line 1077, in scan self.add(dist) File "/usr/local/lib/python3.8/dist-packages/pkg_resources/__init__.py", line 1096, in add dists.sort(key=operator.attrgetter('hashcmp'), reverse=True) File "/usr/local/lib/python3.8/dist-packages/pkg_resources/__init__.py", line 2631, in hashcmp self.parsed_version, File "/usr/local/lib/python3.8/dist-packages/pkg_resources/__init__.py", line 2685, in parsed_version raise packaging.version.InvalidVersion(f"{str(ex)} {info}") from None pkg_resources.extern.packaging.version.InvalidVersion: Invalid version: '0.23ubu ntu1' (package: distro-info)  thanks

Answer: This turns out to be by design by the setuptools developers, please see some commentary on it at https://github.com/pypa/setuptools/issues/3772#issuecomment-1384342813. In short, it's by been removed to enforce strict semantic versioning of packages. If you can't upgrade the versioning to be semantic versioning compatible, please pin setuptools to version < 66. Here is Python's versioning specification: https://peps.python.org/pep-0440/
Question: I'm using Treesitter with Neovim v0.8.2 and Python. With a default configuration of those 3, python docstrings are highlighted as strings, and I'd like to highlight them as comments. I've tried creating a ~/.config/nvim/after/syntax/python.vim file with syn region Comment start=/"""/ end=/"""/  and I expected """<things here>""" to be highlighted as comments. I'm guessing this is because treesitter is disabling syntax highlighting, so on that note has anyone been able to add custom highlighting rules to Treesitter or after it?

Answer: checkout https://github.com/nvim-treesitter/nvim-treesitter/issues/4392 it's quite the read, but you can override how treesitter parses the objects in the document/buffer, can't take credit for the below solution, but should get you where you need to be. after/queries/python/highlights.scm   extends ; Module docstring (module . (expression_statement (string) @comment)) ; Class docstring (class_definition body: (block . (expression_statement (string) @comment))) ; Function/method docstring (function_definition body: (block . (expression_statement (string) @comment))) ; Attribute docstring ((expression_statement (assignment)) . (expression_statement (string) @comment))
Question: I am trying to import sklearn library by writing code like from sklearn.preprocessing import MinMaxScaler but it kept showing same error. I tried uninstalling and reinstalling but no change. Command prompt is also giving same error. Recently I installed some python libraries but that never affected my enviroment. I also tried running the code in jupyter notebook. When I tried to import numpy like import numpy as np, it ran successfully. So the problem is only with sklearn. Also, I have worked with sklearn before but have never seen such an error.

Answer: Run pip3 install --upgrade scipy OR upgrade whatever tool that tried to import np.int and failed np.int is same as normal int of python and scipy was outdated for me
Question: I'm really struggling with this issue, and can't seem to find an answer anywhere. I've got a text file which has name of the station and location, the task is to print out the names of the stations all underneath each other in order and same for the locations. In my text file the names of the stations are always made up of two words and the location is 3 words. text_file = "London Euston 12 London 56, Aylesbury Vale 87 Parkway 99, James Cook 76 University 87, Virginia Water 42 Surrey 78"  Desired outcome would be: Stations: London Euston Aylesbury Vale James Cook Virginia Water Locations: 12 London 56 87 Parkway 99 76 University 87 42 Surrey 78  my current code: replaced = text_file.replace(","," ") replaced_split = replaced.split() i = 0 b = 2 stations = [] locations = [] while b < len(replaced_split): locations.append(replaced_split[b:b+3]) b += 5 while i < len(replaced_split): stations.append(replaced_split[i:i+2]) i += 5 for x in range(len(stations)): print(stations[x]) for y in range(len(locations)): print(dates[y])  The outcome I'm receiving is printing lists out: ['London', 'Euston'] ['Aylesbury', 'Vale'] ['James', 'Cook'] ['Virginia', 'Water'] ['12', 'London', '56'] ['87', 'Parkway', '99'] ['76', 'University', '87'] ['42', 'Surrey', '78'] 

Answer: This is a simple a straight forward approach using for-loop instead of while loop. What the code does: It splits your string into substrings, where every substring is separated by comma and space. After that it splits those substrings again by space then joins the first two elements of each substring to create station and appends it to the stations list, and finally joins the rest of the substring leaving the first two elements as the location and appends that to the locations list. Now you loop the lists to print their elements stations = [] locations = [] for char in text_file.split(", "): parts = char.split(" ") stations.append(" ".join(parts[:2])) locations.append(" ".join(parts[2:])) print("Stations:") for station in stations: print(station) print("\nLocations:") for location in locations: print(location)  You can also choose to unpack stations and locations lists instead of using a traditional for-loop to print elements one at a time: print("Stations:") print(*stations, sep='\n') print("\nLocations:") print(*locations, sep='\n')   Stations: London Euston Aylesbury Vale James Cook Virginia Water Locations: 12 London 56 87 Parkway 99 76 University 87 42 Surrey 78
Question: The code works fine in Postman and provides a valid response but fails to generate the OpenAPI/Swagger UI automatic docs. class Role(str, Enum): Internal = "internal" External = "external" class Info(BaseModel): id: int role: Role class AppInfo(Info): info: str @app.post("/api/v1/create", status_code=status.HTTP_200_OK) async def create(info: Info, apikey: Union[str, None] = Header(str)): if info: alias1 = AppInfo(info="Portal Gun", id=123, role=info.role) alias2 = AppInfo(info="Plumbus", id=123, , role=info.role) info_dict.append(alias1.dict()) info_dict.append(alias2.dict()) return {"data": info_dict} else: raise HTTPException( status_code=status.HTTP_404_NOT_FOUND, detail=f"Please provide the input" )  Error received: TypeError: Object of type 'type' is not JSON serializable 

Answer: Issue The reason you get the following error in the console (Note that this error could also be raised by other causes—see here): TypeError: Object of type 'type' is not JSON serializable  as well as the error below in the browser, when trying to load the OpenAPI/Swagger UI autodocs at /docs: Fetch error Internal Server Error /openapi.json  is due to the following line in your code: apikey: Union[str, None] = Header(str) ^^^  Solution When declaring a Header parameter (or any other type of parameter, i.e., Path, Query, Cookie, etc), the first value that is passed to the Header class constructor (i.e., __init__  method) is the default value, which can either be None or some default value based on the type you specified for the parameter—in your case that could be some string value, e.g., 'some-api-key', not the type str). Since you defined the parameter as Optional, you could simply pass None as the default value: apikey: Union[str, None] = Header(None)  Please have a look at this answer and this answer for more details on Optional parameters in FastAPI.
Question: I usually like to call some functions during debugging in the console just to see some quick results. However with async functions, this doesn't seem to be possible: import asyncio async def func1(): print('func1') def func2(): print('func2') async def main(): task = asyncio.create_task(func1()) await task # put a break point here asyncio.run(main())  Let's say we put a break point in the line of await task Now if I call func2() in the console it will print 'func2' perfectly fine. However, if I enter await task in console, I will get the below error:  File ".../anaconda3/lib/python3.9/asyncio/base_events.py", line 585, in _check_running raise RuntimeError( RuntimeError: Cannot run the event loop while another loop is running  python 3.9 pycharm 2022.3.1 Is there any way I can call the async functions in the console like the non-async functions?

Answer: You can suspend the current_task and then run the event loop until the task is done. import asyncio from asyncio import tasks def wait(task_or_coro): task = asyncio.ensure_future(task_or_coro) loop, current_task = task.get_loop(), tasks.current_task() tasks._leave_task(loop, current_task) while not task.done(): loop._run_once() tasks._enter_task(loop, current_task) return task.result()  Call wait(task) instead of await task.
Question: On Windows 10 I want to read data from UDP port 9001. I have created the following script which does not give any output (python 3.10.9): import socket sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) sock.bind(("", 9001)) while True: data, addr = sock.recv(1024) print(f"received message: {data.decode()} from {addr}")  I checked that a device is sending UDP data on port 9001 using wireshark. But the above code just "runs" on powershell without any output (and without any errors). Any ideas how to fix this? I found this page with a powershell script that is supposed to listen to a UDP port. So I tried this and created a file Start-UDPServer.ps1 with the content as described in that page as follows: function Start-UDPServer { [CmdletBinding()] param ( # Parameter help description [Parameter(Mandatory = $false)] $Port = 10000 ) # Create a endpoint that represents the remote host from which the data was sent. $RemoteComputer = New-Object System.Net.IPEndPoint([System.Net.IPAddress]::Any, 0) Write-Host "Server is waiting for connections - $($UdpObject.Client.LocalEndPoint)" Write-Host "Stop with CRTL + C" # Loop de Loop do { # Create a UDP listender on Port $Port $UdpObject = New-Object System.Net.Sockets.UdpClient($Port) # Return the UDP datagram that was sent by the remote host $ReceiveBytes = $UdpObject.Receive([ref]$RemoteComputer) # Close UDP connection $UdpObject.Close() # Convert received UDP datagram from Bytes to String $ASCIIEncoding = New-Object System.Text.ASCIIEncoding [string]$ReturnString = $ASCIIEncoding.GetString($ReceiveBytes) # Output information [PSCustomObject]@{ LocalDateTime = $(Get-Date -UFormat "%Y-%m-%d %T") SourceIP = $RemoteComputer.address.ToString() SourcePort = $RemoteComputer.Port.ToString() Payload = $ReturnString } } while (1) }  and started it in an Powershell terminal (as admin) as .\Start-UDPServer.ps1 -Port 9001  and it returned to the Powershell immediately without ANY output (or error message). Maybe windows is broken? If there is a solution to finally listen to UDP port 9001, I still strongly prefer a python solution!

Answer: As far as I can see, your posted Python code should have given you an error when run if it was receiving any data, so that suggests that the data was not getting to the process at all. I'd recommend checking your Windows Firewall settings for that port, and any other host-based firewalls you might be running. But also, the recv() method does not return a tuple. recvfrom() does, so the following code works: import socket sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) sock.bind(("", 9001)) while True: data, addr = sock.recvfrom(1024) print(f"received message: {data.decode()} from {addr}")  A tangential note: the Powershell script does not actually start a UDP server, it just creates a function to do so. So you need to add a line Start-UDPServer -Port 9001 at the end to call the function if you want it to actually listen for datagrams.
Question: If I have a a data frame df = pd.DataFrame({'A': [1.1, 2.2, 3.3], 'B': [4.4, 5.5, 6.6]})  I can use Great Expectations to check the name and dtypes of the columns like so: import great_expectations as ge df_asset = ge.from_pandas(df) # List of expectations df_asset.expect_column_to_exist('A') df_asset.expect_column_to_exist('B') df_asset.expect_column_values_to_be_of_type('A', 'float') df_asset.expect_column_values_to_be_of_type('B', 'float') if df_asset.validate()["success"]: print("Validation passed") else: print("Validation failed")  But how can I do a similar thing to check the index of the data frame? I.e. if the data frame was instead df = pd.DataFrame({'A': [1.1, 2.2, 3.3], 'B': [4.4, 5.5, 6.6]}).set_index('A')  I am looking for something like df_asset.expect_index_to_exist('idx') df_asset.expect_index_values_to_be_of_type('idx', 'float')  to replace in the list of expectations

Answer: One quick hack is to use .reset_index to convert the index into a regular column: import great_expectations as ge df_asset = ge.from_pandas(df.reset_index()) # List of expectations df_asset.expect_column_to_exist('A') df_asset.expect_column_to_exist('B') df_asset.expect_column_values_to_be_of_type('A', 'float') df_asset.expect_column_values_to_be_of_type('B', 'float') # index-related expectations df_asset.expect_column_to_exist('index') df_asset.expect_column_values_to_be_of_type('index', 'int') if df_asset.validate()["success"]: print("Validation passed") else: print("Validation failed")  Note that the default name for an unnamed index is 'index', but you can also control it with kwarg names (make sure you have pandas>=1.5.0). Here is an example: df_asset = ge.from_pandas(df.reset_index(names='custom_index_name'))  This could be useful when you want to avoid clashes with existing column names. This approach can also be used for multiple indexes by providing a tuple of custom names.
Question: I am trying to do this exercise from momepy (https://github.com/pysal/momepy/blob/main/docs/user_guide/getting_started.ipynb), but on the third codeblock f, ax = plt.subplots(figsize=(10, 10)) buildings.plot(ax=ax) ax.set_axis_off() plt.show()  I got the following error AttributeError Traceback (most recent call last) Cell In[12], line 5 1 buildings = gpd.read_file(momepy.datasets.get_path('bubenec'), 2 layer='buildings') 4 f, ax = plt.subplots(figsize=(10, 10)) ----> 5 buildings.plot(ax=ax) 6 ax.set_axis_off() 7 plt.show() File ~/miniconda3/envs/testbed/lib/python3.10/site-packages/geopandas/plotting.py:925, in GeoplotAccessor.__call__(self, *args, **kwargs) 923 kind = kwargs.pop("kind", "geo") 924 if kind == "geo": --> 925 return plot_dataframe(data, *args, **kwargs) 926 if kind in self._pandas_kinds: 927 # Access pandas plots 928 return PlotAccessor(data)(kind=kind, **kwargs) File ~/miniconda3/envs/testbed/lib/python3.10/site-packages/geopandas/plotting.py:689, in plot_dataframe(df, column, cmap, color, ax, cax, categorical, legend, scheme, k, vmin, vmax, markersize, figsize, legend_kwds, categories, classification_kwds, missing_kwds, aspect, **style_kwds) 686 markersize = df[markersize].values 688 if column is None: --> 689 return plot_series( 690 df.geometry, 691 cmap=cmap, 692 color=color, ... ---> 67 geom = shapely.geos.lgeos.GEOSGeom_clone(geom._ptr) 68 return shapely.geometry.base.geom_factory(geom) 70 # fallback going through WKB AttributeError: module 'shapely.geos' has no attribute 'lgeos'  Not entirely sure if I am missing a module or if there are library incompatibilities

Answer: In the new version(2.0.0 released on 12 December 2022) of shapely there is no attribute 'lgeos'. Link to document: https://pypi.org/project/Shapely/#history You can check version by: shapely.__version__  If you want to use lgeos in your code you can downgrade your version to 1.8.5 %pip install shapely==1.8.5  I was able to successfully import on version 1.8.5 on Jupyter. from shapely.geos import lgeos
Question: I need the function do_it retrying up to 10 times if any exception is raised, with an interval of 0.5 to 1.5 seconds between retries. I am trying to do it with backoff decorator and the following code: import backoff x = 0 @backoff.on_exception(max_tries=10) def do_it(): global x x = x + 1 print(f'x: {x}') try: z = downloaded_from_somewhere('http://somewhere.com/z') # download may fail y = 1/z # z may be equal to zero, that will raise a division per zero and fail return y except: return f'Error after {x} tries' do_it()  The function may raise an exception due to different failures in try: After 10 retries it will return Error after {x} tries The backoff decorator library documentation is precarious. How to correctly specify the decorator parameters @backoff.on_exception(max_tries=10, something_else=blablabla, and_other=blabla2) to have the function working properly?

Answer:  but if the function fail after the 10 retries, how to instruct my code on how to move forward  First, a caveat: the usage of the backoff decorator you show in your question is invalid; you must provide the wait_gen and exception parameters. If you're using the backoff.on_exception method, then you want your function to raise an exception on failure. This is how the backoff decorator knows to retry your function. For example, consider the following, in which we have a function that fails 80% of the time: import backoff import random @backoff.on_exception(backoff.constant, ValueError, interval=1, max_tries=5) def do_it(): x = random.randint(0, 100) if x < 80: print("Failed!") raise ValueError("Failed!") print("Success!") do_it()  This will retry do_it() up to five times; if it fails more than five times, the exception bubbles up to the calling code. Running the above looks like this if it succeeds within max_tries: Failed! Failed! Success!  And like this if it fails five times: Failed! Failed! Failed! Failed! Failed! Traceback (most recent call last): File "/home/lars/tmp/python/backofftest.py", line 15, in <module> do_it() File "/home/lars/.local/share/virtualenvs/python-LD_ZK5QN/lib64/python3.11/site-packages/backoff/_sync.py", line 105, in retry ret = target(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lars/tmp/python/backofftest.py", line 10, in do_it raise ValueError("Failed!") ValueError: Failed!  Presumably if the function fails more than max_tries times, you would catch that exception in the calling code and exit gracefully with an appropriate error message: try: do_it() except ValueError as err: sys.exit("failed to do the thing")  (Note that in this example I'm using a constant backoff period via backoff.constant; in practice, you would probably use backoff.expo instead for exponential backoff behavior.)
Question: Let's assume I have the following dataframe: import pandas as pd d = {'col1': [1, 2,3,4], 'col2': [4, 2, 1, 3], 'col3': [1,0,1,1], 'outcome': [1,0,1,0]} df = pd.DataFrame(data=d)  I want this dataframe sorted by col1 and col2 on the minimum value. The order of the indexes should be 2, 0, 1, 3. I tried this with df.sort_values(by=['col2', 'col1']), but than it takes the minimum of col1 first and then of col2. Is there anyway to order by taking the minimum of two columns?

Answer: Using numpy.lexsort: order = np.lexsort(np.sort(df[['col1', 'col2']])[:, ::-1].T) out = df.iloc[order]  Output:  col1 col2 col3 outcome 2 3 1 1 1 0 1 4 1 1 1 2 2 0 0 3 4 3 1 0  Note that you can easily handle any number of columns: df.iloc[np.lexsort(np.sort(df[['col1', 'col2', 'col3']])[:, ::-1].T)] col1 col2 col3 outcome 1 2 2 0 0 2 3 1 1 1 0 1 4 1 1 3 4 3 1 0