eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: For example, let us say I want to fill_null(strategy="zero") only the numeric columns of my DataFrame. My current strategy is to do this: import polars as pl import polars.selectors as cs df = pl.DataFrame( [ pl.Series("id", ["alpha", None, "gamma"]), pl.Series("xs", [None, 100, 2]), ] ) final_df = df.select(cs.exclude(cs.numeric())) final_df = final_df.with_columns( df.select(cs.numeric()).fill_null(strategy="zero") ) print(final_df)  shape: (3, 2) ┌───────┬─────┐ │ id ┆ xs │ │ --- ┆ --- │ │ str ┆ i64 │ ╞═══════╪═════╡ │ alpha ┆ 0 │ │ null ┆ 100 │ │ gamma ┆ 2 │ └───────┴─────┘  Are there alternative, either more idiomatic or more efficient methods to achieve what I'd like to do?

Answer: pl.DataFrame.select returns a dataframe that contains only the columns listed as arguments. Alternatively, pl.DataFrame.with_columns adds columns to the dataframe (and replaces columns with the same name). Especially, this provides you with the tools to perform the filling without an intermediate dataframe. You can simply use pl.DataFrame.with_columns to fill missing values only in numeric columns (i.e. replace them with their filled versions). df.with_columns( cs.numeric().fill_null(strategy="zero") )  shape: (3, 2) ┌───────┬─────┐ │ id ┆ xs │ │ --- ┆ --- │ │ str ┆ i64 │ ╞═══════╪═════╡ │ alpha ┆ 0 │ │ null ┆ 100 │ │ gamma ┆ 2 │ └───────┴─────┘
Question: I have created a pandas dataframe as follows: import pandas as pd import numpy as np ds = {'col1' : [1,1,1,1,1,1,1,2,2,2,2,3,3,3,3,3,4,4,4,4,4,4,4,4,4], 'col2' : [12,3,4,5,4,3,2,3,4,6,7,8,3,3,65,4,3,2,32,1,2,3,4,5,32], } df = pd.DataFrame(data=ds)  The dataframe looks as follows: print(df) col1 col2 0 1 12 1 1 3 2 1 4 3 1 5 4 1 4 5 1 3 6 1 2 7 2 3 8 2 4 9 2 6 10 2 7 11 3 8 12 3 3 13 3 3 14 3 65 15 3 4 16 4 3 17 4 2 18 4 32 19 4 1 20 4 2 21 4 3 22 4 4 23 4 5 24 4 32  Based on the values of column col1, I need to extract:  3 random records where col1 == 1 2 random records such that col1 = 2 2 random records such that col1 = 3 3 random records such that col1 = 4  Can anyone help me please?

Answer: I would shuffle the whole input with sample(frac=1), then compute a groupby.cumcount to select the first N samples per group (with map and boolean indexing) where N is defined in a dictionary: # {col1: number of samples} n = {1: 3, 2: 2, 3: 2, 4: 3} out = df[df[['col1']].sample(frac=1) .groupby('col1').cumcount() .lt(df['col1'].map(n))]  Shorter code, but probably less efficient, using a custom groupby.apply with a different sample for each group: n = {1: 3, 2: 2, 3: 2, 4: 3} out = (df.groupby('col1', group_keys=False) .apply(lambda g: g.sample(n=n[g.name])) )  Example output:  col1 col2 0 1 12 3 1 5 4 1 4 7 2 3 8 2 4 11 3 8 13 3 3 17 4 2 18 4 32 24 4 32
Question: I want to "hack" python's import so that it will first search in the path I specified and fallback to the original if not found. The traditional way of using sys.path will not work if there is a __init__.py, see below.  CASE 1: The following works: Files: . ├── a │ ├── b.py # content: x='x' │ └── c.py # content: y='y' ├── hack │ └── a │ └── b.py # content: x='hacked' └── test.py # content: see below  # test.py import sys sys.path.insert(0, 'hack') # insert the hack path from a.b import x from a.c import y print(x, y)  Running test.py gives hacked y as desired, where x is hacked :)  CASE 2: However, if there is a __init__.py in a, it will not work. Files: . ├── a │ ├── b.py │ ├── c.py │ └── __init__.py # <- NOTE THIS ├── hack │ └── a │ └── b.py └── test.py  Running test.py gives x y, where x is not hacked :(  CASE 3: To fix case 2, I tried adding __init__.py to the hack path, but this disables the fallback behavior. Files: . ├── a │ ├── b.py │ ├── c.py │ └── __init__.py ├── hack │ └── a │ ├── b.py │ └── __init__.py # <- NOTE THIS └── test.py  Running test.py raises the following error as there is no c.py in the hack path and it fails to fallback to the original path. ModuleNotFoundError: No module named 'a.c'   My question is, how to make case 2 work? Additional background: The above cases are just simplified examples. In the real situation,  Both a and hack/a are large repos with many subfolders and files.  The imported modules (e.g. a.b) may also contain import statements that need to be hacked.   Therefore, ideally the solution would be to only add a few lines of code at the top of test.py rather than modifying exisiting code. UPDATE: I have come up with a solution below (I cannot accept my own answer in 2 days). If you have better solutions or suggestions, please feel free to discuss.

Answer: The solution is to overwrite the default __import__ function (which is used by import statements) so that it first tries to import from the hack folder. __import = __import__ # save the original def _import(name, *a, **b): try: return __import('hack.'+name, *a, **b) except ImportError: return __import(name, *a, **b) __builtins__.__import__ = _import # overwrite with our own
Question: I have a question to NumPy experts. Consider a NumPy scalar: c = np.arange(3.0).sum(). If I try to multiply it with a custom sequence like e.g. class S: def __init__(self, lst): self.lst = lst def __len__(self): return len(self.lst) def __getitem__(self, s): return self.lst[s] c = np.arange(3.0).sum() s = S([1, 2, 3]) print(c * s)  it works and I get: array([3., 6., 9.]). However, I can't do so with a list. For instance, if I inherit S from list and try it, this does not work anymore class S(list): def __init__(self, lst): self.lst = lst def __len__(self): return len(self.lst) def __getitem__(self, s): return self.lst[s] c = np.arange(3.0).sum() s = S([1, 2, 3]) print(c * s)  and I get "can't multiply sequence by non-int of type 'numpy.float64'". So how does NumPy distinguish between the two cases? I am asking because I want to prevent such behavior for my "S" class without inheriting from list.  UPD Since the question has been misunderstood a few times, I try to stress more exactly what the problem is. It's not about why the list does not cope with multiplication by float and the error is raised. It is about why in the first case (when S is NOT inherited from list) the multiplication is performed by the object "c" and in the second case (when S is inherited from list) the multiplication is delegated to "s". Apparently, the method c.__mul__ does some checks which pass in the first case, but fail in the second case, so that s.__rmul__ is called. The question is essentially: What are those checks? (I strongly doubt that this is anything like isinstance(other, list)).

Answer: As pointed out by @hpaulj (thanks a lot!) this question was already around: Array and __rmul__ operator in Python Numpy. The mechanics of the problem was explained very well in this answer: stackoverflow.com/a/38230576/901925. However, the proposed solution of inheriting from np.ndarray is certainly a "dirty" one. In the reply immediately after, a solution based on the function __numpy_ufunc__ is proposed. The latter however is called __array_ufunc__ in modern NumPy. This function can be just set to None in the definition of the class "S". This leads to delegation of multiplcation to s.__rmul__ without attempts to perform it via c.__mul__.
Question: I know how to plot a line on the second axis over a VERTICAL bar chart. Now I want the bars HORIZONTAL and the line top-to-bottom, just like the whole chart rotates 90°. If I simply replace bar with barh, the line is still left-to-right... Can I do this with Matplotlib? Here is a sample for VERTICAL bar chart: import matplotlib.pyplot as plt import pandas as pd df = pd.DataFrame( { "A": [2, 4, 8], "B": [3, 5, 7], "C": [300, 100, 200], }, index=["a", "b", "c"], ) ax0 = df.plot(y=df.columns[:2], kind="bar", figsize=(10.24, 5.12)) ax1 = df["C"].plot( kind="line", secondary_y=True, color="g", marker=".", ) plt.tight_layout() plt.show()  Let me stress: I do see those questions related to VERTICAL bar charts. Now I'm asking about HORIZONTAL bar charts. So this is not a duplicate question.

Answer: Pandas plotting doesn't readily support a secondary x axis. Instead, you can directly plot via matplotlib. (Note that df.plot(...) plots via pandas. Pandas plotting is a pandas specific interface towards matplotlib, and only supports a subset of matplotlib's functionality.) import matplotlib.pyplot as plt import pandas as pd df = pd.DataFrame( { "A": [2, 4, 8], "B": [3, 5, 7], "C": [300, 100, 200], }, index=["a", "b", "c"], ) ax0 = df.plot(y=df.columns[:2], kind="barh", figsize=(10.24, 5.12)) ax1 = ax0.twiny() ax1.plot(df["C"], df.index, color="g", marker=".", ) plt.tight_layout() plt.show()   PS: To exchange the top and bottom axes, you can use: ax0 = df.plot(y=df.columns[:2], kind="barh", figsize=(10.24, 5.12)) ax1 = ax0.twiny() ax1.plot(df["C"], df.index, color="g", marker=".", ) # change the direction of the y-axis ax0.invert_yaxis() # set the x-axis for ax0 at the top ax0.tick_params(top=True, bottom=False, labeltop=True, labelbottom=False) # for the ticks and tick labels ax0.xaxis.set_label_position('top') # for the axis label # set the x-axis for ax1 at the bottom ax1.tick_params(top=False, bottom=True, labeltop=False, labelbottom=True) ax1.xaxis.set_label_position('bottom')
Question: I'm working on a Python Turtle graphics program and I'm trying to use the exitonclick method to close the window when it's clicked. However, it doesn't seem to be working. from turtle import Turtle, Screen rem = Turtle() screen = Screen() rem.fd(70) def clear(): screen.clearscreen() screen.listen() screen.onkey(fun=clear,key = "c") screen.exitonclick()  When I run it and try to exit the program by clicking in the screen it works fine and quits but when I press c and cleared the screen then if I try to exit by clicking nothing happens.

Answer: screen.clearscreen() completely resets the window. That includes removing all modifications to it made through exitonclick. The easiest solution is just to call exitonclick again in your custom function: from turtle import Turtle, Screen rem = Turtle() screen = Screen() rem.fd(70) def clear(): screen.clearscreen() screen.exitonclick() screen.listen() screen.onkey(fun=clear,key = "c") screen.exitonclick()
Question: I am trying to make image recognition program using pyautogui. I want the program to print "I can see it!" when it can see a particular image and "Nope nothing there" when it can't. I'm testing for this by using pyautogui.locateOnScreen. If the call returns anything but None, I print the first message. Otherwise, I print the second. However, my code is giving me an ImageNotFoundException instead of printing in the second case. Why is this happening? import pyautogui while True: if pyautogui.locateOnScreen('Dassault Falcon 7X.png', confidence=0.8, minSearchTime=5) is not None: print("I can see it!") time.sleep(0.5) else: print("Nope nothing there") 

Answer: From the pyautogui docs:  NOTE: As of version 0.9.41, if the locate functions can’t find the provided image, they’ll raise ImageNotFoundException instead of returning None.  Your program assumes pre-0.9.41 behavior. To update it for the most recent version, replace your if-else blocks with try-except: from pyautogui import locateOnScreen, ImageNotFoundException while True: try: pyautogui.locateOnScreen('Dassault Falcon 7X.png') print("I can see it!") time.sleep(0.5) except ImageNotFoundException: print("Nope nothing there")
Question: I would like to create functions using the normal def procedure in Python with labels assigned to the namespace that are pulled from a list of strings. How can this be achieved? An example problem: given an arbitrary list of strings exampleList=['label1','label2','label3',...] of length k, initialize k def defined functions of the form exampleList=['label1','label2','label3','label4'] def label1(arg: str): if len(arg)>exampleList[0] print('the word '+arg+' has more letters than the word '+exampleList[0]) def label2(arg: str): if len(arg)>exampleList[1] print('the word '+arg+' has more letters than the word '+exampleList[1]) # ...etc.  programmatically. If the functions to be defined only involve operations implementable within lambda functions, then this can be done with lambda functions by updating the globals() dict. E.g., one can write something like exampleList=['label1','label2','label3'] exampleOperation= lambda j,k: len(j)-len(k) globals().update(zip(exampleList,[lambda k:exampleOperation(j,k) for j in exampleList]))  The above example initialized some lambda functions with labels assigned from the list exampleList. I would, however, like to accomplish the analogous task with normal def defined functions rather than lambda functions.

Answer: you could use a class with a factory function class MyLabels: def __init__(self, label_list): for label in label_list: setattr(self, label, self.label_factory(label)) def label_factory(self, label): def my_func(arg: str): if len(arg) > len(label): print(f'the word {arg} has more letters than the word {label}') return my_func exampleList=['label1','label2','label3','label4'] ml = MyLabels(exampleList) ml.label1('stackoverflow') #the word stackoverflow has more letters than the word label1
Question: I have data that includes a full name and first name, and I need to make a new column with the last name. I can assume full - first = last. I've been trying to use slice with an index the length of the first name + 1. But that index is a series, not an integer. So it's returning NaN. The commented lines show the things I tried. It took me a while to realize what the series/integer issue was. It seems this shouldn't be so difficult. Thanks import pandas as pd columns = ['Full', 'First'] data = [('Joe Smith', 'Joe'), ('Bobby Sue Ford', 'Bobby Sue'), ('Current Resident', 'Current Resident'), ('', '')] df = pd.DataFrame(data, columns=columns) #first_chars = df['First'].str.len() + 1 #last = df['Full'].str[4:] #last = df['Full'].str[first_chars:] #last = df['Full'].str.slice(first_chars) #last = df.Full.str[first_chars:] #pd.DataFrame.insert(df, loc=2, column='Last', value=last) #df['Last'] = df.Full.str[first_chars:] #df['Last'] = str(df.Full.str[first_chars:]) #first_chars = int(first_chars) #df['Last'] = df['Full'].apply(str).apply(lambda x: x[first_chars:]) df['Last'] = df['Full'].str.slice(df['First'].str.len() + 1) print(df) 

Answer: Edit: Use removeprefix instead of replace to deal with cases where first and last names are the same: df['Last'] = df.apply(lambda row: row['Full'].removeprefix(row['First']).strip(), axis=1)   Full First Last 0 Joe Smith Joe Smith 1 Bobby Sue Ford Bobby Sue Ford 2 Current Resident Current Resident 3 4 Joe Joe Joe Joe   Original answer: Use apply on axis=1 to replace each name: df['Last'] = df.apply(lambda row: row['Full'].replace(row['First'], '').strip(), axis=1)   Full First Last 0 Joe Smith Joe Smith 1 Bobby Sue Ford Bobby Sue Ford 2 Current Resident Current Resident 3
Question: In Python, some programs create the error RecursionError: maximum recursion depth exceeded in comparison or similar. This is because there is a limit set for how deep the recursion can be nested. To get the current value for maximum recursion depth, use import sys sys.getrecursionlimit()  The default value seems to be 1000 or 1500 depending on the specific system/python/... combination. The value can be increased to have access to deeper recursion. import sys sys.setrecursionlimit(LIMIT)  where LIMIT is the new limit. This is the error that I get when I exceed the 'limit' limit: OverflowError: Python int too large to convert to C int  Why is the 'limit' limit int(2**32 / 2) - 31? I would expect that it maybe is int(2**32 / 2) - 1 as it makes sense that integers are signed (32-bit) numbers, but why - 31? The question first came up as I was testing with 3.9.13 (64-bit). I also tested with 3.12 (64-bit). Edit: Here is a demo on IDLE with Python 3.9.13. Python 3.9.13 (tags/v3.9.13:6de2ca5, May 17 2022, 16:36:42) [MSC v.1929 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license()" for more information. >>> import sys >>> v = 2**31-1 >>> print(v) 2147483647 >>> sys.setrecursionlimit(v) Traceback (most recent call last): File "<pyshell#3>", line 1, in <module> sys.setrecursionlimit(v) OverflowError: Python int too large to convert to C int >>> v = 2**31-30 >>> sys.setrecursionlimit(v) Traceback (most recent call last): File "<pyshell#5>", line 1, in <module> sys.setrecursionlimit(v) OverflowError: Python int too large to convert to C int >>> v = 2**31-31 >>> sys.setrecursionlimit(v) >>> 

Answer: Usually the limit "limit" would be 2**31 - 1, i.e. the max value for a signed C integer (although the highest possible limit is documented as being platform-dependent). The lower usable range you get within IDLE is because it wraps the built-in function to impose a lower limit: >>> import sys >>> sys.setrecursionlimit <function setrecursionlimit at 0xcafef00d> >>> sys.setrecursionlimit.__wrapped__ <built-in function setrecursionlimit>  This is specific to IDLE, and you should see the "full" 2**31-1 depth when using the same interpreter on the same platform otherwise. You can find the monkeypatch here in CPython sources: Lib/idlelib/run.py:install_recursionlimit_wrappers Usable values are reduced by RECURSIONLIMIT_DELTA=30, and a comment on wrapper function references bpo-26806 (IDLE not displaying RecursionError tracebacks and hangs), where Terry J. Reedy mentions the reason for patching a lower limit:  IDLE adds it own calls to the call stack before (and possibly after) running user code  You should also see that in the docstring:
Question: I have two text files. One contains a very long list of strings (100 GB), the other contains about 30 strings. I need to find which lines in the second file are also in the first file and write them to another,third text file. Manually searching for each line is a pain, so I wanted to write a script to do it automatically. For this I choose Python because it is the only language that I know even a little. Essentially I tried copying this answer since I'm too inexperienced to write my own code: Compare 2 files in Python and extract differences as a strings smallfile = 'smalllist.txt' bigfile = 'biglist.txt' def file_2_list(file): with open(file) as file: lines = file.readlines() lines = [line.rstrip() for line in lines] return lines def diff_lists(lst1, lst2): differences = [] both = [] for element in lst1: if element not in lst2: differences.append(element) else: both.append(element) return(differences, both) listbig = file_2_list(bigfile) listsmall = file_2_list(smallfile) diff, both = diff_lists(listbig, listsmall) print(both)  I wanted it to print me the lines that are in both lists. However it gave me a "memory error". But I'm already using a 64-bit version of Python, so the memory limit shouldn't be an issue? (I have 16 GB RAM) So how can you avoid this “memory error”? Or maybe there is a better way to accomplish this task?

Answer: The file.readlines method reads the entirety of a file into memory, which you should avoid when the file is that large. You can instead read the lines of the smaller file into a set, and then iterate over the lines of the larger file to find the common lines by testing if a line is in the set: def common_lines(small_file, big_file): small_lines = set(small_file) return [line for line in big_file if line in small_lines] with open(smallfile) as file1, open(bigfile) as file2: both = common_lines(file1, file2)
Question: Suppose you have a JSON file like this: { "a": 0 } { "a": 1 }  It's not JSONL, because each object takes more than one line. But it's not a single valid JSON object either. It's sequentially listed pretty-printed JSON objects. json.loads in Python gives an error about invalid formatting if you attempt to load this, and the documentation indicates it only loads a single object. But tools like jq can read this kind of data without issue. Is there some reasonable way to work with data formatted like this using the core json library? I have an issue where I have some complex objects and while just formatting the data as JSONL works, for readability it would be better to store the data like this. I can wrap everything in a list to make it a single JSON object, but that has downsides like requiring reading the whole file in at once. There's a similar question here, but despite the title the data there isn't JSON at all.

Answer: You can partially decode text as JSON with json.JSONDecoder.raw_decode. This method returns a 2-tuple of the parsed object and the ending index of the object in the string, which you can then use as the starting index to partially decode the text for the next JSON object: import json def iter_jsons(jsons, decoder=json.JSONDecoder()): index = 0 while (index := jsons.find('{', index)) != -1: data, index = decoder.raw_decode(jsons, index) yield data  so that: jsons = '''\ { "a": 0 } { "a": 1 }''' for j in iter_jsons(jsons): print(j)  outputs: {'a': 0} {'a': 1}  Demo here Note that the starting index as the second argument to json.JSONDecoder.raw_decode is an implementation detail, and that if you want to stick to the publicly documented API you would have to use the less efficient approach of slicing the string (which involves copying the string) from the index before you pass it to raw_decode: def iter_jsons(jsons, decoder=json.JSONDecoder()): index = 0 while (index := jsons.find('{', index)) != -1: data, index = decoder.raw_decode(jsons := jsons[index:]) yield data
Question: I'm writing an interpolation routine and have a dictionary which stores the function values at the fitting points. Ideally, the dictionary keys would be 2D Numpy arrays of the fitting point coordinates, np.array([x, y]), but since Numpy arrays aren't hashable these are converted to tuples for the keys. # fit_pt_coords: (n_pts, n_dims) array # fn_vals: (n_pts,) array def fit(fit_pt_coords, fn_vals): pt_map = {tuple(k): v for k, v in zip(fit_pt_coords, fn_vals)} ...  Later in the code I need to get the function values using coordinates as keys in order to do the interpolation fitting. I'd like this to be within @jax.jited code, but the coordinate values are of type <class 'jax._src.interpreters.partial_eval.DynamicJaxprTracer'>, which can't be converted to a tuple. I've tried other things, like creating a dictionary key as (x + y, x - y), but again this requires concrete values, and calling .item() results in an ConcretizationTypeError. At the moment I've @jax.jited all of the code I can, and have just left this code un-jitted. It would be great if I could jit this code as well however. Are there any better ways to do the dictionary indexing (or better Jax-compatible data structures) which would allow all of the code to be jitted? I am new to Jax and still understading how it works, so I'm sure there must be better ways of doing it...

Answer: There is no way to use traced JAX values as dictionary keys. The problem is that the key values will not be known until runtime within the XLA compiler, and XLA has no dictionary-like data structure that such lookups can be lowered to. There are imperfect solutions, such as keeping the dictionary on the host and using something like io_callback to do the dict lookups on host, but this approach comes with performance penalties that will likely make it impractical. Unfortunately, your best approach for doing this efficiently under JIT would probably be to switch to a different interpolation algorithm that doesn't depend on hash table lookups.