eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: If I pickle a function with dill that contains a global, somehow that global state isn't respected when the function is loaded again. I don't understand enough about dill to be anymore specific, but take this working code for example: import multiprocessing import dill def initializer(): global foo foo = 1 def worker(arg): return foo with multiprocessing.Pool(2, initializer) as pool: res = pool.map(worker, range(10)) print(res)  This works fine, and prints [1, 1] as expected. However, if I instead pickle the initializer and worker functions using dill's recurse=True, and then restore them, it fails: import multiprocessing import dill def initializer(): global foo foo = 1 def worker(arg): return foo with open('funcs.pkl', 'wb') as f: dill.dump((initializer, worker), f, recurse=True) with open('funcs.pkl', 'rb') as f: initializer, worker = dill.load(f) with multiprocessing.Pool(2, initializer) as pool: res = pool.map(worker, range(2))  This code fails with the following error:  File "/tmp/ipykernel_158597/1183951641.py", line 9, in worker return foo ^^^ NameError: name 'foo' is not defined  If I use recurse=False it works fine, but somehow pickling them in this way causes the code to break. Why?

Answer: With the recurse=True option, dill.dump builds a new globals dict for the function being serialized with objects that the function refers to also recursively serialized. The side effect is that when deserialized with dill.load, these objects are reconstructed as new objects, including the globals dict for the function. This is why, after deserialization, the globals dicts of the functions become different objects from each other, so that changes made to the globals dict of the initializer function have no effect on the globals dict of the worker function. You can verify this behavior by checking the identity of the global namespace in which a function object is defined and runs under, availble as the __globals__ attribute of the function object: import dill def initializer(): global foo foo = 1 def worker(arg): return foo print(id(globals())) print(id(initializer.__globals__)) print(id(worker.__globals__)) with open('funcs.pkl', 'wb') as f: dill.dump((initializer, worker), f, recurse=True) with open('funcs.pkl', 'rb') as f: initializer, worker = dill.load(f) print('-- dilled --') print(id(globals())) print(id(initializer.__globals__)) print(id(worker.__globals__))  This outputs something like: 124817730351552 124817730351552 124817730351552 -- dilled -- 124817730351552 124817727897280 124817728060352  Demo: https://replit.com/@blhsing1/RelievedPrimeLaws
Question: I would like to simplify below regex logic with one regex statement. Then it's easier to understand the logic. import re content = """ [ 1.765989] initcall init_module.cfi_jt+0x0/0x8 [altmode_glink] returned 0 after 379 usecs [ 0.001873] initcall selinux_init+0x0/0x1f8 returned 0 after 85 usecs """ for line in content.split("\n"): m = re.search("\[\s+(?P<ts>[\d.]+)\] initcall init_module.*\[(?P<func>[^ ]+)\] returned (?P<ret>[\d-]+) after (?P<val>[\d-]+) usecs",line) if m: ts = m['ts'] func = m['func'] ret = m['ret'] val = m['val'] print(ts,func,ret,val) else: m = re.search("\[\s+(?P<ts>[\d.]+)\] initcall (?P<func>[^ ]+)\+.*? returned (?P<ret>[\d-]+) after (?P<val>[\d-]+) usecs",line) if m: ts = m['ts'] func = m['func'] ret = m['ret'] val = m['val'] print(ts,func,ret,val)  Current output is good: 1.765989 altmode_glink 0 379 0.001873 selinux_init 0 85 

Answer: The tricky part is apparently how you identify a function name, which can be be matched by non-left-bracket characters followed by either a right bracket or a plus sign followed by non-spaces: for ts, func, ret, val in re.findall( r'\[\s*([\d.]+)] initcall .*?([^[]+)(?:]|\+\S+) ' r'returned (\d+) after (\d+) usecs', content): print(ts, func, ret, val)  This outputs: 1.765989 altmode_glink 0 379 0.001873 selinux_init 0 85  Demo here
Question: I have a dataframe that has worksite info. District# Site# Address 1 1 123 Bayview Ln 1 2 456 Example St 2 36 789 Hello Dr 2 44 789 Hello Dr  I am trying to transform this dataframe to add a column with the highest Site# as well as the distinct addresses when I group by District#. Here is an example of what I want the output to look like: District# Site# Address MaxSite# All District Addresses 1 1 123 Bayview Ln 2 123 Bayview Ln,456 Example St 1 2 456 Example St 2 123 Bayview Ln,456 Example St 2 36 789 Hello Dr 44 789 Hello Dr 2 44 789 Hello Dr 44 789 Hello Dr  I am able to get the Max Site# by doing df['MaxSite#'] = df.groupby(by='District#')['Site#'].transform('max')  But I am trying to find a similar way to list all of the unique addresses when I groupby District#. I have tried doing .transform('unique') but that is not a valid function name and doing .agg(['unique']) returns dimensions that do not match

Answer: You can use groupby and agg to get the Max Site Number and List all the addresses Then merge back to the original dataframe: grouped_df = df.groupby('District#').agg(Max_Site_Num=('Site#', 'max'), All_District_Addresses=('Address', lambda x: list(set(x))).reset_index() df = df.merge(grouped_df,on='District#')  Output:  District# Site# Address Max_Site_Num All_District_Addresses 0 1 1 123 Bayview Ln 2 [123 Bayview Ln, 456 Example St] 1 1 2 456 Example St 2 [123 Bayview Ln, 456 Example St] 2 2 36 789 Hello Dr 44 [789 Hello Dr] 3 2 44 789 Hello Dr 44 [789 Hello Dr]
Question: Referring to an item of a dict_keys, dict_values, or dict_items object by index raises a type error. For example: > my_dict = {"foo": 0, "bar": 1, "baz": 2} > my_dict.items()[0] --------------------------------------------------------------------------- TypeError Traceback (most recent call last) ----> 1 my_dict.items()[0] TypeError: 'dict_items' object is not subscriptable  My question is not how to address this. I am aware that one can just call list() or tuple() on a dict_keys, dict_values, or dict_items object and then subscript that. My question is why does this behaviour still persist in python given that the order of items in a dictionary has been guaranteed since python 3.7.0. Why is it not possible (or not desirable) to refactor the dict_keys, dict_values, and dict_items types so that they are subscriptable? The documentation for the stdlib describes these types as "view objects". I imagine that is where the answer to my question lies but I can't find a more detailed description of what a "view object" actually is. Can anyone enlighten me on this matter as well?

Answer: "Ordered" doesn't mean "efficiently indexable". There is no more efficient way to retrieve the Nth item of a dict view than to iterate N steps. Allowing indexing would give the impression that views can support this operation efficiently, which would in turn encourage extremely inefficient code. Trying to change the implementation so indexing is efficient would make other dict operations much less efficient, which isn't worthwhile for the use cases dicts are actually designed for.
Question: I am trying to get rolling rank based on group by. This is my demo data and "Rolling 2Y Rank' is expected column. The way how it works is I intend to group by each "ID" and calculate rank based on its own historical "Value". df = pd.DataFrame({ 'Year': [2000,2000,2000,2001,2001,2001,2002,2002,2002,2003,2003,2003], 'ID': ['A','B','C','A','B','C','A','B','C','A','B','C'], 'Value': [5,1,2,3,4,3,2,7,1,1,13,23], 'Rolling 2Y Rank': [np.nan,np.nan,np.nan, 2,1,1,2,1,2,2,1,1]})     Year ID Value Rolling 2Y Rank     2000 A 5 nan   2000 B 1 nan   2000 C 2 nan   2001 A 3 2   2001 B 4 1   2001 C 3 1   2002 A 2 2   2002 B 7 1   2002 C 1 2   2003 A 1 2   2003 B 13 1   2003 C 23 1    For example, for "ID"="A":  For first instance ("Year"=="2020"), there is no only one data, so it has nothing to compare against so it is NA. For second instance, we compare 3 with 5 (rolling 2-year window), so we get rank 2 (because 5 is larger than 3). For the third instance, we compare 2 against 3, so get the rank of 2 and so on so forth.  df[df['ID']=='A']  I have tried and it did not work out: df.groupby('ID')['Value'].rolling(2).rank() 

Answer: You want the reversed (descending) rank, so pass that to the rank function. Then you can do a merge to assign back to the original dataframe: df = df.merge( df.groupby('ID').rolling(2, on='Year')['Value'].rank(ascending=False), on=['ID','Year'], )  Output:  Year ID Value_x Value_y 0 2000 A 5 NaN 1 2000 B 1 NaN 2 2000 C 2 NaN 3 2001 A 3 2.0 4 2001 B 4 1.0 5 2001 C 3 1.0 6 2002 A 2 2.0 7 2002 B 7 1.0 8 2002 C 1 2.0 9 2003 A 1 2.0 10 2003 B 13 1.0 11 2003 C 23 1.0
Question: I'm trying to reorder the columns in a Polars dataframe and put 5 columns out of 100 first (the document must unfortunately be somewhat readable in excel). I can't seem to find an easy way to do this. Ideally, I'd like something simple like df.select( 'col2', 'col1', r'^.*$', # the rest of the columns, but this throws a duplicate column name error )  Negative lookahead is not supported so it's not possible to make a regex that excludes my selected columns. I could make two overlapping selections, drop the columns from one selection, and then join them, but this does not seem like it would be the intended way to do this. Every other solution I've found involves explicitly naming every single column, which I'm trying to avoid as the columns get added or change names somewhat frequently.

Answer: You can combine pl.exclude with the walrus operator. Suppose you have something like df=pl.DataFrame( [ pl.Series('c', [1, 2, 3], dtype=pl.Int64), pl.Series('b', [2, 3, 4], dtype=pl.Int64), pl.Series('fcvem', [4, 5, 6], dtype=pl.Int64), pl.Series('msoy', [4, 5, 6], dtype=pl.Int64), pl.Series('smrn', [4, 5, 6], dtype=pl.Int64), pl.Series('z', [4, 5, 6], dtype=pl.Int64), pl.Series('wxous', [4, 5, 6], dtype=pl.Int64), pl.Series('uusn', [4, 5, 6], dtype=pl.Int64), pl.Series('ydj', [4, 5, 6], dtype=pl.Int64), pl.Series('squr', [4, 5, 6], dtype=pl.Int64), pl.Series('yyx', [4, 5, 6], dtype=pl.Int64), pl.Series('nl', [4, 5, 6], dtype=pl.Int64), pl.Series('a', [0, 1, 2], dtype=pl.Int64), ] )  and you want the first 3 columns to be 'a', 'b', 'c'. You can do: df.select(*(start_cols:=['a','b','c']), pl.exclude(start_cols))  This creates a list called start_cols which contains 'a','b','c'. The asterisk unwraps the list and then pl.exclude uses the contents of start_cols to tell polars to return everything except start_cols. If you prefer, you could do this syntax instead: df.select((start_cols:=['a','b','c'])+ [pl.exclude(start_cols)])
Question: I have a use case where I need the permutations of boolean. However I do not need them when they are reversed. So I can do something like this: import itertools [ p for p in itertools.product([True,False],repeat=4) if p!=p[::-1] ]  This issue is that this is a list, stored in memory, and with increasing repeats I will run into memory errors. Is it possible to "subset" an iterator while keeping it as an iterator? If possible I am looking for a general answer, that applies to any sub setting strategy on any iterator

Answer: You use a generator expression, syntactically identical to a listcomp, but using parentheses rather than square brackets as the delimiter: ( p for p in itertools.product([True,False],repeat=4) if p!=p[::-1] )  If the generator expression is the sole argument to a function, you don't even need the extra parentheses (the function call parentheses serve double duty delimiting the function argument and the genexpr).
Question: What is Python idiom in uwsgi for if __name__ == '__main__': main()  I found here a long string in uwsgi instead of __main__ What __name__ string does uwsgi use?. But it looks like a workaround. Is there better way to call function once, when uwsgi starts my python script?

Answer: It depends what you are trying to guard against. There are three use cases I can think of:  Do something only if running a real server, and not unit tests. Do something in a module if it is the main WSGI file. Do something after uwsgi has forked a worker process.  Do something only if running a real server, and not unit tests The uwsgi binary provides a module called uwsgi that is otherwise not importable. This is best used for detecting if you are running unit tests or not. See Unit testing Flask app running under uwsgi try: import uwsgi except ImportError: UWSGI_SERVER = False else: UWSGI_SERVER = True if UWSGI_SERVER: do_something() else: import test_utils test_utils.mock_do_somethng()  Do something in a module if it is the main WSGI file This implies the above, that if your main module has a uwsgi_file_* module name, then you are running a uwsgi server. It also lets you know which file was run as the main module. That is, the one that contains your application. Maybe you are running a handful of micro-services that all use the same core, but need to be configured slightly differently depending on which service is running. File: myapp.py import sys if __name__ == 'uwsgi_file_myapp': assert 'uwsgi_file_myapp' in sys.modules, ( 'other modules can also see what the main module is' ) connect_to_redis()  See What __name__ string does uwsgi use? for more details. Do something after uwsgi has forked a worker process Useful for doing one-time initialisations per worker process. Such as connecting to a database. try: import uwsgi except ImportError: def postfork(func): """Dummy decorator for unit tests""" return func else: from uwsgidecorators import postfork DB = None @postfork def init_db(): global DB from datetime import datetime DB = f'some-database connection, set @ {datetime.now().isoformat()}' def application(env, start_response): start_response('200 OK', [('Content-Type','text/html')]) return [f'{DB=!r}'.encode()]  This program will need to be run with the --master option. eg. uwsgi --http :8080 --master --wsgi-file myapp.py
Question: I want to plot a figure in pyplot that has 3 subplots, arranged vertically. The aspect ratios of the first one is 1:1, while for the other two it is 2:1. And the heights of each of these plots should be the same. This would mean that the left and right boundaries of the 1st plot are quite far away from the boundaries of the canvas. Could someone please help me with how to go about doing this? I have tried meddling with the hspace command, but that ends up changing the spacing of all the subplots, not just the first one. I have also tried the following by creating subplots within a subplot, but the wspace doesnt seem to make any difference in the plots: fig = plt.figure(figsize=(3.385, 2*3)) outer = gridspec.GridSpec(2, 1, wspace=2.0, hspace=0.1, height_ratios=[1,3]) inner1 = gridspec.GridSpecFromSubplotSpec(1, 1, subplot_spec=outer[0], wspace=0.1, hspace=0.1) inner2= gridspec.GridSpecFromSubplotSpec(2, 1, subplot_spec=outer[1], wspace=0.1, hspace=0.1) ax1 = plt.Subplot(fig, inner1[0]) (ax2,ax3)= plt.Subplot(fig, inner2[0]), plt.Subplot(fig, inner2[1]) fig.add_subplot(ax1); fig.add_subplot(ax2); fig.add_subplot(ax3) ax1.plot(x1, y1, "-", color="k" ) ax2.plot(x2, y2, "-") ax3.plot(x3,y3, "--) plt.show()   Could someone please help me with how this could be done?

Answer: Create 3 vertically arranged subplots, their heights will by equal be default. Then set the aspect ratios as desired. import matplotlib.pyplot as plt fig, axes = plt.subplots(nrows=3) axes[0].set_aspect(1) axes[1].set_aspect(.5) axes[2].set_aspect(.5)
Question: I want to create a categorical column, where each category has a descriptive name for self-documentation. I have a list of integers equivalent to the physical values in the categorical column, and I want to make the categorical column without creating an intermediate list of strings to pass to pl.Series. import polars as pl dt = pl.Enum(["0", "1", "2"]) s1 = pl.Series(["0", "0", "2", "1"], dtype=dt) physical = list(s1.to_physical()) print(f"{physical=}") s2 = pl.Series([str(p) for p in physical], dtype=dt) assert s1.equals(s2) # turning physical to strings just to create the series which is stored as ints is a waste of compute power # how to construct a series from the physical values? s2 = pl.Series.from_physical(physical, dtype=dt) assert s1.equals(s3)  This prints physical=[0, 0, 2, 1]  Then it errors because Series.to_physical doesn't exist. Is there a function like from_physical that would make this snippet run to completion without erroring on the final assertion?

Answer: You can simply cast to the enum datatype. assert s1.equals(s1.to_physical().cast(dt)) # True
Question: Scenario: With a dataframe with duplicated indices, I want to groupby while keeping the max value. I found the solution to this in Drop duplicates by index, keeping max for each column across duplicates however, this gets the max value of each column. This mixed the data of different rows, keeping the max values. Question: If instead of mixing the values of different rows, I want to keep a single row, where the value of a column "C" is the highest among the rows with the same index (in this case I will select the row with the highest value in "C" and keep all values for that row, not mixing with high values of other columns from other rows), how should the groupby be performed? What I tried: From the question linked, I got df.groupby(df.index).max()  and tried to modify it to: df.groupby(df.index)['C'].max()  but this deletes the other columns of the dataframe.

Answer: You don't provide a sample of your data so I'm just going for a general approach. That said, you can sort the dataframe by C, then groupby with head: # this assumes that index has only one level df.sort_values('C', ascending=False).groupby(level=0).head(1)  Or: df.sort_values('C').groupby(level=0).tail(1)  Also take a look at this related question (not by the index, but a column):
Question: What I want to achieve is a more elegant and direct method of annotating points with x and y position from a pandas dataframe with a corresponding label from the same row. This working example works and results in what I want, but I feel there must be a more elegant solution out there without having to store individual columns in separate lists first and having to iterate over them. My concern is that having these separate lists could results in misalignment of labels with data in cases of larger and complicated datasets with missing values, nans, etc. In this example, x = Temperature, y = Sales and the label is the Date. import pandas as pd import matplotlib.pyplot as plt d = {'Date': ['15-08-24', '16-08-24', '17-08-24'], 'Temperature': [24, 26, 20], 'Sales': [100, 150, 90]} df = pd.DataFrame(data=d)  Which gives:  Date Temperature Sales 0 15-08-24 24 100 1 16-08-24 26 150 2 17-08-24 20 90  Then: temperature_list = df['Temperature'].tolist() sales_list = df['Sales'].tolist() labels_list = df['Date'].tolist() fig, axs = plt.subplots() axs.scatter(data=df, x='Temperature', y='Sales') for i, label in enumerate(labels_list): axs.annotate(label, (temperature_list[i], sales_list[i])) plt.show()  What I aim for - but does not work - is something along the lines of: fig, axs = plt.subplots() axs.scatter(data=df, x='Temperature', y='Sales') axs.annotate(data=df, x='Temperature', y='Sales', text='Date') # this is invalid plt.show()  Suggestions welcome. If there is no way around the iterative process, perhaps there is at least a fail-safe method to warrant correct attribution of labels to corresponding data points.

Answer: You probably can't avoid the iteration, but you can remove the need to create lists by using df.iterrows(). This has the added benefit that you are not decoupling any data from your DataFrame. import pandas as pd import matplotlib.pyplot as plt d = {'Date': ['15-08-24', '16-08-24', '17-08-24'], 'Temperature': [24, 26, 20], 'Sales': [100, 150, 90]} df = pd.DataFrame(data=d) fig, axs = plt.subplots() axs.scatter(data=df, x='Temperature', y='Sales') for i, row in df.iterrows(): axs.annotate(row["Date"], (row["Temperature"], row["Sales"])) plt.show()
Question: I was surprised to find out that both the two call in main.py works: import package_top package_top.module.hello() # I thought this won't work ... package_top.hello() # I thought this is the only way  package_top package_top/ ├── __init__.py └── module.py  __init__.py from .module import hello  module.py def hello(): print("module_hello")  From my understanding, when I call import package_top in main.py, it will run __init__.py which adds hello to namespace package_top, and then package_top is added to namespace main, so I can access hello with package_top.hello(). What I don't understand is why the second call also works? I expect the prgram to throw out an error at the second line.

Answer: The Python import machinery modifies the package_top module object and adds the module attribute as part of importing the child module. See the Python import docs on submodules:  When a submodule is loaded using any mechanism (e.g. importlib APIs, the import or import-from statements, or built-in __import__()) a binding is placed in the parent module’s namespace to the submodule object.  Other import statements such as from other_package import x, y only add the x and y names to the importing module object. In your case, from .module import hello, is both importing the function hello, and also importing the submodule module of package_top, so your import statement creates both package_top.hello and package_top.module, which in turn gives you access to package_top.module.hello. You can confirm this by printing the module scope after importing: from .module import hello print(dir())  prints: ['__builtins__', (more default __stuff__ ...), 'hello', 'module']  Therefor package_top.module.hello() works without problems. Remove the from .module import hello statement and you will see the submodule is not imported, and the package_top.module.hello() statement will fail.
Question: The following is the minimal code. A spurious zero is printed between second and third bars that I am unable to get rid of in the plot. Please help me fix the code. A minimal working example is below:  import pandas as pd import matplotlib.pyplot as plt import seaborn as sns data = { 'Temp': [5, 10, 25, 50, 100, 5, 10, 25, 50, 100, 5, 10, 25, 50, 100, 5, 10, 25, 50, 100], 'Measurement': ['mean', 'mean', 'mean', 'mean', 'mean', 'std', 'std', 'std', 'std', 'std', 'min', 'min', 'min', 'min', 'min', 'max', 'max', 'max', 'max', 'max'], 'Value': [-0.03, -0.05, -0.07, -0.09, -0.09, 0.24, 0.44, 0.69, 1.11, 1.45, -2.36, -4.56, -5.86, -11.68, -14.68, 1.30, 3.50, 5.26, 9.28, 11.14] } df_melted = pd.DataFrame(data) print(df_melted) barplot = sns.barplot(x='Temp', y='Value', hue='Measurement', data=df_melted, palette='viridis') for p in barplot.patches: height = p.get_height() x = p.get_x() + p.get_width() / 2. if height >= 0: barplot.text(x, height + 0.1, f'{height:.1f}', ha='center', va='bottom', fontsize=10, color='black', rotation=90) else: barplot.text(x, height - 0.1, f'{height:.1f}', ha='center', va='top', fontsize=10, color='black', rotation=90) plt.ylim([-25,25]) plt.show()  Verions: Python 3.12.4, matplotlib 3.9.1 and Seaborn 0.13.2 PS: Edited to include maplotlib version.

Answer: The additional 0.0 is from the legend patches (in fact four 0.0s on top of each other). You can iterate over the bar containers instead: for c in barplot.containers: for p in c: height = p.get_height() x = p.get_x() + p.get_width() / 2. if height >= 0: barplot.text(x, height + 0.1, f'{height:.1f}', ha='center', va='bottom', fontsize=10, color='black', rotation=90) else: barplot.text(x, height - 0.1, f'{height:.1f}', ha='center', va='top', fontsize=10, color='black', rotation=90)   An easier way to label the bars is, howver, using bar_label instead of re-inventing the wheel. for c in barplot.containers: barplot.bar_label(c, fmt='%.1f', rotation=90, padding=2)