eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I want to create a new dataframe from my original dataframe that aggregates by two columns and has a calculated column dependent on the sum of selected rows from two other columns. Here is a sample df: df = pd.DataFrame([['A','X',2000,5,3],['A','X',2001,6,2],['B','X',2000,6,3],['B','X',2001,7,2],['C','Y',2000,10,4],['C','Y',2001,12,4],['D','Y',2000,11,2],['D','Y',2001,15,1]], columns=['ctry','rgn','year','val1, val2']))  and what it looks like:  ctry rgn year val1 val2 0 A X 2000 5 3 1 A X 2001 6 2 2 B X 2000 6 3 3 B X 2001 7 2 4 C Y 2000 10 4 5 C Y 2001 12 4 6 D Y 2000 11 2 7 D Y 2001 15 1  I ultimately want a new dataframe that gets rid of the ctry column and groups by the rgn and year, and has a calculated column value dependent on val1 and val2 such that the sum of the product of val1 and val2 is divided by the sum of val2 for a rgn and year: df['value'] = ∑(val1*val2)/∑val2 for each rgn and year  rgn year value 0 X 2000 5.5 1 X 2001 6.5 2 Y 2000 10.333333 3 Y 2001 12.6  I ended up successfully doing so: df['calc'] = df['val1'] * df['val2'] new_df = df.groupby(['rgn', 'year']).sum() new_df['value'] = new_df['calc']/new_df['val2'] new_df = new_df.reset_index().rename_axis(None, axis=1) new_df = new_df.drop(columns=['ctry', 'val1', 'val2', 'calc'])  However, I'd like to know if there is a more succinct way that doesn't require all these steps, perhaps using the lambda function. Appreciate any help I get. Thanks!

Answer: Pre-compute value = val1*val2, perform a groupby.sum, then compute value/val1: out = (df.eval('value = val1*val2') .groupby(['rgn', 'year'], as_index=False) [['value', 'val2']].sum() .assign(value=lambda x: x['value']/x.pop('val2')) )  Less efficient, but more flexible, you can also use groupby.apply: out = (df.groupby(['rgn', 'year']) .apply(lambda x: (x['val1']*x['val2']).sum()/x['val2'].sum(), include_groups=False) .reset_index(name='value') )  Output:  rgn year value 0 X 2000 5.500000 1 X 2001 6.500000 2 Y 2000 10.333333 3 Y 2001 12.600000
Question: Imagine a process, where we do several pandas groupbys. We start with a df like so: import numpy as np import pandas as pd np.random.seed(1) df = pd.DataFrame({ 'id': np.arange(10), 'a': np.random.randint(1, 10, 10), 'b': np.random.randint(1, 10, 10), 'c': np.random.randint(1, 10, 10) }) df Out[23]: id a b c 0 0 2 8 2 1 1 8 8 9 2 2 7 2 9 3 3 3 8 4 4 4 5 1 9 5 5 6 7 8 6 6 3 8 4 7 7 5 7 7 8 8 3 2 6 9 9 5 1 2  We perform a groupby 'a' (The 'max' agg is just an example) new_df = df.groupby('a').agg('max').reset_index() Out[25]: a id b c 0 2 0 8 2 1 3 8 8 6 2 5 9 7 9 3 6 5 7 8 4 7 2 2 9 5 8 1 8 9  And I want to keep track of the original id to which group it belongs. For example, id 0 belongs to a = 2, 1 to 8, 2 to 7, (3, 6, 8) belongs to 3 etc..  Afterword we perform another groupby: new_df.groupby('b').agg('max').reset_index() Out[28]: b a id c 0 2 7 2 9 1 7 6 9 9 2 8 8 8 9  Now we have a continued mapping, Where group a (2, 3, 8) belongs to group 8 (of b) (5, 6) = 7 7 = 2  And this result in a long map where the original id: 0 => a = 2 => b = 8 (where b = 8 is the final group that interests me) 1 => a = 8 => b = 8 2 => a = 7 => b = 2  And so on.. Now I do this in order to reduce a lot of entities in my data so that I can group them in the same bucket somehow. And I need to map them from their original id to a new id, that is being represented after many iterations of groupby. In the end I want to see something like so Out[32]: id grp 0 0 8 1 1 8 2 2 2 3 3 8 4 4 7 5 5 7 6 6 8 7 7 7 8 8 8 9 9 7  again, because id=0 went to a=2 and a=2 went to b=8.. Any solution or suggestion will be most welcome. Even carrying the values in a column dedicated to this, with each group by. And when we aggregate, we can do a set addition...

Answer: This looks to me like a graph problem. You could build a directed graph from successive groupby using networkx, then pair the roots/leaves of each subgraph: import networkx as nx G = nx.DiGraph() ref = 'id' cols = ['a', 'b'] for col in cols: tmp = df.groupby(ref, as_index=False)[[ref, col]].max() G.add_edges_from((((ref, r), (col, x)) for r, x in zip(tmp[ref], tmp[col]))) ref = col out = [] for c in nx.weakly_connected_components(G): H = G.subgraph(c) roots = [n for n, d in H.in_degree() if d==0] leaves = [n for n, d in H.out_degree() if d==0] out.extend([dict((r, l)) for r in roots for l in leaves]) out = (pd.DataFrame(out) .sort_values(by='id', ignore_index=True) )  Output:  id b 0 0 8 1 1 8 2 2 2 3 3 8 4 4 7 5 5 7 6 6 8 7 7 7 8 8 8 9 9 7  Graph:
Question: I have dataframe Polars with a lot of columns. I need to create new columnt with the result of my own function  df = pl.DataFrame({ 'Stock_name': ['reserve', 'fullfilment', 'ntl', 'ntl'], 'doc_type': ['sales', 'moving', 'sales', 'corr'], }) def doc_check(row): if (row['Stock_name'] == 'reserve') or (row['Stock_name'] == 'ntl'): if row['doc_type'] == 'sales': doc = 'sales' else: doc = row['doc_type'] else: doc = row['doc_type'] return doc df = df.apply(doc_check).alias('doc_type_new') df['doc_type_new'] = df.apply(doc_check)  But I received an error "TypeError: tuple indices must be integers or slices, not str". Where is an error in my code?

Answer: You can use pl.struct to extract the two columns and then use map_elements to map over it: df = df.with_columns( doc_type_new=pl.struct("Stock_name", "doc_type").map_elements( doc_check, return_dtype=pl.String ) ) print(df)  Output: shape: (4, 3) ┌─────────────┬──────────┬──────────────┐ │ Stock_name ┆ doc_type ┆ doc_type_new │ │ --- ┆ --- ┆ --- │ │ str ┆ str ┆ str │ ╞═════════════╪══════════╪══════════════╡ │ reserve ┆ sales ┆ sales │ │ fullfilment ┆ moving ┆ moving │ │ ntl ┆ sales ┆ sales │ │ ntl ┆ corr ┆ corr │ └─────────────┴──────────┴──────────────┘  Note that in this case, your function can be implemented using native polars API which will run much faster: df = df.with_columns( doc_type_new=pl.when( pl.col("Stock_name").is_in(["reserve", "ntl"]) & (pl.col("doc_type") == "sales") ) .then(pl.lit("sales")) .otherwise(pl.col("doc_type")) ) print(df)  (Same output.)
Question: I am creating a AI agent in my Jupyter Notebook on Anaconda. When I enter the following settings I get an error. import os os.environ("OPENAI_API_KEY")=openai_api_key os.environ("OPENAI_MODEL_NAME")="gpt-3.5-turbo" Cell In[47], line 1 os.environ("OPENAI_API_KEY")=openai_api_key ^ SyntaxError: cannot assign to function call here. Maybe you meant '==' instead of '='?  This seems to be a standard statement that works on lightning.ai or even VScode editor but why doesnt it work on my Jupyter notebook running on Localhost? I am running this on my Macbook. Next I was planning to create a few agents to create a bot to read and respond to a blog.

Answer: Use Square brackets instead of parentheses. os.environ["OPENAI_API_KEY"]=openai_api_key os.environ["OPENAI_MODEL_NAME"]="gpt-3.5-turbo"  It'll solve the issue.
Question: I'm trying to practice coding again and currently having problems with figuring out why my function doesn't work. Here's my sample Table for reference:    diastolic systolic     80.0 130.0   77.0 126.0   92.0 152.0   76.0 147.0   70.0 127.0   64.0 119.0   72.0 135.0   84.0 137.0   85.0 165.0   81.0 156.0    and here's the function I created to categorize the data: def blood_pressure_cat(diastolic, systolic): """ diastolic: float systolic: float """ if systolic <= 90 and diastolic <= 60: return "Hypotension" elif systolic < 120 and diastolic < 80: return "Normal" elif systolic < 130 and diastolic < 80: return "Elevated" elif systolic < 140 and diastolic < 90: return "Stage 1" elif systolic < 180 and diastolic < 90: return "Stage 2" elif systolic >= 180 and diastolic >= 90: return "Hypertensive Crisis" else: return np.nan  This is how I execute my function: blood_pressure_cat(DF["diastolic"], DF["systolic"])  And getting this error: ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()  If I change the "and"s in that function I get this one instead: Cannot perform 'rand_' with a dtyped [float64] array and scalar of type [bool]  Can somebody help me understand why it is not working or if there's any alternative way to run this code?

Answer: A very simple approach would be to vectorize your function, for example by decorating the function: import numpy as np @np.vectorize def blood_pressure_cat(diastolic, systolic): """ diastolic: float systolic: float """ if systolic <= 90 and diastolic <= 60: return "Hypotension" elif systolic < 120 and diastolic < 80: return "Normal" elif systolic < 130 and diastolic < 80: return "Elevated" elif systolic < 140 and diastolic < 90: return "Stage 1" elif systolic < 180 and diastolic < 90: return "Stage 2" elif systolic >= 180 and diastolic >= 90: return "Hypertensive Crisis" else: return np.nan blood_pressure_cat(DF["diastolic"], DF["systolic"])  Output: array(['Stage 1', 'Elevated', 'nan', 'Stage 2', 'Elevated', 'Normal', 'Stage 1', 'Stage 1', 'Stage 2', 'Stage 2'], dtype='<U8')  Note that the returned NaNs will be strings. To get real NaNs, instead of a decorator, manually wrap the function and pass the otypes=[object] parameter: blood_pressure_cat = np.vectorize(blood_pressure_cat, otypes=[object]) blood_pressure_cat(DF["diastolic"], DF["systolic"])  Output: array(['Stage 1', 'Elevated', nan, 'Stage 2', 'Elevated', 'Normal', 'Stage 1', 'Stage 1', 'Stage 2', 'Stage 2'], dtype=object)  As a new column: DF['cat'] = blood_pressure_cat(DF["diastolic"], DF["systolic"])  Also be aware that this vectorization is a wrapper around a python loop. Therefore you could also use pure python with zip: DF['cat'] = [blood_pressure_cat(d, s) for d,s in zip(DF['diastolic'], DF['systolic'])]  This gives a similar result than using apply with axis=1 but avoids creating a Series for each row (which is expensive). Output:  diastolic systolic cat 0 80.0 130.0 Stage 1 1 77.0 126.0 Elevated 2 92.0 152.0 NaN 3 76.0 147.0 Stage 2 4 70.0 127.0 Elevated 5 64.0 119.0 Normal 6 72.0 135.0 Stage 1 7 84.0 137.0 Stage 1 8 85.0 165.0 Stage 2 9 81.0 156.0 Stage 2
Question: I have a datetime field, and I want to filter the rows that has the same date value. models.py class EntryMonitoring(models.Model): student = models.ForeignKey('Student', models.DO_NOTHING) clockin = models.DateTimeField() clockout = models.DateTimeField(null=True)  views.py def check_attendance(request, nid): day = EntryMonitoring.objects.filter( clockout__isnull=False, '# same value', student=nid ).annotate(...) # do something  I wanted to add inside that filter query that clockin__date has the same value as clockout__date. Is that possible? If it is, what would be the right query to filter it?

Answer: Yes, it is possible. You should use the TruncDate function: from django.db.models import F from django.db.models.functions import TruncDate entries = EntryMonitoring.objects.filter( clockout__isnull=False, student=nid, clockin__date=TruncDate(F("clockout")) ).annotate(...)
Question: This is my DataFrame: import pandas as pd df = pd.DataFrame( { 'high': [110, 110, 101, 101, 115, 300], } )  And this is the expected output. Column bin should be created:  high bin 0 110 105.0 1 110 105.0 2 101 100.0 3 101 100.0 4 115 111.0 5 300 220.0  Basically bin is created by using pd.cut: import numpy as np evaluation_bins = [100, 105, 111, 120, 220, np.inf] df['bin'] = pd.cut(df['high'], bins=evaluation_bins, include_lowest=True, right=False)  This gives me the category itself but I want the left edge as the output. Honestly there was not much I could try to do. I could get the dtypes of df by df.dtypes but I don't know how to continue.

Answer: Pass evaluation_bins to labels inside pd.cut, without the final edge (np.inf): df['bin'] = pd.cut(df['high'], bins=evaluation_bins, labels=evaluation_bins[:-1], include_lowest=True, right=False)  Output:  high bin 0 110 105 1 110 105 2 101 100 3 101 100 4 115 111 5 300 220
Question: I'm working with a pandas DataFrame and I noticed a difference in behavior when using the in operator. Here’s an example to illustrate this: import pandas as pd df = pd.DataFrame({'a': [4, 5, 6], 'b': [7, 8, 9]}) print(1 in df) print(type(df)) print(1 in df["a"]) print(type(df["a"]))  Output: False <class 'pandas.core.frame.DataFrame'> True <class 'pandas.core.series.Series'>  The most straightforward difference is that of course one object is a DataFrame, the other is a Series; nonetheless, I was not really expecting 1 to be found in the Index of the Series and this expression evaluating to True. Especially as it is False for the DataFrame. Is there an explanation why I should have been expecting this?

Answer: In both cases the in operator calls __contains__ to test membership. Both pd.DataFrame and pd.Series are subclasses of NDFrame, which has this method defined as follows:  def __contains__(self, key) -> bool: """True if the key is in the info axis""" return key in self._info_axis  So, under the hood the following happens: print(df._info_axis) # Index(['a', 'b'], dtype='object') print(df.__contains__(1)) # False # approx.: 1 in ['a', 'b']  print(df['a']._info_axis) # RangeIndex(start=0, stop=3, step=1) print(df['a'].__contains__(1)) # True # approx.: 1 in [0, 1, 2]  I.e., the difference lies in the fact that a df uses df.columns as the 'info axis', while a series, such as df['a'], naturally must use series.index.
Question: I would like to write CSV data directly from a bytes (or string) object in memory to duckdb database file (i.e. I want to avoid having to write and read the temporary .csv files). This is what I've got so far: import io import duckdb data = b'a,b,c\n0,1,2\n3,4,5' rawtbl = duckdb.read_csv( io.BytesIO(data), header=True, sep="," ) con = duckdb.connect('some.db') con.sql('CREATE TABLE foo AS SELECT * FROM rawtbl')  which throws following exception: --------------------------------------------------------------------------- IOException Traceback (most recent call last) Cell In[1], line 10 5 rawtbl = duckdb.read_csv( 6 io.BytesIO(data), header=True, sep="," 7 ) 9 con = duckdb.connect('some.db') ---> 10 con.sql('CREATE TABLE foo AS SELECT * FROM rawtbl') IOException: IO Error: No files found that match the pattern "DUCKDB_INTERNAL_OBJECTSTORE://2843be5a66472f9c"  However, it is possible to do: >>> duckdb.sql('CREATE TABLE foo AS SELECT * FROM rawtbl') >>> duckdb.sql('show tables') ┌─────────┐ │ name │ │ varchar │ ├─────────┤ │ foo │ └─────────┘ >>> duckdb.sql('SELECT * from foo') ┌───────┬───────┬───────┐ │ a │ b │ c │ │ int64 │ int64 │ int64 │ ├───────┼───────┼───────┤ │ 0 │ 1 │ 2 │ │ 3 │ 4 │ 5 │ └───────┴───────┴───────┘  since rawtbl is a duckdb.duckdb.DuckDBPyRelation object. But that is the in-memory duckdb database, not the 'some.db' file. Question How to read csv data directly from bytes (or a string) to duckdb database file, without using intermediate CSV files? Versions duckdb 0.10.2 on Python 3.12.2 on Ubuntu

Answer: You can create the connection before you use read_csv and pass the connection into it. import io import duckdb from pathlib import Path data = b'a,b,c\n0,1,2\n3,4,5' db_path = 'some.db' Path(db_path).unlink(missing_ok=True) with duckdb.connect(db_path) as con: rawtbl = duckdb.read_csv( io.BytesIO(data), header=True, sep=",", connection=con, ) con.execute(''' CREATE TABLE foo as select * from rawtbl ''') with duckdb.connect(db_path) as con: res = con.sql('select * from foo') print(res) # ┌───────┬───────┬───────┐ # │ a │ b │ c │ # │ int64 │ int64 │ int64 │ # ├───────┼───────┼───────┤ # │ 0 │ 1 │ 2 │ # │ 3 │ 4 │ 5 │ # └───────┴───────┴───────┘
Question: I submitted a question a week ago about persistent processes after terminating the ProcessPoolExecutor, but there have been no replies. I think this might be because not enough people are familiar with how ProcessPoolExecutor is coded, so I thought it would be helpful to ask a more general question to those who use the multiprocessing module. In the Python documentation, it states that  On POSIX using the spawn or forkserver start methods will also start a resource tracker process which tracks the unlinked named system resources (such as named semaphores or SharedMemory objects) created by processes of the program. When all processes have exited the resource tracker unlinks any remaining tracked object.  However, there is nothing in the documentation stating how to shut down this resource tracker when it is no longer needed. As far as I can tell, the tracker PID is not available to the ProcessPoolExecutor, but I did read somewhere that it might be accessible using a Pool instead. Can anyone confirm if this is true before I refactor my code?

Answer: You may use an internal method _stop to achieve this, but ... it should be done with caution due to the potential risks involved while using internal and/or undocumented features, Below an example of code demonstrating what is said above: from concurrent.futures import ProcessPoolExecutor from multiprocessing import resource_tracker def example_function(x): return x * x if __name__ == '__main__': with ProcessPoolExecutor() as executor: results = list(executor.map(example_function, range(10))) # Manually stop the resource tracker resource_tracker._resource_tracker._stop() print("Resource tracker stopped.")  The code exits on my Linux Mint 21.2 Xfce machine with exit code 0, so I assume it does what it is intended to do.
Question: I'm looking for a good way to skip dictionaries that contain certain keys among multiple dictionaries that are different. Right now I'm chaining .get() methods on a dictionary object and continuing on whatever matches, which is working but its kinda messy. I'm doing something like: ... if my_dict.get('some_key', my_dict.get('other_key', my_dict.get('some_other_key'))): continue 

Answer: You can use any with a generator expression that iterates over the possible keys and tests if the dict has any of the keys: if any(key in my_dict for key in ('some_key', 'other_key', 'some_other_key')): continue  Alternatively, you can use set.isdisjoint to test if the set of keys is disjoint with the dict keys: if not {'some_key', 'other_key', 'some_other_key'}.isdisjoint(my_dict): continue  And as @deceze points out, you can also simply test if the set of keys intersects with the set of dict keys, which looks more readable: if {'some_key', 'other_key', 'some_other_key'}.intersection(my_dict): continue  But be aware that unlike set.isdisjoint, set.intersection creates a new set and does not short-circuit on the first match, so is more expensive both time and space-wise.
Question: I'm trying to create a tree in NiceGui where every element has a toggle button. This is what I got so far: from nicegui import events, ui def toggleUpdate(e: events.GenericEventArguments) -> None: print(tree._props['nodes']) tree = ui.tree([ {'id': 'numbers', 'description': 'Just some numbers', 'update': True, 'children': [ {'id': '1', 'description': 'The first number', 'update': False}, {'id': '2', 'description': 'The second number', 'update': True}, ]}, {'id': 'letters', 'description': 'Some latin letters', 'update': True, 'children': [ {'id': 'A', 'description': 'The first letter', 'update': True}, {'id': 'B', 'description': 'The second letter', 'update': True}, ]}, ], label_key='id', on_select=lambda e: ui.notify(e.value)) tree.add_slot('default-header', ''' <span :props="props">Node <strong>{{ props.node.id }}</strong></span> ''') tree.add_slot('default-body', r''' <span :props="props">Description: "{{ props.node.description }}" <q-toggle v-model="props.node.update" label="Elemente updaten" left-label @update:model-value="() => $parent.$emit('toggleUpdate', props.node)" /></span> ''') ui.button("Test", on_click=lambda: toggleUpdate("TEST")) tree.on('toggleUpdate', toggleUpdate) tree.on('update:model-value', lambda e: ui.notify(e.args)) ui.run()  The tree works fine, but the data is not changed when I toggle the button and also the toggleUpdate event is not triggered. What am I doing wrong?

Answer: The Author of NiceGui found the solution. You have to use $parent.$parent.$parent.$emit() for the toggle. Full answer: https://github.com/zauberzeug/nicegui/discussions/3085#discussioncomment-9462868
Question: I am trying to turn all of the white pixels in this image red, but when I run the program the shape is fine but the red pixels are rotated 90 degrees:  This is the current code that I am using to do this: import cv2 as cv import numpy as np import os from matplotlib import pyplot as plt import cv2 as cv def get_white_pixels(image_as_array): threshold = 40 indices = np.where(image_as_array >= threshold) width=image.shape[1] height=image.shape[0] cartesian_y=height-indices[0]-1 np_data_points=np.column_stack((indices[1],cartesian_y)) return cartesian_y, np_data_points, width,height image = cv.imread("framenumber0.jpg") ind, pixels, width, height = get_white_pixels(image) #Goes through every pixel and changes its values for i in range(0, len(pixels)): loc_x = int(pixels[i][0]) loc_y = int(pixels[i][1]) image[loc_x,loc_y] = (0,0,255) cv.imshow('Modified Image', image) cv.waitKey(0) cv.destroyAllWindows()  I do need the location of the white points as I will use them later for the second part of the project. I suspect the problem has something to do with the np.column_stack(). I have been reading the info page of the function but I still do not understand why this happens. If you want to replicate here is also the image that I am using: 

Answer: Change this line to:  # image[loc_x,loc_y] = (0,0,255) image[loc_y,loc_x] = (0,0,255)  along with  # cartesian_y=height-indices[0]-1 cartesian_y=indices[0]  You need to know that numpy and OpenCV arrays are y,x ones, where other image processing packages work the x,y way. Here the result of running the fixed code:  As mentioned in a comment by aRTy  You are effectively flipping the y-coordinate top-down in this line: cartesian_y=height-indices[0]-1. If you are converting from OpenCV/numpy orientation (where y=0 is top) to matplotlib orientation (where y=0 is bottom) then you need to do that when you reach the matplotlib section, not before.   The required effect of getting the thresholded image which is apparently a grayscale one can also be achieved using simplified code separating the channels and running OpenCV threshold on one of them: import cv2 as cv, numpy as np threshold = 40 image = cv.imread("srcImage.jpg") _, G, _ = cv.split(image) _, thresh = cv.threshold(G, threshold, 255, cv.THRESH_BINARY) black=np.zeros_like(thresh) red_image = cv.merge([black, black, thresh]) cv.imshow('White areas in source Image marked red', red_image) cv.waitKey(0) cv.destroyAllWindows()  The result appears to be the same even if two channels are skipped from consideration.