eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Given a dataframe:  a b c u 5 0 3 v 3 7 9 w 3 5 2  I'd like to select rows/columns in a dataframe using a function. This function gets the dataframe and returns a tuple of lists of labels, e.g. it returns ['v', 'u'], ['c'] using this lambda: get_labels = lambda df: (['v', 'u'], ['c'])  If I use .loc with the label literals, I get the desired result: df.loc[['v', 'u'], ['c']] c v 9 u 3  The documentation says about using a function (a callable), that a function returning two lists of labels is valid ("one of the above"):  A callable function with one argument (the calling Series or DataFrame) and that returns valid output for indexing (one of the above)  But if I use .loc with the function, which I believe returns the same list tuple, I get an error: df.loc[get_labels] unhashable type: 'list'  What's wrong? (as a possible clue: Pandas seems trying to create an Index object with the result of the function -- don't know why, no explanation in the documentation.)  Full code: import numpy.random as npr import pandas as pd npr.seed(0) df = pd.DataFrame(npr.randint(10, size=(3,3)), index=['u','v','w'], columns=['a','b','c']) get_labels = lambda df: (['v', 'u'], ['c']) print(df.loc[['v', 'u'], ['c']]) print(df.loc[get_labels]) 

Answer: As noted in the loc documentation, you need to pass:   A single label, e.g. 5 or 'a', (note that 5 is interpreted as a label of the index, and never as an integer position along the index). A list or array of labels, e.g. ['a', 'b', 'c']. A slice object with labels, e.g. 'a':'f'. A boolean array of the same length as the axis being sliced, e.g. [True, False, True]. An alignable boolean Series. The index of the key will be aligned before masking. An alignable Index. The Index of the returned selection will be the input. A callable function with one argument (the calling Series or DataFrame) and that returns valid output for indexing (one of the above)   Although maybe a bit ambiguous in the documentation, all of these are applicable to one axis. df.loc['a'] # single label df.loc[['a', 'b', 'c']] # list of labels df.loc['a':'f'] # slice df.loc[[True, False, True]] # list of booleans df.loc[pd.Series([True], index=['a']) # series df.loc[pd.Index(['a'])] # index df.loc[lambda x: ['a', 'b']] # callable  In your case, the callable returns two indexers, making it as if you were passing them to the index locator: df.loc[(['v', 'u'], ['c']), :]  You would need one callable per axis: get_index = lambda x: ['u', 'v'] get_col = lambda x: ['c'] df.loc[get_index, get_col]   You need to call your function: df.loc[get_labels(df)]  To use in a chain: df.pipe(lambda x: x.loc[get_labels(x)])  In my opinion, it might be better to have the function perform the selection directly: def get_labels(df): return df.loc[['v', 'u'], ['c']] df.pipe(get_labels)  Output:  c v 9 u 3
Question: I want to store this data frame df = pd.DataFrame({ 'id':[1,1,2,2], 'gender':["m","m","f","f"], 'val1':[1,2,5,6], 'val2':[3,4,7,8] }).set_index(['id','gender'])  as a json file that contains lists of dictionaries like this: d = [{'id':1, 'gender':"m", 'val1':[1,2], 'val2':[3,4]}, {'id':2, 'gender':"f", 'val1':[5,6], 'val2':[7,8]}]  All my attempts to use variations of df.to_dict(orient = '...') or df.to_json(orient = '...') did not produce the desired output.

Answer: You first need to aggregate as list with groupby.agg and reset_index: (df.groupby(['id', 'gender']).agg(list) .reset_index().to_json(orient='records') )  Output: [{"id":1,"gender":"m","val1":[1,2],"val2":[3,4]},{"id":2,"gender":"f","val1":[5,6],"val2":[7,8]}]
Question: I'm making some console app on Python with Curses (window-curses) library. At some point I need to save window state (or maybe whole terminal state) to some object/variable and restore it in future. What's the proper way to do this? I found methods in module documentaion which do this via saving state to the file. But maybe there exists some other way to do this in memory.

Answer: As you found, you can use things like the putwin function to save a pad to a file and getwin to restore it from a file. If you want to keep this in memory, rather than a file on disk, you can use a BytesIO object in place of a real file handle. import io import curses curses.initscr() pad = curses.newpad(100, 100) # ... do things with the pad # save the pad in memory f = io.BytesIO() pad.putwin(f) # later recall the data f.seek(0) # reset the cursor to the beginning of the "file" pad.getwin(f)  You could also write some functions to describe this another way: def save_win(win) -> bytes: f = io.BytesIO() win.putwin(f) bytes_data = f.getvalue() return bytes_data def load_win(bytes_data: bytes) -> curses.window: f = io.BytesIO(bytes_data) f.seek(0) return curses.getwin(f)
Question: Here the server side code with python & flask: from flask import Flask, request, send_file import io import zipfile import bitstring app = Flask(__name__) @app.route('/s/', methods=['POST']) def return_files_tut(): try: f = io.BytesIO() f.write("abcd".encode()) return send_file(f, attachment_filename='a.ret.zip') except Exception as e: return str(e) if __name__ == '__main__': app.run(debug=True)  Bellowing is curl command: λ curl -X POST http://localhost:5000/s/ -o ret.tmp % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 4 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 curl: (18) transfer closed with 4 bytes remaining to read  How should I use curl to receive the file?

Answer: When data is written, the pointer moves to the next position in the file. This means that before the data is read to make it available as a download, the pointer points to the end and no further data is read. So the download fails. In order to offer the file as a download, the pointer within the file must be reset to the beginning using seek(0). The attachment_filename attribute of send_file() is now deprecated and has been replaced by download_name. @app.post('/s/') def return_files_tut(): f = io.BytesIO() f.write("abcd".encode()) f.seek(0) return send_file(f, download_name='a.ret.zip')
Question: Assume groups will have more than n memebers, I want to take every nth row from each group. I looked at https://pandas.pydata.org/pandas-docs/version/0.17.0/generated/pandas.core.groupby.GroupBy.nth.html but that only takes one row from each group. For example:  import pandas as pd x = pd.DataFrame.from_dict({'a': [1, 1, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3], 'b': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]}) a b 0 1 1 1 1 2 2 2 3 3 2 4 4 2 5 5 3 6 6 3 7 7 3 8 8 3 9 9 3 10 10 3 11 11 3 12  And if we are keeping every 2nd row, this would be the desired outcome:  a b 1 1 2 3 2 4 6 3 7 8 3 9 10 3 11 

Answer: Use GroupBy.cumcount with modulo and compare by 1: out = x[x.groupby('a').cumcount() % 2 == 1] print (out) a b 1 1 2 3 2 4 6 3 7 8 3 9 10 3 11  How it working: print (x.assign(counter=x.groupby('a').cumcount(), mod2 = x.groupby('a').cumcount() % 2, mask = x.groupby('a').cumcount() % 2 == 1)) a b counter mod2 mask 0 1 1 0 0 False 1 1 2 1 1 True 2 2 3 0 0 False 3 2 4 1 1 True 4 2 5 2 0 False 5 3 6 0 0 False 6 3 7 1 1 True 7 3 8 2 0 False 8 3 9 3 1 True 9 3 10 4 0 False 10 3 11 5 1 True 11 3 12 6 0 False  Another idea with iloc in GroupBy.apply: out = x.groupby('a', group_keys=False)[x.columns].apply(lambda x: x.iloc[1::2])
Question: I'm trying to convert a python dictionary into an excel file. But I didn't manage to format the dataframe on a way that is how I want the excel file to look like. I have a dict in the shape as below: data = { 'Key 1': { 'PP': {'A': 105.08, 'B': 9.03, 'C': 0.12, 'D': 3.18, 'E': 0.5}, 'RP': {'A': 43.35, 'B': 6.92, 'C': -0.13, 'D': 3.03, 'E': -0.1}, 'SC': {'A': 36.15, 'B': 7.3, 'C': -0.01, 'D': 2.32, 'E': 0.34} }, 'Key2': { 'PP': {'A': 616.68, 'B': 11.09, 'C': 15.47, 'D': 3.82, 'E': 12.45}, 'RP': {'A': 416.92, 'B': 7.77, 'C': 6.48, 'D': 2.78, 'E': 5.25}, 'SC': {'A': 298.54, 'B': 9.57, 'C': 13.67, 'D': 3.51, 'E': 10.67} } }  And I got almost what I wanted in the output using this code: df = pd.DataFrame.from_dict( {(i,j):data[i][j] for i in data.keys() for j in data[i]}, orient = 'index' )  The output looks like:  And I want this to look like: 

Answer: From your current approach, you can chain unstack and post-process the columns with swaplevel and sort_index: df = (pd.DataFrame.from_dict( {(i,j):data[i][j] for i in data.keys() for j in data[i]}, orient = 'index' ) .unstack().swaplevel(axis=1).sort_index(axis=1) )  Or, change the dictionary comprehension to: out = pd.DataFrame.from_dict({k: {(k1, k2): v for k1, d2 in d.items() for k2, v in d2.items()} for k, d in data.items()}, orient='index')  Output:  PP RP SC A B C D E A B C D E A B C D E Key 1 105.08 9.03 0.12 3.18 0.50 43.35 6.92 -0.13 3.03 -0.10 36.15 7.30 -0.01 2.32 0.34 Key2 616.68 11.09 15.47 3.82 12.45 416.92 7.77 6.48 2.78 5.25 298.54 9.57 13.67 3.51 10.67
Question: I would like to keep columns that contains word "FAIL". Input data:    Values1 Values2 Values3 Status1 Status2 Status3     1 1 1 PASS PASS FAIL   2 2 2 PASS PASS PASS   3 3 3 PASS PASS PASS   4 4 4 PASS FAIL PASS    Expected output:    Status2 Status3     PASS FAIL   FAIL PASS    Current Output:    Status1 Status2 Status3     PASS PASS FAIL   PASS FAIL PASS    My code: import pandas as pd values = range(1,5) status_pass = ["PASS"]*len(values) status1 = status_pass[1:]+["FAIL"] status2 = status1[::-1] df = pd.DataFrame({"Values1":values,"Values2":values,"Values3":values,"Status1":status_pass,"Status2":status1,"Status3":status2}) # drop unwanted rows words_to_keep = ["FAIL"] df = df[df.stack().groupby(level=0).apply( lambda x: all(x.str.contains(w, case=False).any() for w in words_to_keep))] # Filter by column name df = df.filter(like='Status', axis=1) 

Answer: Use boolean indexing with any: df.loc[:, df.eq('FAIL').any()] # or for multiple words # the mask doesn't matter as long # as you have True/False df.loc[:, df.isin(words_to_keep).any()]  Output:  Status2 Status3 0 PASS FAIL 1 PASS PASS 2 PASS PASS 3 FAIL PASS  How it works: df.eq('FAIL').any() Values1 False Values2 False Values3 False Status1 False Status2 True Status3 True dtype: bool # which is equivalent to df.loc[:, [False, False, False, False, True, True]]
Question: I am creating a program in Customtkinter where I want to the user to input their ingredients one ingredient at a time- ideally I want to have the text field and then they have the option to add additional text fields below the original while retaining the original. I didn't want to set a fixed number of fields as I won't know in advance how many they will be inputting- if this isn't possible I have alternate fixes but I was looking for where I should be looking in the documentation to resolve my issue. The goal is for the form to add on additional elements until they are finished and they can save the whole form. I want the user to be able to edit the different elements of the form and then submit all at once. import customtkinter as ctk ctk.set_appearance_mode("Dark") ctk.set_default_color_theme("dark-blue") class root(ctk.CTk): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) self.title("GenSoft") self.geometry("800x480") self.resizable() self.state("normal") self.iconbitmap() self.grid_columnconfigure(1, weight=1) self.grid_columnconfigure((2, 3), weight=0) self.grid_rowconfigure((0, 1, 2), weight=1) self.tabview = ctk.CTkTabview(self, width=800, height=480, corner_radius=20, fg_color="#ffffff", segmented_button_selected_color="red", segmented_button_fg_color="black", segmented_button_selected_hover_color="green", segmented_button_unselected_hover_color="green") self.tabview.pack(padx=20, pady=20) self.tabview.add("Home") self.tabview.add("New Recipe") self.tabview.add("Saved Recipe") self.tabview.tab("Home").grid_columnconfigure(0, weight=1) # configure grid of individual tabs self.tabview.tab("New Recipe").grid_columnconfigure(0, weight=1) self.tabview.tab("Saved Recipe").grid_columnconfigure(0, weight=1) self.ingredientEntry = ctk.CTkEntry(self.tabview.tab("New Recipe"), placeholder_text="ingredient") self.ingredientEntry.pack(padx=20, pady=10) if __name__ == "__main__": app = root() app.mainloop() main()  It is very basic at the moment- I haven't managed to find any examples of what I am looking for but I feel like it is a fairly common feature that is implemented.

Answer: Add a button that creates a new entry field each time it's clicked: class root(ctk.CTk): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) # Your already existing code self.ingredientEntries = [] # The list that stores all the ingredient entries self.addButton = ctk.CTkButton(self.tabview.tab("New Recipe"), text="Add Ingredient", command=self.add_ingredient) self.addButton.pack(padx=20, pady=10) def add_ingredient(self): newEntry = ctk.CTkEntry(self.tabview.tab("New Recipe"), placeholder_text="ingredient") newEntry.pack(padx=20, pady=10) self.ingredientEntries.append(newEntry) # Add the new entry to your list  Hopefully it works out for you as intended 😊.
Question: I have a dataframe like so (simplified for this example)  Site LocationName Resource# 01 Test Name 5 01 Testing 6 02 California 10 02 Texas 11 ...  Each site has their own mapping for LocationName and Resource# For example:  Site 01 has a mapping of {'Test Name': 'Another Test', 'Testing': 'RandomLocation'} for LocationName and a mapping of {5: 5000} for Resource# Site 02 has a mapping of {'California': 'CA-123'} for LocationName and a mapping of {10: '10A', 11: '11B'} for Resource#  I am trying to map each respective site with their mappings for the different columns. If the mapping does not exist, I want the field to be None/blank. My ideal output is: Site# LocationName Resource# 01 Another Test 5000 01 RandomLocation 02 CA-123 10A 02 11B  My idea was to filter for each site and run map on the series df01 = df[df.Site == '01'] df01 = df['LocationName'].map({'Test Name': 'Another Test', 'Testing': 'RandomLocation'})  But this returns SettingWithCopyWarning since I am performing these operations on a copy. Is there a simple way to achieve this?

Answer: Yes there is. You can use the map function within the apply method on your dataframe. import pandas as pd data = { 'Site': ['01', '01', '02', '02'], 'LocationName': ['Test Name', 'Testing', 'California', 'Texas'], 'Resource#': [5, 6, 10, 11] } df = pd.DataFrame(data) mappings = { '01': { 'LocationName': {'Test Name': 'Another Test', 'Testing': 'RandomLocation'}, 'Resource#': {5: '5000', 6: None} }, '02': { 'LocationName': {'California': 'CA-123', 'Texas': None}, 'Resource#': {10: '10A', 11: '11B'} } } def apply_mappings(row): site = row['Site'] if site in mappings: location_map = mappings[site]['LocationName'] row['LocationName'] = location_map.get(row['LocationName']) resource_map = mappings[site]['Resource#'] row['Resource#'] = resource_map.get(row['Resource#'], None) return row df = df.apply(apply_mappings, axis=1) print(df)  which gives you your expected output:  Site LocationName Resource# 0 01 Another Test 5000 1 01 RandomLocation None 2 02 CA-123 10A 3 02 None 11B
Question: I have to pip install dependencies after setting up a virtual environment for an intro python course and I'm getting an error message: ERROR: Could not find a version that satisfies the requirement sourcedefender==10.0.13 (from versions: 12.0.2, 12.0.3) ERROR: No matching distribution found for sourcedefender==10.0.13 It has a requirements.txt file that contains all dependencies. I looked on the sourcedefender release page on pypi and it doesn't have any releases listed besides 12.0.2 and 12.0.3. I installed the same dependencies for a similar assignment last week and had no issues. I've tried Python 3.10.11 (which is the version I've been using all semester) and 3.8.10. When I put sourcedefender== 12.0.2 in the requirements.txt file, it says it needs updated versions of other dependencies, and when I edit those into the requirements.txt file, it gives an error about a NoneType and seems to just break. I'm not sure why this is happening, can someone help me? Is it possible that they removed the sourcedefender release that I need?

Answer: I can't seem to comment as I dont have enough reputation so I will leave my answer here. Sometimes the right answer is the most obvious one.  Someone probably made a mistake or typo.  or this version used to be hosted on pypi and was recently taken down.   Do you know what version you installed for your previous assignment? When you installed it previously, the files might have been downloaded locally so you wouldnt have to get them from pypi. I also looked into it and tried to install it and got this error when running pip3 install sourcedefender==10.0.13 ERROR: Could not find a version that satisfies the requirement sourcedefender==10.0.13 (from versions: 12.0.2, 12.0.3) ERROR: No matching distribution found for sourcedefender==10.0.13  I also checked the pypi and you are right there are only 2 versions as written in the error above. You should contact your lecturer and ask them whats up. Don't waste too much time on this. If you are expected to solve this yourself. Then use version 12.0.2 and try to see which packages need to be upgraded and slowly change their versions until it works.
Question: I am trying to go over a loop to add columns into a empty dataframe. each column might have a different length. Look like the final number of rows are defined by lenght of first column added. The columns with a Longer length will be cut values. How to always keep all the values of each column when column length are different? Thanks Here is case 1 with first column has lower length, then 2nd column's value will be cut import pandas as pd df_profile=pd.DataFrame() df_profile['A']=pd.Series([1,2,3,4]) df_profile['B']=pd.Series([10,20,30,40,50,60]) print(df_profile) A B 0 1 10 1 2 20 2 3 30 3 4 40  Here are case 2 with first column has highest length, then it is find for other columns import pandas as pd df_profile=pd.DataFrame() df_profile['A']=pd.Series([1,2,3,4,5,6,7,8]) df_profile['B']=pd.Series([10,20,30,40,50,60]) df_profile['C']=pd.Series([100,200,300,400,500,600]) df_profile['D']=pd.Series([100,200]) print(df_profile) A B C D 0 1 10.0 100.0 100.0 1 2 20.0 200.0 200.0 2 3 30.0 300.0 NaN 3 4 40.0 400.0 NaN 4 5 50.0 500.0 NaN 5 6 60.0 600.0 NaN 6 7 NaN NaN NaN 7 8 NaN NaN NaN 

Answer: You can use pd.concat to add another Series, e.g.: # you have already this dataframe: df_profile = pd.DataFrame() df_profile["A"] = pd.Series([1, 2, 3, 4]) # you can use pd.concat to add another Series: out = pd.concat([df_profile, pd.Series([10, 20, 30, 40, 50, 60], name="B")], axis=1) print(out)  Prints:  A B 0 1.0 10 1 2.0 20 2 3.0 30 3 4.0 40 4 NaN 50 5 NaN 60
Question: Here is a recursive R function involving a matrix S from the parent environment: f <- function(m, k, n) { if(n == 0) { return(100) } if(m == 1) { return(0) } if(!is.na(S[m, n])) { return(S[m, n]) } s <- f(m-1, 1, n) i <- k while(i <= 5) { if(n > 2) { s <- s + f(m, i, n-1) } else { s <- s + f(m-1, 1, n-1) } i <- i + 1 } if(k == 1) { S[m, n] <- s } return(s) }  Here is the result of a call to this function: > n <- 4 > S <- matrix(NA_real_, nrow = n, ncol = n) > f(n, 1, n) [1] 127500  Now, here is the Python version: import numpy as np def f(m, k, n): if n == 0: return 100 if m == 1: return 0 if S[m-1, n-1] is not None: return S[m-1, n-1] s = f(m-1, 1, n) i = k while i <= 5: if n > 2: s = s + f(m, i, n-1) else: s = s + f(m-1, 1, n-1) i = i + 1 if k == 1: S[m-1, n-1] = s return s  The call: >>> n = 4 >>> S = np.full((n, n), None) >>> f(n, 1, n) 312500  The Python result is different from the R result. Why? One gets identical results if one replaces  if S[m-1, n-1] is not None: return S[m-1, n-1]  with  if k == 1 and S[m-1, n-1] is not None: return S[m-1, n-1]  I also have a C++ version of this function, and it has the same behavior as the R function.

Answer: Side-effects such as a function modifying external state are a code smell. Your R code doesn't do this but your Python code does. Assignment to outer scope variables within a function scope can behave differently in R to Python (and actually within R and Python depending on whether the object is mutable). Instinctively I feel uncomfortable that without passing S as an argument you're modifying it within the function, with S[m, n] <- s in R and S[m-1, n-1] = s in Python. Here's a simpler example where we define S as a matrix containing one value, 0, which we increment every time we call a recursive function. The function, f(), has one parameter, k. It calls itself k times, incrementing the only value in S every time. First in R: S <- matrix(0) f <- function(k) { if (k == 1) { return(S) } S[1,1] <- S[1,1] + 1 f(k - 1) } f(5) # [,1] # [1,] 0  The output is 0 and will always be zero no matter how many times you call the function, as S is only modified in the function scope. Conversely, in Python: S = np.zeros((1,1)) def f(k): if k == 1: print(S) return S[0,0] += 1 f(k - 1) f(5) # [[4.]]  As you can see, even though we modify S[0,0] within the function and do not pass it as a parameter, this actually modifies it in the outer scope (or parent environment in R terms). This may or may not be the reason for the difference in your example - your code has a lot of moving parts. But I would not expect R and Python to behave the same way when modifying variables in this way. More generally, I think you'd need a pretty compelling reason to write code which references (let alone modifies) variables from within a function without passing them as arguments.
Question: I am updating pandas version from 1.3.5 to 2.2.2 on an old project. I am not very familiar with pandas, and I am stuck with a Future Warning: FutureWarning: The behavior of 'to_datetime' with 'unit' when parsing strings is deprecated. In a future version, strings will be parsed as datetime strings, matching the behavior without a 'unit'. To retain the old behavior, explicitly cast ints or floats to numeric type before calling to_datetime. Here is the causes the error: >>> from io import StringIO >>> import json >>> import pandas >>> >>> str = '{"Keyword":{"0":"TELECOM","1":"OPERATOR","Total":"Total"},"Type":{"0":"job_field","1":"title_class","Total":null},"Seniority Score":{"0":0.0,"1":-1.0,"Total":-1.0}}' >>> df = pandas.read_json(StringIO(str), typ='frame', orient='records')  From what I get, it has to do something with integers and floats, probably being represented as strings in the json, but I tried various combinations and can not get arround the warning. I am confused becase I am not calling the to_datetime function at all.

Answer: This error is due to having a mix of number-like and strings in the index. A minimal reproducible example that triggers the error would be: s = '{"A":{"0":"X","Y":"Y"}}' pandas.read_json(StringIO(s), typ='frame', orient='records')  Pandas uses to_datetime to infer the dtype of the index, which triggers the warning. This should most likely be considered a minor bug since this only triggers a warning. Maybe still worth reporting but it might be that no action will be taken.
Question: I have a buffer of float16 elements in the memory I read this buffer to a list but since I use np.ctypeslib I can't read it as float16 so I read it as uint16 and now I want to represent it as float16 simple convert by list.astype(float16) won't help I found this table https://gist.github.com/gregcotten/9911bda086c5850cb513b2980cca7648 and according to that table it works. I want to write a python function for convention unint16_as_float16 but don't know what are the rules. appreciate your help thanks!

Answer: I found a solution by numpy uint16_list.view(np.float16)