eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a Polars dataframe: df = pl.DataFrame( { "nrs": [1, 2, 3, None, 5], "names": ["foo", "ham", "spam", "egg", None], "random": np.random.rand(5), "A": [True, True, False, False, False], } )  How can I compute the ratio between the number of rows where A==True, to the number of rows where A==False? Note that A is always True or False. I found a solution, but it seems a bit clunky: ntrue = df.filter(pl.col('A')==1).shape[0] ratio = ntrue/(df.shape[0]-ntrue) 

Answer: You can leverage polars' expression API as follow. df.select(pl.col("A").sum() / pl.col("A").not_().sum()).item()  The summing works as A is a boolean column. If this is not the case, you can exchange pl.col("A") for another corresponding boolean expression.
Question: Say I have a dataframe with column ['Subtype'] and ['Building Condition'], Dataframe 1:    Subtype Building Condition     A Good   B Bad   C Bad    I want to map dataframe 1 using another dataframe based on the values of those two columns Dataframe 2:     Good Bad     A Repair Retrofit   B Retrofit Reconstruct   C Reconstruct Reconstruct    Iterate over the first dataframe and use the pd.df.at function for each row and append them to an empty list. #assignment of intervention based on subtype and building condition intervention_list = [] for index, row in bldg_df.iterrows(): # print(bldg_df.at[index, 'Subtype']) intervention = matrix_df.at[bldg_df['Subtype'][index], bldg_df['Building Condition'][index]] intervention_list.append(intervention) bldg_df['Intervention'] = intervention_list  So the resulting dataframe would be    Subtype Building Condition Intervention     A Good Repair   B Bad Reconstruct   C Bad Reconstruct    This worked but I'm just thinking if there's a faster and more efficient way of going about this. Maybe using the map or merge as_of function?

Answer: The documented approach in such case is to use indexing lookup on the underyling numpy array: # factorize the requested name idx, cols = pd.factorize(df1['Building Condition']) # reorder and slice df1['Intervention'] = (df2.reindex(index=df1['Subtype'], columns=cols) .to_numpy()[np.arange(len(df1)), idx] )  Output:  Subtype Building Condition Intervention 0 A Good Repair 1 B Bad Reconstruct 2 C Bad Reconstruct  This is also the most efficient approach:
Question: I'm struggling with understanding the way torch.permute() works. In general, how exactly is an n-D tensor permuted? An example with explaination for a 4-D or higher dimension tensor is highly appreciated. I've search across the web but did not find any clearly explaination.

Answer: All tensors are contiguous 1D data lists in memory. What differs is the interface PyTorch provides us with to access them. This all revolves around the notion of stride, which is the way this data is navigated through. Indeed, on a higher level, we prefer to reason our data in higher dimensions by using tensor shapes. The following example and description are still valid for higher-dimensional tensors. The permutation operator offers a way to change how you access the tensor data by seemingly changing the order of dimensions. Permutations return a view and do not require a copy of the original tensor (as long as you do not make the data contiguous), in other words, the permuted tensor shares the same underlying data. At the user interface, permutation reorders the dimensions, which means the way this tensor is indexed changes depending on the order of dimensions supplied to the torch.Tensor.permute method. Take a simple 3D tensor example: x shaped (I=3,J=2,K=2). Given i<I, j<J, and k<K, x could naturally be accessed via x[i,j,k]. Concerning the underlying data being accessed, since the stride of x is (JK=4, J=2, 1), then x[i,j,k] corresponds to _x[i*JK+j*J+1] where _x is the underlying data of x. By "corresponds to", it means the data array associated with tensor x is being accessed with the index i*JK+j*J+1. If you now were to permute your dimensions, say y = x.permute(2,0,1), then the underlying data would remain the same (in fact data_ptr would yield the same pointer) but the interface to y is different! We have y with a shape of (K,I,J) and accessing y[i,j,k] translate to x[k,i,j] ie. dim=2 move to the front and dim=0,1 moved to the back... After permutation the stride is no longer the same, y has a stride of (IJ, I, 1) so y[i,j,k] corresponds to _x[i*IJ+j*I+1]. To read more about views and strides, refer to this.
Question: Given a raw email, how can I validate the DKIM signature with Python? Ideally I’d like more than just a pass / fail result, I’d like to know details of any issues. I’ve found the dkimpy package, but the API isn’t obvious to me.

Answer: For a simple pass/fail validation: import dkim # dkimpy # Returns True/False res = dkim.verify(mail_data.encode())  For something more nuanced you can do this: d = dkim.DKIM(mail_data.encode(), logger=None, minkey=1024, timeout=5, tlsrpt=False) try: d.verify() # If it fails, a `dkim.ValidationError` exception will be thrown with details except dkim.ValidationError as e: print(e) # dkim.ValidationError: body hash mismatch (got b'PXUrNdoTzGcLtd4doJs+CufsiNvxoM5Q3SUPGi00C+I=', expected b'ax9SInd7Z3AQjRzcZSnY6UK392QEvjnKrjhAnsqfDnM=')
Question: I have a very large arrow dataset (181GB, 30m rows) from the huggingface framework I've been using. I want to randomly sample with replacement 100 rows (20 times), but after looking around, I cannot find a clear way to do this. I've tried converting to a pd.Dataframe so that I can use df.sample(), but python crashes everytime (assuming due to large dataset). So, I'm looking for something built-in within pyarrow. df = Dataset.from_file("embeddings_job/combined_embeddings_small/data-00000-of-00001.arrow") df=df.to_table().to_pandas() #crashes at this line random_sample = df.sample(n=100)  Some ideas: not sure if this is w/replacement import numpy as np random_indices = np.random.randint(0, len(df), size=100) # Take the samples from the dataset sampled_table = df.select(random_indices)  Using huggingface shuffle  sample_size = 100 # Shuffle the dataset shuffled_dataset = df.shuffle() # Select the first 100 rows sampled_dataset = df.select(range(sample_size))  Is the only other way through terminal commands? Would this be correct: for i in {1..30}; do shuf -n 1000 -r file > sampled_$i.txt; done  After getting each chunk, the plan is to run each chunk through a random forest algoritm. What is the best way to go about this? Also, I would like to note that whatever solution should make sure the indices do not get reset when I get the output subset.

Answer: A bit late, but I just had to write a function to randomly sample a pyarrow Table. It produces the sample directly from a pyarrow Table without converting to a pandas dataframe. def sample_table(table: pa.Table, n_sample_rows: int = None) -> pa.Table: if n_sample_rows is None or n_sample_rows >= table.num_rows: return table indices = random.sample(range(table.num_rows), k=n_sample_rows) return table.take(indices)
Question: What is the cleanest way to commit data stored in instances of a dataclass contained in a list to SQLite with sqlite3's executemany? For example: @dataclass class Person: name: str age: int a = Person("Alice", 21) b = Person("Bob", 22) c = Person("Charlie", 23) people = [a, b, c] # ... cursor.executemany(f"INSERT INTO {table} ({headers}) ({placeholders})", people) # <-- fails, obviously  Is there a built-in mechanism that does the work for me (goes over each attribute of the dataclass), or an idiomatic way, or, if not, how do I implement an adapter for this without explicitly listing the attributes one by one? I could convert it to a list of tuples, that works out of the box, yet feels redundant.

Answer: Use the astuple() function to convert the dataclass instances to tuples. from dataclasses import dataclass, fields, astuple headers = ','.join(f.name for f in fields(Person)) people = map(astuple, [a, b, c]) placeholders = ','.join(['?'] * len(headers)) table = 'persons' cursor.executemany(f"INSERT INTO {table} ({headers}) ({placeholders})", people)
Question: I am getting this error "AxiosError: Request failed with status code 403" while uploading image on streamlit app. this is my code screenshot of error import streamlit as st from PIL import Image # Title st.title('Image Viewer') #Upload an image uploaded_image = st.file_uploader("Upload an image", type=["jpg", "jpeg", "png", "webp"]) # Check the type of the uploaded image if uploaded_image is not None: # Open the image using PIL image = Image.open(uploaded_image) # Display the image st.image(image, use_column_width=True)  I checked all the necessary packages is installed along with tried running same project in newly created environment but it didn't solve anything.

Answer: If you are running it locally try:  streamlit run app.py --server.enableXsrfProtection false
Question: I am trying to use the NebulaGraphStore class from llama_index via from llama_index.graph_stores.nebula import NebulaGraphStore as suggested by the llama_index documentation, but the following error occurred: ModuleNotFoundError Traceback (most recent call last) Cell In[2], line 1 ----> 1 from llama_index.graph_stores.nebula import NebulaGraphStore ModuleNotFoundError: No module named 'llama_index.graph_stores'  I tried updating llama_index (version 0.10.5) with pip install -U llama-index but it doesn't work. How can I resolve this?

Answer: According to latest doc of llama-index, all graph-store module are not included in llama-index core packages and needs to install it by pip: %pip install llama-index-llms-openai %pip install llama-index-embeddings-openai %pip install llama-index-graph-stores-nebula %pip install llama-index-llms-azure-openai  https://docs.llamaindex.ai/en/stable/examples/index_structs/knowledge_graph/NebulaGraphKGIndexDemo.html
Question: I'm creating a simple bot using Telebot. The start func offers two buttons to the user (Currency/Weather). I expect the bot to move to the func "where_to_go" when a button is pressed. It happens only when I press the button twice or send any other second message. Then "where_to_go" func reads the first message. So it works, but why do I have to send one more message to push it? How can I fix it? Here is the code: @bot.message_handler(commands = ['start','main']) def start(message): markup = types.ReplyKeyboardMarkup() btn1 = types.KeyboardButton('Currency Calc') markup.row(btn1) btn2 = types.KeyboardButton('Weather Today') markup.row(btn2) file = open('./Start Photo.png', 'rb') bot.send_photo(message.chat.id, file, reply_markup=markup) bot.send_message(message.chat.id, f"Good night, {message.from_user.first_name} {message.from_user.last_name} 💋. \n") bot.send_message(message.chat.id, f"I'm PogodiPogoda Bot. Choose a button or just talk to me.", reply_markup=markup) bot.register_next_step_handler(message, where_to_go) def where_to_go(message): if message.text == 'Currency Calc': bot.register_next_step_handler(message, converter) elif message.text == 'Weather Today': bot.register_next_step_handler(message, city)  The bot works well when he make the following step (from "where_to_go" to the funcs "converter" or "city").

Answer: You can try this : @bot.message_handler(commands = ['start','main']) def start(message): markup = types.ReplyKeyboardMarkup() btn1 = types.KeyboardButton('Currency Calc') markup.row(btn1) btn2 = types.KeyboardButton('Weather Today') markup.row(btn2) file = open('./Start Photo.png', 'rb') bot.send_photo(message.chat.id, file, reply_markup=markup) bot.send_message(message.chat.id, f"Good night, {message.from_user.first_name} {message.from_user.last_name} 💋. \n") bot.send_message(message.chat.id, f"I'm PogodiPogoda Bot. Choose a button or just talk to me.", reply_markup=markup) # Register next step handler directly within the start function bot.register_next_step_handler(message, where_to_go) def where_to_go(message): if message.text == 'Currency Calc': converter(message) # Call the function directly instead of registering another next step handler elif message.text == 'Weather Today': city(message) # Call the function directly instead of registering another next step handler  @bot will move to the where_to_go function immediately after the user selects an option, without the need for an additional message
Question: From what I've managed to figure out so far, this may only be a problem in macOS. Here's my MRE: from multiprocessing import Pool, Manager from functools import partial def foo(d, n): d.setdefault("X", []).append(n) def main(): with Manager() as manager: d = manager.dict() with Pool() as pool: pool.map(partial(foo, d), range(5)) print(d) if __name__ == "__main__": main()  Output: {'X': []}  Expected output: {'X': [0, 1, 2, 3, 4]}  Platform: macOS 14.3.1 Python 3.12.2  Maybe I'm doing something fundamentally wrong but I understood that the whole point of the Manager was to handle precisely this kind of scenario. EDIT There is another hack which IMHO should be unnecessary but even this doesn't work (produces identical output): from multiprocessing import Pool, Manager def ipp(d, lock): global gDict, gLock gDict = d gLock = lock def foo(n): global gDict, gLock with gLock: gDict.setdefault("X", []).append(n) def main(): with Manager() as manager: d = manager.dict() lock = manager.Lock() with Pool(initializer=ipp, initargs=(d, lock)) as pool: pool.map(foo, range(5)) print(d) if __name__ == "__main__": main() 

Answer: This is a crappy "answer" and I apologize for that and will delete it once you post a comment that you have seen it. I get the distinct impression that the answer is related to the "sub-list" not being managed per that post I mentioned. For example: This seems to work: def foo(d, n): d.setdefault("X", []).append(n) # the thing being updated is managed def main(): with Manager() as manager: d = manager.dict() d["X"] = manager.list() with Pool() as pool: pool.map(partial(foo, d), range(5)) print(list(d["X"])) if __name__ == "__main__": main()  as does: def foo(d, n): target = d.setdefault("X", []) target.append(n) d["X"] = target # the thing being updated is managed def main(): with Manager() as manager: d = manager.dict() with Pool() as pool: pool.map(partial(foo, d), range(5)) print(d) if __name__ == "__main__": main()  So updating a "managed" object seems to work while updating an unmanaged list even inside a managed container object does not.
Question: I have the following pandas dataframe import pandas as pd import random random.seed(42) pd.DataFrame({'index': list(range(0,10)), 'cluster': [random.choice(['S', 'C', ]) for l in range(0,10)]}) index cluster 0 0 S 1 1 S 2 2 C 3 3 S 4 4 S 5 5 S 6 6 S 7 7 S 8 8 C 9 9 S  I would like to create 5 new columns, one for each unique value of the cluster column, which will be the cumulative sum of appearances of each value. The output pandas dataframe should look like this : pd.DataFrame({'index': list(range(0,10)), 'cluster': [random.choice(['S', 'C', ]) for l in range(0,10)], 'cumulative_S': [1,2,2,3,4,5,6,7,7,8], 'cumulative_C': [0,0,1,1,1,1,1,1,2,2]}) index cluster cumulative_S cumulative_C 0 0 S 1 0 1 1 S 2 0 2 2 C 2 1 3 3 S 3 1 4 4 S 4 1 5 5 S 5 1 6 6 S 6 1 7 7 S 7 1 8 8 C 7 2 9 9 S 8 2  How can I achieve that ?

Answer: Code df is your input dataframe tmp = (pd.get_dummies(df['cluster']) .cumsum()[df['cluster'].unique()] .add_prefix('cumulative_') ) out = pd.concat([df, tmp],axis=1)  out  index cluster cumulative_S cumulative_C 0 0 S 1 0 1 1 S 2 0 2 2 C 2 1 3 3 S 3 1 4 4 S 4 1 5 5 S 5 1 6 6 S 6 1 7 7 S 7 1 8 8 C 7 2 9 9 S 8 2
Question: Why does re.compile.findall not find "um" if "um" is at the beginning of the string (it works fine is "um" isn't at the beginning of the string, as per the last 2 lines below) >>> s = "um" >>> re.findall(r"\bum\b", s, re.IGNORECASE) ['um'] >>> re.compile(r"\bum\b").findall(s, re.IGNORECASE) [] >>> re.compile(r"\bum\b").findall(s + " foobar", re.IGNORECASE) [] >>> re.compile(r"\bum\b").findall("foobar " + s, re.IGNORECASE) ['um']  I would have expected the two options to be identical. What am I missing?

Answer: You intended to pass re.IGNORECASE to the compile() function, but in the failing cases you're actually passing it to the findall() method. There it's interpreted as an integer giving the starting position for the search to begin. Its value as an integer isn't defined, but happens to be 2 today: >>> int(re.IGNORECASE) 2  Rewrite the code to work as intended, and it's fine; for example: >>> re.compile(r"\bum\b", re.IGNORECASE).findall(s + " foobar") # pass to compile() ['um']  As originally written, it can't work unless "um" starts at or after position 2: >>> re.compile(r"\bum\b").findall(" " + s, re.IGNORECASE) [] >>> re.compile(r"\bum\b").findall(" " + s, re.IGNORECASE) # starts at 2 ['um']
Question: I am writing this to have a reference post when it comes to plotting circles with a scatter plot but the user wants to provide a real radius for the scattered circles instead of an abstract size. I have been looking around and there are other posts that explain the theory of it, but have not come accross a ready-to-use function. I have tried myself and hope that I am close to finding the solution. This is what I have so far: import matplotlib.pyplot as plt import numpy as np def true_scatter(x, y, r, ax, **kwargs): # Should work for an equal aspect axis ax.set_aspect('equal') # Access the DPI and figure size dpi = ax.figure.dpi fig_width_inch, _ = ax.figure.get_size_inches() # Calculate plot size in data units xlim = ax.get_xlim() plot_width_data_units = xlim[1] - xlim[0] # Calculate the scale factor: pixels per data unit plot_width_pixels = fig_width_inch * dpi scale = plot_width_pixels / plot_width_data_units # Convert radius to pixels, then area to points squared radius_pixels = r * scale area_pixels_squared = np.pi * (radius_pixels ** 2) area_points_squared = area_pixels_squared * (72 / dpi) ** 2 # Scatter plot with converted area scatter = ax.scatter(x, y, s=area_points_squared, **kwargs) return scatter # Example with single scatter fig, ax = plt.subplots() ax.set_xlim(-2, 2) ax.set_ylim(-2, 2) scatter = true_scatter(0, 0, 1, ax) ax.set_xlim(-2, 2) ax.set_ylim(-2, 2) plt.grid() plt.show()  Wrong single scatter plot Unfortunately, it is not quite the answer. I get a circle of radius ~1.55 instead of 1. Would anyone be able to spot what is wrong with my approach? Thank you!

Answer: It is not very clear whether you want r to be the radius or the diameter of the circle. The code below supposes it is the radius (just leave out the * 2 if you want the diameter. For a discussion about how the dot size is measured, see this post. The code below converts the radius in data units to pixels, and then to "points". Three different sizes are tested. import matplotlib.pyplot as plt def true_scatter(x, y, r, ax, **kwargs): # the radius is given in data coordinates in the x direction # Should work for an equal aspect axis ax.set_aspect('equal') # measure the data coordinates in pixels radius_in_pixels, _ = ax.transData.transform((r, 0)) - ax.transData.transform((0, 0)) # one "point" is 1/72 of an inch radius_in_points = radius_in_pixels * 72.0 / ax.figure.dpi # the scatter plot size is set in square point units area_points_squared = (radius_in_points * 2) ** 2 # Scatter plot with converted area scatter = ax.scatter(x, y, s=area_points_squared, **kwargs) return scatter # Example with three scatter dots with different radii fig, ax = plt.subplots(figsize=(5, 5)) ax.set_xlim(-2, 2) ax.set_ylim(-2, 2) scatter = true_scatter(0, 0, 1.5, ax) scatter = true_scatter(0, 0, 1, ax) scatter = true_scatter(0, 0, 0.5, ax) ax.grid() plt.show()
Question: What is the windows equivalent of pwd.getpwuid? I believe pwd is an UNIX package. Here is the code and the error when using windows import pwd file_owner_name = pwd.getpwuid(file_owner_uid).pw_name import pwd ModuleNotFoundError: No module named 'pwd' 

Answer: Unfortunately not supported by Python out of the box. Methods like pathlib.Path.owner() always return NotImplementedError on Windows. You'll need to call Windows functions directly.  The easiest way is to install PyWin32. Then: import win32security path = './' security_descriptor = win32security.GetFileSecurity(path, win32security.OWNER_SECURITY_INFORMATION) owner_sid = security_descriptor.GetSecurityDescriptorOwner() name, domain, type = win32security.LookupAccountSid(None, owner_sid) print(f'{domain}\\{name}') # BOPPREH-DESKTOP\boppreh   Alternatively, you can use ctypes to avoid an extra dependency, but the code is a lot more complicated, and most mistakes will result in segfaults or unreliable behavior instead of nice exceptions.
Question: I'm working with biologists, who are imaging DNA strands under a microscope, giving greyscale PNGs. Depending on the experiment, there can be a lot of background noise. On the ones where there is quite little, a simple threshold for pixel intensity is generally enough to remove it. However for some, it's not so simple.  (not sure why it's uploading as jpg - this should let you download it as PNG) When I do apply a threshold for pixel intensity, I get this result:  If I raise the threshold any more, I'll start to lose the pixels at the edges of the image that aren't very bright. The noise seems to follow a gaussian distribution based on its location within the image, probably due to the light source of the microscope. What's the best way to compare pixels to the local background noise? Can I integrate the noise distribution? For the moment I've been using imageio.v3 in python3 for basic pixel manipulation, but I'm open to suggestions. EDIT: I tried to use histogram equalisation, but it doesn't quite seem to be what I'm looking for... 

Answer: I am not sure what result you want so I am working somewhat in the dark. I suspect you want a "Local Area Threshold" - see Wikipedia. Also referred to as "Adaptive Threshold". It is available, or can be achieved in many image processing packages, but for brevity and until I (or others) better understand what you want as a result, I will just demonstrate with ImageMagick which is free and can be installed on macOS, Linux and Windows. The command is -lat which means "Local Area Threshold". So, the syntax to highlight in white any pixel that is more than 2% brighter than the surrounding local area of 80x80 is as follows: magick image.png -alpha off -lat 80x80+2% result.png   If you want any pixel more than 2% brighter than its surrounding 10x10 neighbourhood, use: magick image.png -alpha off -lat 10x10+2% result.png    Note that I used -alpha off because your image has a superfluous alpha channel I wanted to discard.  Using OpenCV you'll get something vaguely similar with: #!/usr/bin/env python3 import cv2 as cv # Load image as greyscale im = cv.imread('image.png', cv.IMREAD_GRAYSCALE) # Local area is 49x49, and pixel must be 15 above threshold to show in output thMean = cv.adaptiveThreshold(im,255,cv.ADAPTIVE_THRESH_MEAN_C, cv.THRESH_BINARY,49,-15) cv.imwrite('result-mean.png', thMean) # Local area is 29x29, and pixel must be 10 above threshold to show in output thGauss = cv.adaptiveThreshold(im,255,cv.ADAPTIVE_THRESH_GAUSSIAN_C, cv.THRESH_BINARY,29,-10) cv.imwrite('result-Gauss.png', thGauss)