eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I need to write a custom exec function in python (for several purposes but this is not the problem here, so this custom exec which is called myExec will do exactly as exec for now). I went into this problem : def myExec(code): exec(code) code = """ a = 1 print(a) u = [a for x in range(3)] print(u) """ myExec(code)  Running this program gives 1 Traceback (most recent call last): File "___.py", line 12, in <module> myExec(code) File "___.py", line 2, in myExec exec(code, globals(), locals()) File "<string>", line 4, in <module> File "<string>", line 4, in <listcomp> NameError: name 'a' is not defined  So print(a) went without any problems. But the error occurs with the line u = [a for x in range(3)]. When the generator object is converted into a list, the name a seems undefined. Note that if the line were u = [a, a, a], then, no error is raised. Nor if we use exec instead of myExec. Any reason why and how to solve this ?

Answer: We can explain this behavior if we take a look at the decompiled code. from dis import dis def myExec(code): dis(code)   a = 1 is compiled to STORE_NAME, so it stores a as a local variable here print(a) uses LOAD_NAME to load the local a. It is a local variable, so LOAD_NAME finds it. the list comprehension is compiled into a function that uses LOAD_GLOBAL for a  That's where the error is coming from. a was created as a local variable and is accessed as a global one in the list comprehension. This results in a name error. This also explains why the exec works in the global scope (either by calling exec in the global scope or by passing globals()). Because then STORE_NAME stores a in the current scope (which is global) and LOAD_GLOBAL can find a. If you switch to Python3.12 which implements PEP 709 – Inlined comprehensions you will see that no extra function is created for the list comprehension and a is looked up with LOAD_NAME and can be found. So to fix your issue: either upgrade to Python3.12 or pass the globals()
Question:  I have a table of company data that links subsidiary to parent companies as shown in the left hand side table of the screenshot. I need to transform the data into the table on the right hand side of the screenshot. This requires tracing through the two columns of the table and making the link between individual rows. So far the only thing I have tried is to join the table against itself recursively.. but I am thinking that some kind of tree structure here would make more sense? I.e. creating a branch of all connected companies with the 'ultimate parent company' as the trunk? These concepts are new to me so appreciate any input Reproducible input: df = pd.DataFrame({'Subsidiary Company': ['Company B', 'Company C', 'Company D', 'Company 2', 'Company 3'], 'Parent Company': ['Company A', 'Company B', 'Company C', 'Company 1', 'Company 2']}) 

Answer: You can use networkx to form a directed graph, then loop over the paths with all_simple_paths: import numpy as np import networkx as nx # create the directed graph G = nx.from_pandas_edgelist(df, source='Subsidiary Company', target='Parent Company', create_using=nx.DiGraph) # find roots (final level companies) roots = {v for v, d in G.in_degree() if d == 0} # {'Company 3', 'Company D'} # find leaves (ultimate parents) leaves = {v for v, d in G.out_degree() if d == 0} # {'Company 1', 'Company A'} # function to rename the columns # 0 is the subsidiary company # inf is the ultimate parent # other numbers are intermediates def renamer(x): if x == 0: return 'Subsidiary Company' if np.isfinite(x): return f'Intermediate Parent Company {int(x)}' return 'Ultimate Parent Company' # find the connected nodes # the roots/leaves for each group # iterate over the paths between each root/node combination # create the sliding paths (with range) # convert to DataFrame out = ( pd.DataFrame( [ dict(enumerate(p[i-1:-1])) | {float('inf'): p[-1]} for c in nx.weakly_connected_components(G) for r in c & roots for l in c & leaves for p in nx.all_simple_paths(G, r, l) for i in range(len(p)-1, 0, -1) ] ) .sort_index(axis=1) .rename(columns=renamer) )  Output:  Subsidiary Company Intermediate Parent Company 1 Intermediate Parent Company 2 Ultimate Parent Company 0 Company B NaN NaN Company A 1 Company C Company B NaN Company A 2 Company D Company C Company B Company A 3 Company 2 NaN NaN Company 1 4 Company 3 Company 2 NaN Company 1
Question: This sample multimodal/main.py appears to show Ollama I am trying to do the same with an image loaded from my machine. I am using the gemma2:27b model. The model is working with chat so that is not the issue. my Code import os.path import PIL.Image from dotenv import load_dotenv from ollama import generate load_dotenv() CHAT_MODEL_NAME = os.getenv("MODEL_NAME_LATEST") image_path = os.path.join("data", "image_one.jpg") test_image = PIL.Image.open(image_path) # test 1: for response in generate(CHAT_MODEL_NAME, 'What do you see', images=[test_image], stream=True): print(response['response'], end='', flush=True) # response: ollama._types.RequestError: image must be bytes, path-like object, or file-like object # test 2: bytes for response in generate(CHAT_MODEL_NAME, 'What do you see', images=[test_image.tobytes()], stream=True): print(response['response'], end='', flush=True) # response: Please provide me with the image! # test 3: Path for response in generate(CHAT_MODEL_NAME, 'What do you see', images=[image_path], stream=True): print(response['response'], end='', flush=True) # response: Please provide me with the image!  How do i properly load an image to Gemma Cross posted on issue forum 289

Answer: The Gemma2 models are not multimodal. They accept only text as input. If you want to process images, you need to use PaliGemma which is not supported by Ollama yet (you can follow this issue about it). You may find some PaliGemma examples at the Gemma cookbook github repo.
Question: I am trying to compare two dataframes via dfcompare = (df0 == df1) and nulls are never considered identical (unlike join there is no option to allow nulls to match). My approach with other fields is to fill them in with an "empty value" appropriate to their datatype. What should I use for structs? import polars as pl df = pl.DataFrame( { "int": [1, 2, None], "data" : [dict(a=1,b="b"),dict(a=11,b="bb"),None] } ) df.describe() print(df) df2 = df.with_columns(pl.col("int").fill_null(0)) df2.describe() print(df2) # these error out:... try: df3 = df2.with_columns(pl.col("data").fill_null(dict(a=0,b=""))) except (Exception,) as e: print("try#1", e) try: df3 = df2.with_columns(pl.col("data").fill_null(pl.struct(dict(a=0,b="")))) except (Exception,) as e: print("try#2", e)  Output:  shape: (3, 2) ┌──────┬─────────────┐ │ int ┆ data │ │ --- ┆ --- │ │ i64 ┆ struct[2] │ ╞══════╪═════════════╡ │ 1 ┆ {1,"b"} │ │ 2 ┆ {11,"bb"} │ │ null ┆ {null,null} │ └──────┴─────────────┘ shape: (3, 2) ┌─────┬─────────────┐ │ int ┆ data │ │ --- ┆ --- │ │ i64 ┆ struct[2] │ ╞═════╪═════════════╡ │ 1 ┆ {1,"b"} │ │ 2 ┆ {11,"bb"} │ │ 0 ┆ {null,null} │ └─────┴─────────────┘ try#1 invalid literal value: "{'a': 0, 'b': ''}" try#2 a Error originated just after this operation: DF ["int", "data"]; PROJECT */2 COLUMNS; SELECTION: "None"  My, satisfactory, workaround has been to unnest the columns instead. This works fine (even better as it allow subfield-by-subfield fills). Still, I remain curious about how to achieve a suitable "struct literal" that can be passed into these types of functions. One can also imagine wanting to add a hardcoded column as in df4 = df.with_columns(pl.lit("0").alias("zerocol"))

Answer: A struct literal to use in the context of pl.Expr.fill_null can be created with pl.struct as follows. df.with_columns( pl.col("data").fill_null( pl.struct(a=pl.lit(1), b=pl.lit("MISSING")) ) )  shape: (3, 2) ┌──────┬───────────────┐ │ int ┆ data │ │ --- ┆ --- │ │ i64 ┆ struct[2] │ ╞══════╪═══════════════╡ │ 1 ┆ {1,"b"} │ │ 2 ┆ {11,"bb"} │ │ null ┆ {1,"MISSING"} │ └──────┴───────────────┘
Question: Have a dataframe in following format data = {'regions':["USA", "USA", "USA", "FRANCE", "FRANCE","FRANCE"], 'dates':['2024-08-03', '2024-08-10', '2024-08-17','2024-08-03', '2024-08-10', '2024-08-17'], 'values': [3, 4, 5, 7, 8,0], } df = pd.DataFrame(data) regions dates values 0 USA 2024-08-03 3 1 USA 2024-08-10 4 2 USA 2024-08-17 5 3 FRANCE 2024-08-03 7 4 FRANCE 2024-08-10 8 5 FRANCE 2024-08-17 0  Need to change this df from long to wide format. Use the most recent dates as current date, and the other two dates will be lagged dates. Expected output is like regions dates values_lag2 values_lag1 values USA 2024-08-17 3 4 5 FRANCE 2024-08-17 7 8 0  Currently I used a for loop to manually to change the format. Just wondering if there is a more elegant way to realize it. Thanks

Answer: If same dates per each regions is possible convert column to datetimes, pivoting, change columns names and add column with maximal dates: df['dates'] = pd.to_datetime(df['dates']) out = df.pivot(index='regions', columns='dates', values='values') out.columns = [f'values_lag{i-1}' if i!=1 else 'values' for i in range(len(out.columns), 0, -1)] out = df.groupby('regions')['dates'].max().to_frame().join(out).reset_index() print (out) regions dates values_lag2 values_lag1 values 0 FRANCE 2024-08-17 7 8 0 1 USA 2024-08-17 3 4 5  Another idea if possible different datetimes and need only ordering add sorting, counter by groupby.cumcount and pivoting with helper column g: df['dates'] = pd.to_datetime(df['dates']) df = df.sort_values(['regions', 'dates']) df['g'] = df.groupby('regions').cumcount(ascending=False) out = (df.pivot(index='regions', columns='g', values='values') .sort_index(ascending=False, axis=1)) out.columns=[f'values_lag{i}' if i!=0 else 'values' for i in out.columns] out = df.groupby('regions')['dates'].max().to_frame().join(out).reset_index() print (out) regions dates values_lag2 values_lag1 values 0 FRANCE 2024-08-17 7 8 0 1 USA 2024-08-17 3 4 5
Question: I have begun to learn and implement Polars because of (1) the potential speed improvements and (2) for the promise of being able to process larger-than-memory datasets. However, I'm struggling to see how the second promise is actually delivered in specific scenarios that my use case requires. One specific example I'm struggling with is how to read a multi-GB JSONL file from S3, apply a few transformations, and send the modified records to STDOUT. Gaps in the lazy "sink" methods... As I just raised in GitHub, the sink_*() methods do not support writing to a buffer or file-like - only to a named file path. Otherwise, it seems the simple solution would be something like sink_ndjson(sys.stdout, ...) No clear way to "batch" a DataFrame or LazyFrame into smaller data frames. The next thing I tried was to get smaller batches or dataframes (for instance 100K rows at a time) which I could process in memory and write with write_ndjson(sys.stdout, ...) one at a time until I reach the end of the stream. The closest I could find is LazyFrame.slice(offset, batch_size).collect() - except in practice, this seems to hang/crash on the first invocation rather than reading just the first n records and then proceeding. Even when I set a low number of records in the LazyFrame's schema scan limit. Perhaps this is a bug - but even still, the slice() method does not seem specifically designed for getting incremental batches from the lazy frame. Any help is much appreciated!

Answer: As was mentioned in comments, the streaming engine is undergoing a significant revamp to address the shortcomings of the current implementation. The details of that revamp, as far as I'm aware, haven't been documented so I can't say that this exact use case will be addressed in that revamp. It's not clear, to me, what the benefit is of saving data to stdout (in memory) if your overall data is too big for memory so that enhancement request may be too niche for it to be picked up in general. It seems a more in-mission ask would be for a read_ndjson_batched function similar to the read_csv_batched function. In the interim, you can implement this in python using s3's fsspec handler to read the file in lines and form your own batches.  import polars as pl import fsspec # or the s3 fsspec implementation lib from io import BytesIO s3fs = fsspec.filesystem() # or aforementioned s3 class batch_size = 100_000 with s3fs.open(path, 'rb') as ff: while True: batch = [] dbl_break=False for _ in range(batch_size): line = ff.readline() if not line: dbl_break=True break batch.append(line) if len(batch)==0: break df_batch = pl.read_ndjson(b"\n".join(batch)) do_batch_process(df_batch) if dbl_break: break
Question: Starting with a "pd.DataFrame" df :~  n v 0 1 0.0 1 2 0.0 2 3 0.0 3 4 0.0 4 5 0.0 5 6 0.0  I'd like to add up value in column "v", whereby a cell in column "v" is produced by multiplying previous cell of "v" with a fixed factor, then add current cell value of column "n". (See sample calculation table below)  ## sample calculation table :~ n v[i] n + v[i-1] * fixed factor 1 1.0 = 1 + 0.0 * 0.5 2 2.5 = 2 + 1.0 * 0.5 3 4.25 = 3 + 2.5 * 0.5 4 6.125 = 4 + 4.25 * 0.5 5 8.0625 = 5 + 6.125 * 0.5 6 10.03125 = 6 + 8.063 * 0.5  Managed to do it with row-by-row iteration (see for-loop below). However I think vectorised methods (like cumsum and shift) may be more efficient, but could not figure out how; because cumsum is complicated by multiplication, starting with an empty column v, and need to reference to previous cell of a column. Wonder how to do this with vectorised methods ? To reproduce my code :  df = pd.DataFrame({'n':[1,2,3,4,5,6]}) df['v'] = 0.0 def fnv(nu, vu): return nu + vu * 0.5 for i in range(0, df.shape[0]): df.v.at[i] = fnv(df.n.at[i], df.v.at[i-1] if i>0 else 0.0)   df (RESULTS) :~ n v 0 1 1.00 1 2 2.50 2 3 4.25 3 4 6.12 4 5 8.06 5 6 10.03 

Answer: Since n is variable, you can't easily vectorize this (you could using a matrix operation, see below, but this would take O(n^2) space). A good tradeoff might be to use numba to speed the operation: from numba import jit @jit(nopython=True) def fnv(n, factor=0.5): out = [] prev = 0 for x in n: out.append(x + prev*factor) prev = out[-1] return out df['v'] = fnv(df['n'].to_numpy())  Output:  n v 0 1 1.00000 1 2 2.50000 2 3 4.25000 3 4 6.12500 4 5 8.06250 5 6 10.03125  vectorized approach You could vectorize using a square matrix: x = np.arange(len(df)) tmp = x[:, None]-x df['v'] = np.nansum(np.where(tmp>=0, 0.5**tmp, np.nan) * df['n'].to_numpy(), axis=1)  Intermediates: # tmp [[ 0 -1 -2 -3 -4 -5] [ 1 0 -1 -2 -3 -4] [ 2 1 0 -1 -2 -3] [ 3 2 1 0 -1 -2] [ 4 3 2 1 0 -1] [ 5 4 3 2 1 0]] # tmp >= 0 [[ True False False False False False] [ True True False False False False] [ True True True False False False] [ True True True True False False] [ True True True True True False] [ True True True True True True]] # np.where(tmp>=0, 0.5**tmp, np.nan) * df['n'].to_numpy() [[1. nan nan nan nan nan] [0.5 2. nan nan nan nan] [0.25 1. 3. nan nan nan] [0.125 0.5 1.5 4. nan nan] [0.0625 0.25 0.75 2. 5. nan] [0.03125 0.125 0.375 1. 2.5 6. ]]
Question: I have two dataframe, one is daily and one is quarterly idx = pd.date_range("2023-03-31", periods=100, freq="D") idx_q = idx.to_series().resample("QE").last() df1 = pd.DataFrame({"A": [1, "a", None], "B": [4, None, 6]}, index=idx_q) np.random.seed(42) df2 = pd.DataFrame({"C": np.random.randn(100), "D": np.random.randn(100)}, index=idx) # resample df2 to workdays df2 = df2.resample("B").asfreq() # mask values larger than 0.9 in df2 with NaN df2 = df2.mask(df2 > 0.9) df = df2.join(df1)  I want to join them and ffill quarter data to daily. The problem is my data have None from source and should be kept in result. What's the right way to make this ffill?

Answer: IIUC, you can build a mask from df1 with isna and reindex: df = df1.join(df2) df = df.ffill().mask(df1.isna().reindex(columns=df.columns, fill_value=False))  Another approach could be to use a placeholder instead of NaNs in df1: df = df1.fillna('PLACEHOLDER').join(df2).ffill().replace('PLACEHOLDER', np.nan)  Output:  A B C D 2023-03-31 1 4.0 0.496714 -1.415371 2023-06-30 a NaN 0.496714 0.856399 2023-07-08 NaN 6.0 -0.234587 -1.142970
Question: I have the following states and transition matrix import numpy as np n_states = 3 states = np.arange(n_states) T = np.array([ [0.5, 0.5, 0], [0.5, 0, 0.5], [0, 0, 1] ])  I would like to simulate n_sims paths where each path consist of n_steps. Each path starts at 0. Therefore, I write n_sims = 100 n_steps = 10 paths = np.zeros((n_sims, n_steps), dtype=int)  With the help of np.random.Generator.choice I would like to "fill to the right" the paths using the transition matrix. My attempt look as follows rng = np.random.default_rng(seed=123) for s in range(1, n_steps+1): paths[:,s] = rng.choice( a=n_states, size=n_sim, p=T[paths[:,s-1]] )  This result in the following error:  ValueError: p must be 1-dimensional  How can I overcome this? If possible, I would like to prevent for-loops and vectorize the code.

Answer: IIUC, your process being inherently iterative, you won't benefit much from numpy's vectorization. You might want to consider using pure python: def simulation(T, n_states=3, n_sims=100, n_steps=10, seed=123): rng = np.random.default_rng(seed) start = np.zeros(n_steps, dtype=np.int64) out = [start] for i in range(n_sims-1): a = np.array([rng.choice(n_states, p=T[x]) for x in start]) out.append(a) start = a return np.array(out) simulation(T, n_states=3, n_sims=100, n_steps=10)  Example output: array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 0, 0, 0, 0, 1, 1, 0, 1, 1], [2, 0, 1, 0, 1, 2, 2, 0, 2, 2], [2, 0, 0, 1, 0, 2, 2, 1, 2, 2], [2, 1, 0, 2, 1, 2, 2, 2, 2, 2], [2, 2, 0, 2, 0, 2, 2, 2, 2, 2], [2, 2, 0, 2, 1, 2, 2, 2, 2, 2], [2, 2, 1, 2, 0, 2, 2, 2, 2, 2], [2, 2, 0, 2, 0, 2, 2, 2, 2, 2], [2, 2, 0, 2, 0, 2, 2, 2, 2, 2], [2, 2, 1, 2, 0, 2, 2, 2, 2, 2], [2, 2, 2, 2, 0, 2, 2, 2, 2, 2], [2, 2, 2, 2, 0, 2, 2, 2, 2, 2], [2, 2, 2, 2, 0, 2, 2, 2, 2, 2], [2, 2, 2, 2, 1, 2, 2, 2, 2, 2], [2, 2, 2, 2, 0, 2, 2, 2, 2, 2], [2, 2, 2, 2, 1, 2, 2, 2, 2, 2], [2, 2, 2, 2, 0, 2, 2, 2, 2, 2], ... [2, 2, 2, 2, 2, 2, 2, 2, 2, 2], [2, 2, 2, 2, 2, 2, 2, 2, 2, 2]])
Question: the problem is as the title states. I have a column AX filled with values. The name of the column is "Remarks" and it will contain remarks but some of those remarks are dates and some are full blown notes like "Person A owes Person B X amount." The problem I'm currently facing now is that in xlwings the columns that are just dates like "1/8/24" are converted to the date data type. I do not want this conversion to happen. I want it to remain as "1/8/24" literally and remain as the data type of "Text". The full workflow is as follows:  Read data from excel (I have no write access) Create a new excel workbook Put processed data into new excel workbook  So I tried to fix it in two places  After I read the data I converted the AX columns' values all to string with str(cell.value) among other options, none of which worked. Before the new excel workbook is saved.  Nothing in option 1 worked and I figured that it had something to with how Excel is handling dates. So, I'm now trying to prevent the conversion and just have "1/8/24" appear literally but nothing is working. I checked the documentation and I tried Range.options to prevent the conversion but it doesn't help much. As when I inspected the cell with "1/8/24" it showed up as a datetime.datetime object. Converting that with str just turns it back into a date anyways. So, I figured that I have to find a way to do the converting after it was written into the workbook. I messed around with data types in Excel and I found out that if I used this  Clicked next on everything  Then selected "Text" in the final screen the dates appeared. So, that leads me to try a new option which is to convert the data type of the entire column to just "Text". So I tried out stuff like this sheet.range("AX1").expand("down").api.NumberFormat = "@". But the workbook that was generated still doesn't show "1/8/24" literally. Instead it shows some number like 45299. Surprisingly when I converted that cell into "Long Date" it gets turned into a date "1st August 2024". This is where I stopped working as I ran out of ideas and have no idea how to continue. Any guidance is very much appreciated, thank you.

Answer: Dates, or dates looking strings, are always a pain. I think the simplest way to deal with it in your case is to insert a ' as the first character in every cell to force Excel to treat it like a string, something like .range(f'A{i}').value = f"'{value}" (not tested, assuming that the column of data in question is in column A).
Question: I want to create pandas.Period for a custom time period, for example for a duration starting_time = pd.Timestamp('2024-01-01 09:15:00') and ending_time = pd.Timestamp('2024-01-05 08:17:00'). One way to achieving this is by first getting the pandas.Timedelta and then create pandas.Period. import pandas as pd # Define start and end times starting_time = pd.Timestamp('2024-01-01 09:15:00') ending_time = pd.Timestamp('2024-01-05 08:17:00') # Calculate the duration (period) between the two timestamps period_duration = ending_time - starting_time period_duration_in_minutes = (period_duration.total_seconds()) //60 freq_str = f"{period_duration_in_minutes}min" period = pd.Period(starting_time, freq = freq_str) print(period.start_time) print(period.end_time)  But I need a straightforward approach, something like this (I know this won’t work)- period = pd.Period(start_time = starting_time, end_time=ending_time) 

Answer: You don't need to compute the duration in minutes, just pass the subtraction: pd.Period(starting_time, freq=ending_time-starting_time)  Which is almost your ideal straightforward approach. Output: Period('2024-01-01 09:15', '5702min')  Note that you could also use a function to have the desired parameters: def cust_period(start, end): return pd.Period(value=start, freq=end-start) cust_period(starting_time, ending_time) # Period('2024-01-01 09:15', '5702min')
Question: I'm using OOP for the first time, I want to make the SubWindow class inherit all properties from the MainWindow class, but self.root would be tk.Toplevel() instead of tk.Tk(): import tkinter as tk class MainWindow: def __init__(self, size, title): self.root = tk.Tk() self.root.geometry(size) self.root.title(title) def packAll(self): for widget in self.root.children: self.root.children[widget].pack() class SubWindow(MainWindow): def __init__(self, size, title): super().__init__(size, title)  If I put self.root = tk.Toplevel() after super().__init__(size, title), it still creates another instance of tkinter. Private variables haven't worked either. I haven't found a solution online. How do I solve this?

Answer: What you can do is to pass a required value for self.root to the base class constructor, with a default where the base class chooses tk.Tk(): import tkinter as tk class MainWindow: def __init__(self, size, title, root=None): if root == None: root = tk.Tk() self.root = root self.root.geometry(size) self.root.title(title) def packAll(self): for widget in self.root.children: self.root.children[widget].pack() class SubWindow(MainWindow): def __init__(self, size, title): super().__init__(size, title, tk.Toplevel())  This way only one root is instantiated, occurring before geometry() and title() are called. But, really, you shouldn't have to have if statements when you can just pass in the right values: import tkinter as tk class Window: def __init__(self, root, size, title): self.root = root self.root.geometry(size) self.root.title(title) def packAll(self): for widget in self.root.children: self.root.children[widget].pack() class MainWindow(Window): def __init__(self, size, title): super().__init__(tk.Tk(), size, title) class SubWindow(Window): def __init__(self, size, title): super().__init__(tk.Toplevel(), size, title)