eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'm working with a DataFrame in Python using pandas, and I'm trying to apply multiple conditions to filter rows based on temperature values from multiple columns. However, after applying my conditions and using dropna(), I end up with zero rows even though I expect some data to meet these conditions. The goal is compare with Ambient temp+40 C and if the value is more than this, replace it with NaN. Otherwise, keep the original value. Here's a sample of my DataFrame and the conditions I'm applying: data = { 'Datetime': ['2022-08-04 15:06:00', '2022-08-04 15:07:00', '2022-08-04 15:08:00', '2022-08-04 15:09:00', '2022-08-04 15:10:00'], 'Temp1': [53.4, 54.3, 53.7, 54.3, 55.4], 'Temp2': [57.8, 57.0, 87.0, 57.2, 57.5], 'Temp3': [59.0, 58.8, 58.7, 59.1, 59.7], 'Temp4': [46.7, 47.1, 80, 46.9, 47.3], 'Temp5': [52.8, 53.1, 53.0, 53.1, 53.4], 'Temp6': [50.1, 69, 50.3, 50.3, 50.6], 'AmbientTemp': [29.0, 28.8, 28.6, 28.7, 28.9] } df1 = pd.DataFrame(data) df1['Datetime'] = pd.to_datetime(df1['Datetime']) df1.set_index('Datetime', inplace=True)  Code: temp_cols = ['Temp1', 'Temp2', 'Temp3', 'Temp4', 'Temp5', 'Temp6'] ambient_col = 'AmbientTemp' condition = (df1[temp_cols].lt(df1[ambient_col] + 40, axis=0)) filtered_df = df1[condition].dropna() print(filtered_df.shape)  Response: (0, 99)  Problem: Despite expecting valid data that meets the conditions, the resulting DataFrame is empty after applying the filter and dropping NaN values. What could be causing this issue, and how can I correct it?

Answer: Use DataFrame.where: condition = (df1[temp_cols].lt(df1[ambient_col] + 40, axis=0)) df1[temp_cols] = df1[temp_cols].where(condition)  If need new DataFrame add DataFrame.reindex: temp_cols = ['Temp1', 'Temp2', 'Temp3', 'Temp4', 'Temp5', 'Temp6'] ambient_col = 'AmbientTemp' condition = (df1[temp_cols].lt(df1[ambient_col] + 40, axis=0)) filtered_df = df1.where(condition.reindex(df1.columns, fill_value=True, axis=1)) print(filtered_df) Temp1 Temp2 Temp3 Temp4 Temp5 Temp6 AmbientTemp Datetime 2022-08-04 15:06:00 53.4 57.8 59.0 46.7 52.8 50.1 29.0 2022-08-04 15:07:00 54.3 57.0 58.8 47.1 53.1 NaN 28.8 2022-08-04 15:08:00 53.7 NaN 58.7 NaN 53.0 50.3 28.6 2022-08-04 15:09:00 54.3 57.2 59.1 46.9 53.1 50.3 28.7 2022-08-04 15:10:00 55.4 57.5 59.7 47.3 53.4 50.6 28.9  How it working: #mask print(condition) Temp1 Temp2 Temp3 Temp4 Temp5 Temp6 Datetime 2022-08-04 15:06:00 True True True True True True 2022-08-04 15:07:00 True True True True True False 2022-08-04 15:08:00 True False True False True True 2022-08-04 15:09:00 True True True True True True 2022-08-04 15:10:00 True True True True True True #added Trues values in mask for missing columns - here `AmbientTemp` print (condition.reindex(df1.columns, fill_value=True, axis=1)) Temp1 Temp2 Temp3 Temp4 Temp5 Temp6 AmbientTemp Datetime 2022-08-04 15:06:00 True True True True True True True 2022-08-04 15:07:00 True True True True True False True 2022-08-04 15:08:00 True False True False True True True 2022-08-04 15:09:00 True True True True True True True 2022-08-04 15:10:00 True True True True True True True
Question: given A = np.array([[1,2],[3,4],[5,6],[7,8]]) B = np.array([[9,10,11],[12,13,14]])  matrix multiplication would be if I did C = np.einsum('ij,jk->ik', A, B)  but if I don't multiply j in the input instead using k ... E = np.einsum('ij,kl->il', A, B)  what am I effectively summing across? Is there a way to think about this intuitively? I'm confused because the dimensions end up the same as if it were matrix multiplication. I've tried playing around with different numbers in the matrices A and B to get a feel for it but I'm wondering if someone can break it down for me so I can understand what is going on in this example.

Answer: This sums A over axis=1, B over axis=0 and computes the outer product: np.einsum('ij,kl->il', A, B) np.outer(A.sum(1), B.sum(0)) array([[ 63, 69, 75], [147, 161, 175], [231, 253, 275], [315, 345, 375]])  You could check the individual einsum results: # sum over 1 np.einsum('ij->i', A) # array([ 3, 7, 11, 15]) # sum over 0 np.einsum('kl->l', B) # array([21, 23, 25]) # outer product np.einsum('i,l->il', A.sum(1), B.sum(0)) # array([[ 63, 69, 75], # [147, 161, 175], # [231, 253, 275], # [315, 345, 375]])
Question: I'm trying to match a sentence fragment of the form "after 3 days" or "after 1 month". I want to be particular with the single and plural forms, so "1 day" is valid but "1 days" is not. I have the following code which is nearly there but the first two entries in the failure tests don't fail. Any suggestions please that use syntactical notation as I'd like, if possible, to avoid a set_parse_action() that checks the numeric value against the unit's plurality. from pyparsing import * units = Keyword('days') ^ Keyword('months') unit = Keyword('day') ^ Keyword('month') single = Literal('1') + unit multi = Word(nums) + units after = Keyword('after') + ( single ^ multi ) a = after.run_tests(''' after 1 day after 2 days after 1 month after 2 months ''') print('=============') b = after.run_tests(''' after 1 days after 2 day after 1day after 2days ''', failure_tests = True) print('Success tests', 'passed' if a[0] else 'failed') print('Failure tests', 'passed' if b[0] else 'failed') 

Answer: Only the case after 1 days passes when it should fail, the other three cases fail as expected. The issue is that the check multi = Word(nums) + units uses nums which includes 1, so even if your singular variant does not work, this one will. I looked up how nums is defined, apparently it is nums = '0123456789' (see here). Consequently you remove the 1. This works for me: ... multi_nums = '023456789' # nums excluding 1 single = Literal('1') + unit multi = Word(multi_nums) + units ...  EDIT: The above fails for double digits including 1, see comments. Fixed version as per comments: single = Literal('1') + unit multi = ~Keyword('1') + Word(nums) + units
Question: I don't have a lot of experience with python project organization, but I've been trying to follow modern best practices using a pyproject.toml and structure as in this video (minus the ci bits). This is ultimately a geoprocessing tool for ArcGIS Pro, though I've tried to abstract the logic of the tool from the 'backend' calls to ArcGIS Pro so that I could create other backends in the future, for example for QGIS or geopandas and rasterio. However, the arcpy module that allows one to interface with ArcGIS Pro is not a module you can just install into your environment from pypi. It simply "exists" in the conda environment provided by the install of ArcGIS Pro. I've been writing and running scripts by activating this conda env. I don't think it would work to list arcpy as a dependency in pyproject.toml since it can't be installed. How do you deal with this situation? What are best practices? Note: The arcpy package might be installable via conda, according to this doc from esri. At the very least, I can clone the default env provided by ArcGIS Pro and modify that (to install dev deps). As stated, I'm quite new to python project management (and it's rather confusing), but if I listed my deps in pyproject.toml, generated a requirements.txt, then installed them into a conda dev env from requirements.txt, might that work? pip can see arcpy in the conda env: PS> pip list Package Version --------------------------------- ----------------- anyio 3.5.0 appdirs 1.4.4 arcgis 2.2.0.1 ... 

Answer:  How do you deal with this situation? What are best practices?  So you should list arcpy in your pyproject.toml or requirements.  It simply "exists" in the conda environment provided by the install of ArcGIS Pro  Great. So if you list the dependency, users will know that you require the ArcGIS Pro to work.  I've tried to abstract the logic of the tool from the 'backend' calls to ArcGIS Pro so that I could create other backends in the future, for example for QGIS or geopandas and rasterio.  Great, so you are making two packages. There is "backend" which implements some interface on top of ArcGIS Pro. That interface is then registered in what you would call "frontend" that does not depend on ArcGIS Pro. These two packages have separate requirements. To give a real life example, pyscada project has https://github.com/pyscada/PyScada/blob/main/setup.py#L33 and then has plugins with separate https://github.com/pyscada/PyScada-BACnet/blob/main/setup.py#L33 dependencies.  if I listed my deps in pyproject.toml, generated a requirements.txt, then installed them into a conda dev env from requirements.txt, might that work?  Yes. You either list the dependencies in pyproject.toml like https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#a-full-example or write requirements.txt, which you can then load dynamically from pyproject.toml like How to reference a requirements.txt in the pyproject.toml of a setuptools project? .
Question: In Tkinter Python, I want to create two frames and place one on top of the other, and then insert a matplotlib plot inside each of these two frames. That, I managed to do. The problem is that, when created, these two frames always seem to have different heights (the bottom one is always smaller than the top one). My goal is to get the two frames to have the same height. Here's a simplified version of the code that produces the said undesirable result : import tkinter as tk from matplotlib.backends.backend_tkagg import FigureCanvasTkAgg import matplotlib.pyplot as plt import numpy as np # Root root = tk.Tk() root.state('zoomed') # Upper frame canv = tk.Frame(root) canv.pack() plt.rcParams["axes.prop_cycle"] = plt.cycler(color=["#4C2A85", "#BE96FF", "#957DAD", "#5E366E", "#A98CCC"]) plt.style.use('ggplot') L = [i for i in range(10)] fig, ax = plt.subplots() l = ax.fill_between(L, L) ax.set_title("Upper plot") canvas = FigureCanvasTkAgg(fig, canv) canvas.draw() canvas.get_tk_widget().pack() # Lower frame canv2 = tk.Frame(root) canv2.pack() fig2, ax2 = plt.subplots() l2 = ax2.fill_between(L, L) ax2.set_title("Lower plot") canvas2 = FigureCanvasTkAgg(fig2, canv2) canvas2.draw() canvas2.get_tk_widget().pack() root.mainloop  Thanks to anyone willing to help.

Answer: Different idea: create one FigureCanvasTkAgg but two axis fig, (ax1, ax2) = plt.subplots(2)  and matplotlib will control size. import tkinter as tk from matplotlib.backends.backend_tkagg import FigureCanvasTkAgg import matplotlib.pyplot as plt import numpy as np root = tk.Tk() #root.state('zoomed') plt.rcParams["axes.prop_cycle"] = plt.cycler(color=["#4C2A85", "#BE96FF", "#957DAD", "#5E366E", "#A98CCC"]) plt.style.use('ggplot') L = [i for i in range(10)] fig, (ax1, ax2) = plt.subplots(2) canvas = FigureCanvasTkAgg(fig, root) canvas.draw() canvas.get_tk_widget().pack(fill='both', expand=True) l1 = ax1.fill_between(L, L) ax1.set_title("Upper plot") l2 = ax2.fill_between(L, L) ax2.set_title("Lower plot") fig.tight_layout() # suggestion from @Zakaria comment root.mainloop()
Question: I have a big Polars dataframe with a lot of groups. Now, I want to partition the dataframe by group and save all sub-dataframes. I can easily do this as follows: for d in df.partition_by(["group1", "group2"]): d.write_csv(f"~/{d[0, 'group1']}_{d[0, 'group2']}.csv")  However, the approach above is sequential and slow when the df is very large and has a whole lot of partitions. Is there any Polars native way to parallelize it (the code section above)? If not, how can I do it in a Python native way instead?

Answer: As far as I am aware, Polars does not currently offer a way to parallelize .write_*() / .sink_*() calls. For Parquet files, this would be similar to writing a "Hive Partitioned Dataset":  https://github.com/pola-rs/polars/issues/11500 https://github.com/pola-rs/polars/issues/15441 # CSV Hive partitioning  Multiprocessing In my own testing - a ProcessPool has performed slightly faster than a ThreadPool for this task. With Polars + multiprocessing - you need to use get_context("spawn")  See: https://docs.pola.rs/user-guide/misc/multiprocessing/  import polars as pl from multiprocessing import get_context # from multiprocessing.dummy import Pool # <- threadpool def write_csv(args): group, df = args df.write_csv(f"{group}.csv") if __name__ == "__main__": df = pl.DataFrame({ "group": [1, 2, 1], "val": ["a", "b", "c"] }) with get_context("spawn").Pool() as pool: groups = df.group_by("group") results = pool.map(write_csv, groups) # this is lazy # we must consume `results` - use empty for loop for result in results: ...   (note: You can use group_by instead of partition_by)  Pool() methods There's a lot of discussion out there about the different Pool methods to use:  .map .imap .imap_unordered  In my simple testing, .map produced the fastest results - with similar memory usage. But depending on your data size + system specs, you may need to investigate the others:  https://stackoverflow.com/a/26521507
Question: Currently, I am getting back multiple responses, or the model doesn't know when to end a response, and it seems to repeat the system prompt in the response(?). I simply want to get a single response back. My setup is very simple, so I imagine I am missing implementation details, but what can I do to only return the single response? from langchain_community.llms import Ollama llm = Ollama(model="llama3") def get_model_response(user_prompt, system_prompt): prompt = f""" 
 
system
 { system_prompt } 
 
user
 { user_prompt } 
 
assistant
 """ response = llm.invoke(prompt) return response 

Answer: Using a PromptTemplate from Langchain, and setting a stop token for the model, I was able to get a single correct response. from langchain_community.llms import Ollama from langchain import PromptTemplate # Added llm = Ollama(model="llama3", stop=["
"]) # Added stop token def get_model_response(user_prompt, system_prompt): # NOTE: No f string and no whitespace in curly braces template = """ 
 
system
 {system_prompt} 
 
user
 {user_prompt} 
 
assistant
 """ # Added prompt template prompt = PromptTemplate( input_variables=["system_prompt", "user_prompt"], template=template ) # Modified invoking the model response = llm(prompt.format(system_prompt=system_prompt, user_prompt=user_prompt)) return response
Question: I have individual x arrays for each intensity (Z) array. Which means that the intensity rows will not be stacked on top of each other. I do not want the tiles to "tilt" so they are connected to the row above and below, I just want them to be straight in the y direction and connected in the x direction. Thank you very much in advance! Here is a dummy code: import numpy as np import matplotlib.pyplot as plt y_values = np.array([0, 1, 2, 3, 4, 5], dtype=float) x_list = range(50, 70) x_values_list = [] z_values_list = [] for i in range(len(y_values)): x_value_single_list = [np.array(0.01*x**2 + 0.1*x*i + 0.3 * i, dtype= float) for x in x_list] x_values_list.append( x_value_single_list) z_values_list.append(np.random.rand(20)) z_values_list = [np.array(arr, dtype=float) for arr in z_values_list] x_values_list = [np.array(arr, dtype=float) for arr in x_values_list] fig, ax = plt.subplots() c = ax.pcolormesh(x_values_list, y_values, z_values_list, cmap='viridis') #, shading='auto' plt.colorbar(c) plt.show()   I have tried a bunch of kwargs but a lot of them does not seem to do anything and seem to not be implemented for pcolormesh and more suited for regular line plots. I have obviously tried a lot with ChatGPT.

Answer: If you use flat shading, where x and y represent the corners of each quadrilateral, then you can plot one row at a time. Note I explicitly set the vmin and vmax parameter in the pcolormesh call so that the colour range will be the same for each row even if their actual max and min differ. import numpy as np import matplotlib.pyplot as plt fig, ax = plt.subplots() y_values = np.array([0, 1, 2, 3, 4, 5], dtype=float) x_list = range(50, 70) for i in range(len(y_values) - 1): x_value_single_list = [np.array(0.01*x**2 + 0.1*x*i + 0.3 * i, dtype= float) for x in x_list] z_values = np.random.rand(1, 19) # 2D array but single row. c = ax.pcolormesh(x_value_single_list, y_values[i:i+2], z_values, cmap='viridis', shading='flat', vmin=0, vmax=1) plt.colorbar(c) plt.show()
Question: I have two numpy matrices A and B (of size m x n, m<>n, m=number of rows, n=number of columns). Both are digital matrices, consisting solely of entries 0 or 1. I want to calculate a matrix product C from these two matrices that obeys the following rules in terms of A(row, column): If A(i,j) = 1 and B(i,j)=1, C(i,j)=1. If A(i,j) = 1 and B(i,j)=0, C(i,j)=-q. If A(i,j) = 0, C(i,j) = 0, regardless of the value of B(i,j).  where q satisfies the equation p-(n-p)*q=0 where p is the number of 1s in B(i,:). I could implement this with a couple of for loops, but I am tempted to ask if there is one of those single line pythonic ways of implementing this logic.

Answer: IIUC you can do: Suppose you have two matrices 5x2: # A [[0 1 0 1 1] [1 1 1 1 0]] # B [[0 0 1 1 1] [0 0 1 1 0]]  Then: m, n = A.shape # 5, 2 p = B.sum(axis=1) # [3 2] q = p / (n - p) # [1.5 0.66666667] m1 = A & B m2 = (A & ~B) * -q[:, None] print(m1 + m2)  Prints: [[ 0. -1.5 0. 1. 1. ] [-0.66666667 -0.66666667 1. 1. 0. ]]
Question: I have text containing assorted transactions that I'm trying to parse using regex. Text looks like this: JT Meta Platforms, Inc. - Class A Common Stock (META) [ST]S (partial) 02/08/2024 03/05/2024 $1,001 - $15,000 F S: New S O: Morgan Stanley - Select UMA Account # 1 JT Microsoft Corporation - Common Stock (MSFT) [ST]S (partial) 02/08/2024 03/05/2024 $1,001 - $15,000 F S: New S O: Morgan Stanley - Select UMA Account # 1 JT Microsoft Corporation - Common Stock (MSFT) [OP]P 02/13/2024 03/05/2024 $500,001 - $1,000,000 F S: New S O: Morgan Stanley - Portfolio Management Active Assets Account D: Call options; Strike price $170; Expires 01/17 /2025 C: Ref: 044Q34N6  I've created a regex to parse out individual transactions, denoted by combination of ticker (eg, (MSFT)), type (eg, [ST], [OP]) and amount (eg, $500,000, etc) as follows: transactions = rx.findall(r"\([A-Z][^$]*\$[^$]*\$[,\d]+", text)  Transactions are returned as a list and look like this for example: (META) [ST]S (partial) 02/08/2024 03/05/2024 $1,001 - $15,000  I'd like to add logic to include description details (ie, 'D:...') if they exist. I tried with the below pattern, but it winds up returning just one large transaction since the first two transactions don't have description details (ie, 'D:'). I'd like to see this: (META) [ST]S (partial) 02/08/2024 03/05/2024 $1,001 - $15,000  .. (MSFT) [ST]S (partial) 02/08/2024 03/05/2024 $1,001 - $15,000  .. (MSFT) [OP]P 02/13/2024 03/05/2024 $500,001 - $1,000,000 F S: New S O: Morgan Stanley - Portfolio Management Active Assets Account D: Call options; Strike price $170; Expires 01/17 /2025  What am I doing wrong? rx.findall(r"\([A-Z][^$]*\$[^$]*\$[,\d]+[\s\S]*?D:(.*)", text)  Edit: To deal with cases where colon isn't contiguous to 'D' (imperfect PDF parsing), added to @zdim's answer and this addresses the above issues: rx.findall('\([A-Z][^$]*\$[^$]*\$[,\d]+(?:[^$]*D:?.+)?', text) 

Answer: Make the "rest" after the transaction part (up to and including D:... line) optional trans = re.findall( r'\([A-Z] [^$]* \$[^$]*\$[,\d]+ (?: [^$]* D:.+ )?', text, re.X) for t in trans: print(t,'\n---')  This prints (META) [ST]S (partial) 02/08/2024 03/05/2024 $1,001 - $15,000 --- (MSFT) [ST]S (partial) 02/08/2024 03/05/2024 $1,001 - $15,000 --- (MSFT) [OP]P 02/13/2024 03/05/2024 $500,001 - $1,000,000 F S: New S O: Morgan Stanley - Portfolio Management Active Assets Account D: Call options; Strike price $170; Expires 01/17 /2025  Now if you need to further process that extra text it should be easy.
Question: Using the dictionary: d = {'1': 'a'}  How can I extract the value a using the JMESPath library in Python? The following attempts did not work: import jmespath value = jmespath.search("1", d) # throws jmespath.exceptions.ParseError: invalid token value = jmespath.search("'1'", d) # returns the key '1' instead of the value 'a' 

Answer: You can directly address an identifier if it is an unquoted-string, so A-Za-z0-9_ and if the string starts with A-Za-z_.  From the grammar rule listed above identifiers can be one or more characters, and must start with A-Za-z_. An identifier can also be quoted. This is necessary when an identifier has characters not specified in the unquoted-string grammar rule. In this situation, an identifier is specified with a double quote, followed by any number of unescaped-char or escaped-char characters, followed by a double quote.  Source: JMESPath documentation, Identifiers Since your identifier is 0-9, you have to use double quoted string for this identifier, so "1". So, your working Python code would be: import jmespath d = {'1': 'a'} value = jmespath.search('"1"', d) print(value)
Question: From the post https://stackoverflow.com/a/53504673/9191338 :  spawn objects are cloned and each is finalised in each process fork/forkserver objects are shared and finalised in the main process  This seems to be the case: import os from multiprocessing import Process import multiprocessing multiprocessing.set_start_method('fork', force=True) class Track: def __init__(self): print(f'{os.getpid()=} object created in {__name__=}') def __getstate__(self): print(f'{os.getpid()=} object pickled in {__name__=}') return {} def __setstate__(self, state): print(f'{os.getpid()=} object unpickled in {__name__=}') return self def __del__(self): print(f'{os.getpid()=} object deleted in {__name__=}') def f(x): print(f'{os.getpid()=} function executed in {__name__=}') if __name__ == '__main__': x = Track() for i in range(2): print(f'{os.getpid()=} Iteration: {i}, Process object created') p = Process(target=f, args=(x,)) print(f'{os.getpid()=} Iteration: {i}, Process created and started') p.start() print(f'{os.getpid()=} Iteration: {i}, Process starts to run functions') p.join()  The output is: os.getpid()=30620 object created in __name__='__main__' os.getpid()=30620 Iteration: 0, Process object created os.getpid()=30620 Iteration: 0, Process created and started os.getpid()=30620 Iteration: 0, Process starts to run functions os.getpid()=30623 function executed in __name__='__main__' os.getpid()=30620 Iteration: 1, Process object created os.getpid()=30620 Iteration: 1, Process created and started os.getpid()=30620 Iteration: 1, Process starts to run functions os.getpid()=30624 function executed in __name__='__main__' os.getpid()=30620 object deleted in __name__='__main__'  Indeed the object is only deleted in the main process. My question is, how is this achieved? Although the new process is forked from the main process, after fork, the new process is essentially another process, how can these two processes share gc information? In addition, does the gc information sharing happen for every object, or just the object passed as argument for subprocess?

Answer: on linux using fork start_method when creating child processes, python uses os._exit() (note the underscore) to terminate the child process once the function ends .... this is somewhat equivalent to the process crashing, therefore no destructors have any chance of being called, the process just terminates, and the OS just reclaims whatever resources were allocated to this process. so you shouldn't rely on destructors being called by child processes to release resources. (for example to shut down an external server)
Question: I'd like to implement sympy.Piecewise on numpy arrays. Consider: import numpy as np from sympy import Piecewise x = np.array([0.1,1,2]) y = np.array([10,10,10]) Piecewise((x * y, x > 0.9),(0, True))  However, when I tried running it I got the following error: TypeError: Argument must be a Basic object, not ndarray Is there a way to get around this? I've tried list comprehension. However, it gets more difficult when there are more variables involved.

Answer: The simpy.Piecewise is not directly compatible with numpy arrays. Use np.where to achieve similar functionality: import numpy as np x = np.array([0.1,1,2]) y = np.array([10,10,10]) result = np.where(x > 0.9, x * y, 0)