eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: My DataFrame is: import pandas as pd df = pd.DataFrame( { 'a': [False, True, False, True, False, True, False, True, True, False, False], } )  Expected output is forming groups like this:  a 1 True 2 False 3 True a 5 True 6 False 7 True a 8 True 9 False 10 False  The logic is: Basically I want to form groups where df.a == True and two rows after that. For example, in order to create the first group, the first True should be found which is row 1. Then the first group is rows 1, 2 and 3. For the second group the next True must be found which is not in the first group. That row is row 5. So the second group is consisted of rows 5, 6 and 7. This image clarifies the point:  And this is my attempt that didn't work: N = 2 mask = ((df.a.eq(True)) .cummax().cumsum() .between(1, N+1) ) out = df[mask] 

Answer: Since you problem is inherently iterative, you must loop. The most straightforward option IMO is to use a simple python loop: def split(df, N=2): i = 0 a = df['a'].to_numpy() while i < len(df): if a[i]: yield df.iloc[i:i+N+1] i+=N i+=1 out = list(split(df))  Output: [ a 1 True 2 False 3 True, a 5 True 6 False 7 True, a 8 True 9 False 10 False]  If you want a simple mask and a unique DataFrame as output, you could improve the speed with numba: from numba import jit @jit(nopython=True) def get_indices(a, N=2): i = 0 out = [] while i < len(a): if a[i]: out.extend([True]*(N+1)) i += N+1 else: out.append(False) i += 1 return out[:len(a)] out = df.loc[get_indices(df['a'].to_numpy())] out['group'] = np.arange(len(out))//3  Output:  a group 1 True 0 2 False 0 3 True 0 5 True 1 6 False 1 7 True 1 8 True 2 9 False 2 10 False 2  Timings for N = 2  relative timings:  Timings for N = 100 NB. @Triky's solution is not providing correct results here  relative timings
Question: I was trying defining a trajectory using velocity and curvature by time; i.e. v(t) and k(t). To get x and y position at time t, I used scipy.integrate.quad but it gives a wrong value. Below is what I've tried. import numpy as np from scipy.integrate import quad # Define v(t) and k(t) def v(t): return 10.0 def k(t): return 0.0 def dtheta(t): # heading rate. return v(t) * k(t); # Compute theta(t) by integrating dtheta def theta(t): result, _ = quad(dtheta, 0, t, epsabs=1e-4, limit=100) # initial heading 0 print(f"heading {result}") return result # return 0 # Define the integrands for x(t) and y(t) def dx(t): ret = v(t) * np.cos(theta(t)) print(f"dx {ret}") return ret def dy(t): return v(t) * np.sin(theta(t)) # Integrate to find x(t) and y(t) def x(t): result, _ = quad(dx, 0, t, epsabs=1e-4, limit=100) # initial x 0 return result def y(t): result, _ = quad(dy, 0, t, epsabs=1e-4, limit=100) # initial y 0 return result print(x(0.5))  As you can see, to test the code, I just put really simple values; constant 10 to velocity and constant 0 to curvature. Therefore, the heading should be constantly 0 and the expected x(0.5) is 5.0. However, it gives a strange number 4.235577740302726e-06. I checked that theta(t) always outputs 0 and dx(t) always outputs 10.0, heading 0.0 dx 10.0 heading 0.0 dx 10.0 heading 0.0 dx 10.0 heading 0.0 dx 10.0 heading 0.0 dx 10.0 heading 0.0 dx 10.0 heading 0.0 dx 10.0 heading 0.0 dx 10.0 heading 0.0 dx 10.0 heading 0.0 dx 10.0 heading 0.0 dx 10.0 heading 0.0 dx 10.0 heading 0.0 dx 10.0 heading 0.0 dx 10.0 heading 0.0 dx 10.0 4.235577740302726e-06  It gives me a correct value (i.e. 5.0) if I manually return 0 for theta(t) but I don't like to do this cause it harms its applicability. What am I doing wrong here?

Answer: You can use solve_ivp(), which is designed for solving initial value problems for ODEs: import numpy as np from scipy.integrate import solve_ivp def system(t, y, v, k): x, y_pos, theta = y dxdt = v(t) * np.cos(theta) dydt = v(t) * np.sin(theta) dthetadt = v(t) * k(t) return [dxdt, dydt, dthetadt] def v(t): return 10.0 * np.exp(-t) def k(t): return 0.1 * np.sin(2 * np.pi * t) ts, te = (0, 0.5), np.linspace(0, 0.5, 100) IVP = solve_ivp(lambda t, y: system(t, y, v, k), ts, [0, 0, 0], t_eval=te) x1, y1 = IVP.y[0, -1], IVP.y[1, -1] print(f"x(0.5) = {x1}, y(0.5) = {y1}")  Prints x(0.5) = 3.89226601538334, y(0.5) = 0.4613683813905819
Question: I'm working on a Python script where I need to make an initial request to obtain an ID. Once I have the ID, I need to make several additional requests to get data related to that ID. I understand that these subsequent requests can be made asynchronously to improve performance. However, I'm not sure how to implement this effectively. Here's a simplified version of my current synchronous approach: import requests # Initial request to get the ID response = requests.get('https://api.example.com/get_id') id = response.json()['id'] # Subsequent requests to get data related to the ID data1 = requests.get(f'https://api.example.com/data/{id}/info1').json() data2 = requests.get(f'https://api.example.com/data/{id}/info2').json() data3 = requests.get(f'https://api.example.com/data/{id}/info3').json() # Processing the data process_data(data1, data2, data3)  I would like to make the requests to info1, info2, and info3 asynchronously. How can I achieve this using asyncio or any other library? I've looked into httpx, but I'm not sure how to structure the code correctly. Any help or example code would be greatly appreciated!

Answer: Similar but using a task group (python >= 3.11) import httpx import asyncio async def main(): async with httpx.AsyncClient() as client: response = await client.get('https://api.example.com/get_id') id = response.json()['id'] async with asyncio.TaskGroup() as tg: task1 = tg.create_task(client.get(f'https://api.example.com/data/{id}/info1')) task2 = tg.create_task(client.get(f'https://api.example.com/data/{id}/info2')) task3 = tg.create_task(client.get(f'https://api.example.com/data/{id}/info3')) # Processing the data process_data(task1.result(), task2.result(), task3.result())
Question: h = 16 fig, ax = plt.subplots(ncols=3, nrows=1, figsize=(15, 5)) for i, q_id in enumerate(sorted_indices[0]): logit = itm_logit[:, q_id, :] prob = torch.nn.functional.softmax(logit, dim=1) name = f'{prob[0, 1]:.3f}_query_id_{q_id}' # Attention map attention_map = avg_cross_att[0, q_id, :-1].view(h, h).detach().cpu().numpy() # Image raw_image_resized = raw_image.resize((596, 596)) ax[0].set_title(name) ax[0].imshow(attention_map, cmap='viridis') ax[0].axis('off') ax[1].set_title(caption) ax[1].imshow(raw_image_resized) ax[1].axis('off') ax[2].set_title(f'Overlay: {name}') ax[2].imshow(raw_image_resized) ax[2].imshow(attention_map, cmap='viridis', alpha=0.6) ax[2].axis('off') ax[0].set_aspect('equal') ax[1].set_aspect('equal') ax[2].set_aspect('equal') plt.tight_layout() plt.savefig(f"./att_maps/{name}.jpg") plt.show() break   What I am trying to do is overlay the attention weights on top of the image (on thrid axes), so I can see which part of the image attention weight is more focused on. However, the code that I put only overlap the attention weight on top of the image. What might be the problem in this case?

Answer: The root cause of this is the different resolution of the image and the attention map. This way, the second imshow call reduced the displayed area to a tiny corner of the original image, with an overlay of the 16x16 attention map. To fix this, the attention map needs to be upscaled (e.g. via np.repeat) to the image resolution. Here's an example:  import numpy as np from matplotlib import pyplot as plt from matplotlib import image attention_map = np.random.rand(16, 16) img = image.imread("merlion.jpg") plt.figure("uneven shapes") plt.imshow(img) plt.imshow(attention_map, cmap='viridis', alpha=0.3) # naive upscaling via np.repeat in both dimensions attention_map_upscale = np.repeat(np.repeat(attention_map, img.shape[0] // attention_map.shape[0], axis=0), img.shape[1] // attention_map.shape[1], axis=1) plt.figure("even shapes") plt.imshow(img) plt.imshow(attention_map_upscale, cmap='viridis', alpha=0.3) plt.show()
Question: I've been looking to retrieve group-based expanding means from the following dataset: df = pd.DataFrame({'id':[1,1,1,2,2,2],'y':[1,2,3,1,2,3]})  and df.groupby('id').expanding().mean().values returns the correct: array([[1. ], [1.5], [2. ], [1. ], [1.5], [2. ]])  However, in my specific case I have to deal with some missing values as well, so that: df2 = pd.DataFrame({'id':[1,1,1,2,2,2],'y':[1,pd.NA,3,1,2,3]})  My expected result applying the same logic would be to ignore the NaN in the computation of the mean, so that from df2.groupby('id').expanding().mean().values I would expect array([[1. ], [1.], [2. ], [1. ], [1.5], [2. ]])  Instead, Pandas returns an error due to applying some type assertion to float in the backend. None of my naive attempts (e.g., .expanding().apply(lambda x: np.nansum(x)) are solving this. Any (possibly equally compact) solution?

Answer: You can convert column 'y' with pd.to_numeric, which will coerce pd.NaN into nan. The latter can be interpreted correctly by the following operations: df2["y"] = pd.to_numeric(df2["y"]) df2 = df2.groupby("id").expanding().mean().values  [[1. ] [1. ] [2. ] [1. ] [1.5] [2. ]]
Question: I have a dataframe having 3 columns, two boolean type and one column as string. from pyspark.sql import SparkSession from pyspark.sql.types import StructType, StructField, BooleanType, StringType # Create a Spark session spark = SparkSession.builder \ .appName("Condition Test") \ .getOrCreate() # Sample data data = [ (True, 'CA', None), (True, 'US', None), (False, 'CA', None) ] # Define schema for the dataframe schema = StructType([ StructField("is_flag", BooleanType(), nullable=False), StructField("country", StringType(), nullable=False), StructField("rule", BooleanType(), nullable=True) ]) # Create DataFrame df = spark.createDataFrame(data, schema=schema) # Show initial dataframe df.show(truncate=False) condition = ( (~col("is_flag")) | ((col("is_flag")) & (trim(col("country")) != 'CA') & nvl(col("rule"),lit(False)) != True) ) df = df.filter(condition) # show filtered dataframe df.show(truncate=False)  Above code is returning below data. +-------+-------+----+ |is_flag|country|rule| +-------+-------+----+ |true |CA |NULL| |true |US |NULL| |false |CA |NULL| +-------+-------+----+  However since I'm explicitely mentioning ((col("is_flag")) & (trim(col("country")) != 'CA') & nvl(col("rule"),lit(False)) != True) ie. trim(col("country")) != 'CA' when is_flag is true, I'm not expecting first record, I need results like below. +-------+-------+----+ |is_flag|country|rule| +-------+-------+----+ |true |US |NULL| |false |CA |NULL| +-------+-------+----+  Question: why the above code also returns 1st record |true |CA |NULL|, where as we have explicitly mentioned country != 'CA' when is_flag is true (boolean). However same when confition is applied via sql returns expected result. select * from df where ( not is_flag or (is_flag and trim(country) != 'CA' and nvl(rule,False) != True) ) 

Answer: The condition is invalid because it doesn't consider operator precedence and hence the wrong result. Operator & has a higher precedence than !=. Here's the updated condition with parentheses: condition = ( (~col("is_flag")) | ((col("is_flag")) & (trim(col("country")) != 'CA') & (nvl(col("rule"),lit(False)) != True)) )  Output: +-------+-------+----+ |is_flag|country|rule| +-------+-------+----+ |true |US |NULL| |false |CA |NULL| +-------+-------+----+
Question: I want to use pandas.read_excel to read an Excel file with the option engine="openpyxl". However, I also want to pass additional optimization options to openpyxl such as:  read_only=True data_only=True keep_links=False  How do I do this?

Answer: These are already implemented by default. From version 2.2: def load_workbook( self, filepath_or_buffer: FilePath | ReadBuffer[bytes], engine_kwargs ) -> Workbook: from openpyxl import load_workbook default_kwargs = {"read_only": True, "data_only": True, "keep_links": False} return load_workbook( filepath_or_buffer, **(default_kwargs | engine_kwargs), )
Question: My DataFrame is: import pandas as pd df = pd.DataFrame( { 'x': ['a', 'a', 'a','b', 'b','c', 'c', 'c',], 'y': list(range(8)) } )  And this is the expected output. I want to create column z:  x y z 0 a 0 NaN 1 a 1 NaN 2 a 2 NaN 3 b 3 3 4 b 4 NaN 5 c 5 NaN 6 c 6 NaN 7 c 7 NaN  The logic is: I want to find the first row after the first group of duplicated rows. For example in column x, the value a is the first duplicated value. I want to find one row after the a values end. And then put the y of that row for z column. This is my attempt that did not give me the output: m = (df.x.duplicated()) out = df[m] 

Answer: One option, using a custom mask: # flag rows after the first group m = df['x'].ne(df['x'].iat[0]).cummax() # pick the first one out = df[m & ~m.shift(fill_value=False)]  If your first value is always a and you want to find the first non-a you could also use: m2 = df['x'].eq('a') out = df[m2.shift(fill_value=False) & ~m2]  Or, if you're sure there is at least one row after the leading as: out = df.loc[[df['x'].ne('a').idxmax()]]  Output:  x y 3 b 3  Some intermediates (all approaches):  x y m ~m.shift(fill_value=False) m2 m2.shift(fill_value=False) df['x'].ne('a') 0 a 0 False True True False False 1 a 1 False True True True False 2 a 2 False True True True False 3 b 3 True True False True True 4 b 4 True False False False True 5 c 5 True False False False True 6 c 6 True False False False True 7 c 7 True False False False True
Question: I am trying to implement a generator that will return a pair of a sequence element and a boolean value indicating whether the element is the last one. from collections.abc import Generator, Iterable from itertools import chain, tee from typing import TypeVar _T1 = TypeVar('_T1') _MISSING = object() def pairwise(iterable: Iterable[_T1]) -> Iterable[tuple[_T1, _T1]]: # See https://docs.python.org/3.9/library/itertools.html#itertools-recipes a, b = tee(iterable) next(b, None) return zip(a, b) def annotated_last(sequence: Iterable[_T1]) -> Generator[tuple[_T1, bool], Any, None]: for current_item, next_item in pairwise(chain(sequence, [_MISSING])): is_last = next_item is _MISSING yield current_item, is_last # <-- mypy error  However, mypy returns this error: Incompatible types in "yield" (actual type "tuple[object, bool]", expected type "tuple[_T1, bool]") Tell me how to correctly annotate types in these functions. I am using Python version 3.9.19

Answer: This is because you created a chain iterator like so: chain(sequence, [_MISSING]), and the type inference has to infer the most generic type from these arguments, but _MISSING is object, so it has to be an iterator of object. Note, you can implement the function you want with the signature you want straightforwardly (albeit, less elegantly) doing something like: from collections.abc import Iterator, Iterable from typing import TypeVar _T1 = TypeVar('_T1') def annotated_last(sequence: Iterable[_T1]) -> Iterator[tuple[_T1, bool]]: it = iter(sequence) try: previous = next(it) except StopIteration: return for current in it: yield previous, False previous = current yield previous, True
Question: Here's a classical example of a decorator implemented by a class: class Decorator: def __init__(self, func): self.func = func def __call__(self, *args, **kwargs): self.func(*args, **kwargs)  How to make __call__ have the same signature and type hints as func has? I've tried the following code: from typing import Callable, TypeVar, ParamSpec, Generic PT = ParamSpec('PT') RT = TypeVar('RT') class Decorator(Generic[PT, RT]): def __init__(self, func: Callable[PT, RT]) -> None: self.func = func def __call__(self, *args: PT.args, **kwargs: PT.kwargs) -> RT: return self.func(*args, **kwargs) @Decorator def add(x: int, y: int) -> int: return x + y  But I failed to get the correct argument list of add in PyCharm. Is it PyCharm's fault? 

Answer: You want to explicitly replace the function with a callable with a specific signature, instead of replacing the function with an instance of a callable class that works the same but does not share the signature. This shows the difference: from typing import Callable, TypeVar, ParamSpec, Generic PT = ParamSpec('PT') RT = TypeVar('RT') class Decorator(Generic[PT, RT]): def __init__(self, func: Callable[PT, RT]) -> None: self.func = func def __call__(self, *args: PT.args, **kwargs: PT.kwargs) -> RT: return self.func(*args, **kwargs) def decorator(func: Callable[PT, RT]) -> Callable[PT, RT]: return Decorator(func) @decorator def add_func_decorated(x: int, y: int) -> int: return x + y @Decorator def add_class_decorated(x: int, y: int) -> int: return x + y print(add_func_decorated(1, 2)) print(add_class_decorated(1, 2))  You will will find that both type hints and type checking work correctly for add_func_decorated, in PyCharm as well. While, for add_class_decorated type checking works, but the type hint displays as you showed in your question.
Question: I am implementing a transition tables as a class attribute with, class Phase: num_states = 15 transition_table = [[False for _ in range(num_states)] for _ in range(num_states)]  but it failed with NameError: name 'num_states' is not defined. However, 1d array works as expected, class Phase: num_states = 15 transition_table = [False for _ in range(num_states)] # this works  I was wondering why this is the case, as num_states is defined before transition_table and it should have access to the previous one? Edit: this is due to Python's implement of class scopes; Use class attribs in outer scope for loop will not give an error, but inner scope will. There is a link in the comment that explains this in detail.

Answer: It looks like num_states is out of scope from the class definition. You can workaround this by passing the class variable into a lambda and run the list comprehension inside the lambda class Phase: num_states = 15 transition_table = (lambda x=num_states: [[False for _ in range(x)] for _ in range(x)])()
Question: I have a 3D space where positions are stored as tuples, eg: (2, 0.5, -4). If I want to know the distance between two points I just do dist = (abs(x1 -x2), abs(y1 - y2), abs(z1 - z2)) and if I want a radius distf = (dist[0] + dist[1] + dist[2]) / 3. Now I have boxes each defined by two min / max positions (eg: (-4 8 -16) to (4, 12, 6)) and I want to know the distance between my point to the closest one: What is the simplest way to know the distance to the closest face in all 3 directions, or 0 in case the position is inside a box? Just looking for the lightest solution that doesn't require numpy or libraries other than defaults like math since I'm not using those in my project. This is my messy solution which should probably work but I'd like to know if there's anything better. point = (8, 12, 16) box_min = (-4, -4, -4) box_max = (4, 4, 4) box_center = ((box_min[0] + box_max[0]) / 2, (box_min[1] + box_max[1]) / 2, (box_min[2] + box_max[2]) / 2) box_scale = (abs(box_max[0] - box_min[0]), abs(box_max[1] - box_min[1]), abs(box_max[2] - box_min[2])) dist = (abs(box_center[0] - point[0]) + box_scale[0] / 2, abs(box_center[1] - point[1]) + box_scale[1] / 2, abs(box_center[2] - point[2]) + box_scale[2] / 2) 

Answer: You can use (dx ** 2 + dy ** 2 + dz ** 2) ** 0.5 for calculating the distance: def _dist(A, B, C): dx = max(B[0] - A[0], 0, A[0] - C[0]) dy = max(B[1] - A[1], 0, A[1] - C[1]) dz = max(B[2] - A[2], 0, A[2] - C[2]) return (dx ** 2 + dy ** 2 + dz ** 2) ** 0.5 def _dist_b(A, B, C): dx = max(B[0] - A[0], 0, A[0] - C[0]) dy = max(B[1] - A[1], 0, A[1] - C[1]) dz = max(B[2] - A[2], 0, A[2] - C[2]) return (dx, dy, dz) print(_dist((8, 12, 16), (-4, -4, -4), (4, 4, 4))) print(_dist_b((8, 12, 16), (-4, -4, -4), (4, 4, 4)))  Prints 14.966629547095765 (4, 8, 12)
Question: I am trying to use the pandas.cumsum() function, but in a way that ignores rows with a value in the ID column that is duplicated and specifically only adds the last value to the cumulative sum, ignoring all earlier values. Example code below (I couldn't share the real code, which is for work). import pandas as pd, numpy as np import random as rand id = ['a','b','c','a','b','e','f','a','b','k'] value = [12,14,3,13,16,7,4,6,10,18] df = pd.DataFrame({'id':id, 'value':value}) df["cumsum_of_value"] = df['value'].cumsum() df["desired_output"] = [ 12,26,29,30,32,39,43,36,30,48 ] df["comments"] = [""]*len(df) df.loc[df.index==0, "comments"]="standard cumsum" df.loc[df.index==1, "comments"]="standard cumsum" df.loc[df.index==2, "comments"]="standard cumsum" df.loc[df.index==3, "comments"]="cumsum of rows 1-3, ignore row 0" df.loc[df.index==4, "comments"]="cumsum of rows 2-4, ignore rows 0, 1" df.loc[df.index==5, "comments"]="cumsum of rows 2-5, ignore rows 0, 1" df.loc[df.index==6, "comments"]="cumsum of rows 2-6, ignore rows 0, 1" df.loc[df.index==7, "comments"]="cumsum of rows 2,4-7, ignore rows 0, 1, 3" df.loc[df.index==8, "comments"]="cumsum of rows 2,5-8, ignore rows 0, 1, 3, 4" df.loc[df.index==9, "comments"]="cumsum of rows 2,5-9, ignore rows 0, 1, 3, 4" print(df)  In this example, there are seven (7) unique values in the ID column (a, b, c ,d, e, f, g), so the cumsum should only ever sum a max of seven (7) records as its output on any row. Is this possible using combinations of functions such as cumsum(), groupby(), duplicated(), drop_duplicates(), and avoiding the use of an iterative loop? I've tried the below df["duped"] = np.where(df["id"].duplicated(keep='last'),0,1) df["value_duped"] = df["duped"] * df["value"] df["desired_output_attempt"] = df["cumsum_of_value"] - df["value_duped"]  But it doesn't come close to the correct answer. I can't think of how to get something like this to result in the desired output without iterating.

Answer: Try: df["out"] = ( df.groupby("id")["value"].transform("diff").fillna(df["value"]).cumsum().astype(int) ) print(df)  Prints:  id value cumsum_of_value desired_output out 0 a 12 12 12 12 1 b 14 26 26 26 2 c 3 29 29 29 3 a 13 42 30 30 4 b 16 58 32 32 5 e 7 65 39 39 6 f 4 69 43 43 7 a 6 75 36 36 8 b 10 85 30 30 9 k 18 103 48 48
Question: I have the following array of four elements: arr = [{"location": 10, "value": 50}, {"location": 21, "value": 70}, {"location": 33, "value": 20}, {"location": 48, "value": 0}]  I would like to create a new array of three elements combining adjacent items: [ {index 0, 1}, {index 1, 2}, {index 2, 3} ] The function to run on each pair is simple: def combine(current, next): return (next["location"] - current["location"]) * current["value"]  Obviously, this can be done using a loop, but is there a more pythonic way of achieving this? I would like to generate the following output: [550, 840, 300] // (21 - 10) * 50, (33 - 21) * 70, (48 - 33) * 20 

Answer: You can use zip(arr, arr[1:]): def get_combine_neighbors(A): _comb = lambda x, y: (y["location"] - x["location"]) * x["value"] return [_comb(x, y) for x, y in zip(A, A[1:])] A = [{"location": 10, "value": 50}, {"location": 21, "value": 70}, {"location": 33, "value": 20}, {"location": 48, "value": 0}] print(get_combine_neighbors(A))  Prints [550, 840, 300]