eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am new to python (I have used R in the past). I have a pandas data frame with one column containing dates. I would like to filter for observations occurring on one specific date. ## Create the pandas DataFrame with column named purchase-date data = ['2023-11-25', '2023-11-24', '2023-11-25', '2023-11-23'] df = pd.DataFrame(data, columns=['purchase-date']) ## convert string format to datetime df['purchase-date'] = pd.to_datetime(df['purchase-date']).dt.date ## Attempt 1: Try setting purchase date as the index and using .loc df.set_index('purchase-date') df.loc['2023-11-24':'2023-11-24'] ## Attempt 2: Filter without making purchase date the index df[(df['purchase-date'] > '2023-11-23') & (df['purchase-date'] < '2023-11-25')]  This is the error I got:  TypeError: '>' not supported between instances of 'datetime.date' and 'str' 

Answer: The issue you're encountering is related to the fact that you're trying to compare a datetime.date object with a string in your second attempt. filter based on dates, you need to compare datetime.date objects : import pandas as pd data = ['2023-11-25', '2023-11-24', '2023-11-25', '2023-11-23'] df = pd.DataFrame(data, columns=['purchase-date']) df['purchase-date'] = pd.to_datetime(df['purchase-date']).dt.date # Filter without making purchase date the index start_date = pd.to_datetime('2023-11-23').date() end_date = pd.to_datetime('2023-11-25').date() filtered_df = df[(df['purchase-date'] > start_date) & (df['purchase-date'] < end_date)] print(filtered_df)
Question: So for example, I have a dataframe like this  Value Placement 0 12 high 1 15 high 2 18 high 3 14 high 4 4 low 5 5 low 6 9 high 7 11 high 8 2 low 9 1 low 10 3 low 11 2 low  I want to create a second dataframe that contains the the highest value in the "Value" column for each set of consecutive rows with "high" placement, and the lowest value in the "Value" column for each set of consecutive rows with "low" placement. So something like  Value Placement 0 18 high 1 4 low 2 11 high 3 1 low  I also don't want to change the order of the rows, as the order of the "highs" and "lows" is critical to the functionality of the project. I could just iterate through the original dataframe and keep track of the the numbers in "Value" until a change in "Placement" is detected, but I've heard dataframe iteration is very slow and should be avoided if possible. Is there some way to do this without iteration? TIA

Answer: Group by consecutive values, swap the sign for Placement that match "low", and get the idxmax per group, then keep the selected rows with loc: # group consecutive rows group = df['Placement'].ne(df['Placement'].shift()).cumsum() # invert the low values, get idxmax per group keep = (df['Value'] .mul(df['Placement'].map({'low': -1, 'high': 1})) .groupby(group, sort=False).idxmax() ) out = df.loc[keep]  If efficiency is a concern, and since groupby is based on a python loop, another approach (that is potentially faster for many groups) would be to stable-sort the rows by value and group (using numpy.lexsort) and keep the highest (after sign swap for "low") using drop_duplicates: group = df['Placement'].ne(df['Placement'].shift()).cumsum() s = df['Value'].mul(df['Placement'].map({'low': -1, 'high': 1})) keep = (group .iloc[np.lexsort([s, group])] .drop_duplicates(keep='last') .index ) out = df.loc[keep]  Note that despite the sorting step, this strategy will maintain the relative original order of the rows. Output:  Value Placement 2 18 high 4 4 low 7 11 high 9 1 low  Comparison of timing:
Question: I have the following variables to use in my Jinja template: list_python_version = [3, 9] all_python_version = [ [3, 8], [3, 9], [3, 10], [3, 11], [3, 12] ]  Is there a way to use a combination of Jinja filters and tests so it iterates over all_python_version, checks that both the first and second elements of the list are greater or equal than the elements of list_python_version, and when the conditions are met, it generates the string for MAJOR.MINOR version and joins all that are valid in a single string? This way, considering the variables above, the rendered template should give 3.9, 3.10, 3.11, 3.12? I have tried the following expression: {{ all_python_version | select('[0] >= minimal_python_version[0] and [1] >= minimal_python_version[1]') | join('.') | join(', ') }}  But it will fail since the select filter asks for a function, and so far I have not found in Jinja's documentation any hint as to how we can use conditionals to filter values inside an expression. Alternatively, the solution could encompass a for loop in Jinja, but I need the string to be rendered in one line, and if we do: {% for version in all_python_version %} {% if version[0] >= list_python_version[0] and version[1] >= list_python_version[1] %} {{ version[0] ~ '.' ~ version[1] ~ ',' }} {% endif %} {% endfor %}  Each version will render in its own line, with the last , also being rendered. Is there a way to do get the versions to be rendered in a single line in pure Jinja and its plugins?

Answer: Conveniently enough, since Jinja allows you to access elements of a list via both list[0] and list.0, this means that you can actually use a dictionary filter on a list. And selectattr is just the filter we need here, since it allows to select items out of a list of dictionaries based on a property of those dictionaries. Still, we cannot properly fit a logical and in there, so we'll need to split the logic in two:  all Python versions where the major version is strictly equal to our comparison point and the minor version is greater or equal all Python versions where the major version is greater than our comparison point, regardless of the minor version  So, we end up with a quite lengthy: {{ all_python_version | selectattr(0, '==', list_python_version.0) | selectattr(1, '>=', list_python_version.1) | list + all_python_version | selectattr(0, '>', list_python_version.0) | list }}   The snippet {%- set list_python_version = [3, 9] -%} {%- set all_python_version = [ [3, 8], [3, 9], [3, 10], [3, 11], [3, 12], ] -%} {{ all_python_version | selectattr(0, '==', list_python_version.0) | selectattr(1, '>=', list_python_version.1) | list + all_python_version | selectattr(0, '>', list_python_version.0) | list }}  Would yield: [[3, 9], [3, 10], [3, 11], [3, 12]]   Then to join everything together, use a combination of map and join in the list of lists and a simple join in the resulting list: {{ ( all_python_version | selectattr(0, '==', list_python_version.0) | selectattr(1, '>=', list_python_version.1) | list + all_python_version | selectattr(0, '>', list_python_version.0) | list ) | map('join', '.') | join(', ') }}  Giving 3.9, 3.10, 3.11, 3.12
Question: I am using this video tutorial: Video I am making a ball (with 0 mass at the moment) in pymunk and trying to show it on pygame but it is not working I tried doing this to make a ball in pymunk and pygame and I was expecting a ball with no movement(I will make it move later on): import pymunk import pygame pygame.init() space=pymunk.Space() FPS=60 clock = pygame.time.Clock() body = pymunk.Body() body.position = 400,400 shape=pymunk.Circle(body,10) space.add(body,shape) (width, height) = (600, 400) display = pygame.display.set_mode((width, height)) def main(): while True: for event in pygame.event.get(): if event.type == pygame.QUIT: return display.fill((255,255,255)) x,y=body.position pygame.draw.circle(display,(255,0,0),(int(x),int(y)),10) clock.tick(FPS) space.step(1/FPS) main() pygame.display.quit() pygame.quit()  I got an error saying that the position is Vec2d(Nan,Nan) Then, I ran this: print(body.position) Output:Vec2d(nan, nan) body.position = 400, 400 print(body.position) Output:Vec2d(nan, nan) 

Answer: When creating the body, you must specify a mass and a moment not equal to 0 (see pymunk.Body): body = pymunk.Body() body = pymunk.Body(1, 1)  In addition, pygame.disaply.flip() is missing after drawing the objects in the scene and if the body should fall, you must also specify the gravity (e.g. space.gravity = (0, 100))
Question: I'd like to know if I can use pydantic to check whether arguments would be fit for calling a type hinted function, but without calling the function. For example, given kwargs = {'x': 1, 'y': 'hi'} def foo(x: int, y: str, z: Optional[list] = None): pass  I want to check whether foo(**kwargs) would be fine according to the type hints. So basically, what pydantic.validate_call does, without calling foo. My research led me to this github issue. The solution works, but it relies on validate_arguments, which is deprecated. It cannot be switched for validate_call, because its return value doesn't have a vd attribute, making the line validated_function = f.vd fail.

Answer: You can extract the annotations attribute from the function and use it to build a pydantic model using the type constructor: def form_validator_model(func: collections.abc.Callable) -> type[pydantic.BaseModel]: ann = func.__annotations__.copy() ann.pop('return', None) # Remove the return value annotation if it exists. return type(f'{func.__name__}_Validator', (pydantic.BaseModel,), {'__annotations__': ann}) def func(a: str, b: int) -> str: return a * b model = form_validator_model(func) model(a='hi', b='bye') # raises ValidationError  The downside is you can't call the arguments positionally.
Question: I have a torch tensor like so: a=[1, 234, 54, 6543, 55, 776]  and other tensors like so: b=[234, 54] c=[55, 776]  I want to create a new mask tensor where the values of a will be true if there is another tensor (b or c) are equal to it. For example, in the tensors we have above I would like to create the following masking tensor: a_masked =[False, True, True, False, True, True] # The first two True values correspond to tensor `b` while the last two True values correspond to tensor `c`.  I have seen other methods to check whether a full tensor is contained in another but this isn't the case here.  Is there a torch way to do this efficiently? Thanks!

Answer: Based on the answers to on the PyTorch forum here, you could explicitly use a for loop, e.g., import torch a = torch.tensor([1, 234, 54, 6543, 55, 776]) b = torch.tensor([234, 54]) c = torch.tensor([55, 776]) a_masked = sum(a == i for i in b).bool() + sum(a == i for i in c).bool() print(a_masked) tensor([False, True, True, False, True, True])  However, there is actually a PyTorch isin function, for which you could do: a_masked = torch.isin(a, torch.cat([b, c]))  This is several times faster than the sum method.
Question: Suppose I have the following table:  import pandas as pd data = pd.DataFrame({ 'Group':['A','A','A','A','B','B'] , 'Month':[1,2,3,4,1,2] , 'Value':[100,300,700,750, 200,400] })  I would like to use groupby and transform functions in pandas to create a new column that is equal to the value of each group in month 2. Here's how the result should look: import pandas as pd data = pd.DataFrame({ 'Group':['A','A','A','A','B','B'] , 'Month':[1,2,3,4,1,2] , 'Value':[100,300,700,750, 200,400] , 'Desired_Result':[300,300,300,300,400,400] })  It seems like there should be a straightforward way of accomplishing this with groupby and transform, but haven't found it yet.

Answer: Use Series.map with filtered rows in boolean indexing: s = data[data['Month'].eq(2)].set_index('Group')['Value'] data['Desired_Result'] = data['Group'].map(s) print (data) Group Month Value Desired_Result 0 A 1 100 300 1 A 2 300 300 2 A 3 700 300 3 A 4 750 300 4 B 1 200 400 5 B 2 400 400  With GroupBy.transform is possible replace non matched values by NaNs and use first: s = data['Value'].where(data['Month'].eq(2)) data['Desired_Result'] = s.groupby(data['Group']).transform('first') print (data) Group Month Value Desired_Result 0 A 1 100 300.0 1 A 2 300 300.0 2 A 3 700 300.0 3 A 4 750 300.0 4 B 1 200 400.0 5 B 2 400 400.0
Question: Say I have two dataframes: data1 = { 'Name': ['Alice', 'Bob', 'Charlie'], 'Age': [25, 30, 35], 'Height': [165, 182, 177] } df1 = pd.DataFrame(data1) data2 = { 'Summary': ["Alice is the youngest", "Bob is the tallest"] } df2 = pd.DataFrame(data2)  Is there a way I could vertically concatenate the two dfs in such a way: +---+---------+-----+----------+ | | Name | Age | Height | +---+---------+-----+----------+ | 0 | Alice | 25 | 165 | | 1 | Bob | 30 | 182 | | 2 | Charlie | 35 | 177 | +---+---------+-----+----------+ | | Summary | +---+--------------------------+ | 0 | "Alice is the youngest" | | 1 | "Bob is the tallest" | +---+--------------------------+  Everything I have attempted so far either groups all fours columns together at the top.

Answer: You can use tabulate and post-process the outputs: import tabulate as t t.PRESERVE_WHITESPACE = True width1 = 10 width2 = width1 * df1.shape[1] + df1.shape[1]*2 str1 = t.tabulate(df1.applymap(f'{{:^{width1}}}'.format), list(df1), tablefmt='outline', stralign='center', numalign='center') str2 = t.tabulate(df2.applymap(f'{{:^{width2}}}'.format), df2.columns, tablefmt='outline', stralign='center', numalign='center') print(str1 + '\n' + '\n'.join(str2.splitlines()[1:]))  Output: +----+------------+------------+------------+ | | Name | Age | Height | +====+============+============+============+ | 0 | Alice | 25 | 165 | | 1 | Bob | 30 | 182 | | 2 | Charlie | 35 | 177 | +----+------------+------------+------------+ | | Summary | +====+======================================+ | 0 | Alice is the youngest | | 1 | Bob is the tallest | +----+--------------------------------------+  Or to adjust df2 dynamically from df1's width: import tabulate as t t.PRESERVE_WHITESPACE = True str1 = tabulate(df1, list(df1), tablefmt='outline', stralign='center', numalign='center') H = str1.split('\n', 1)[0] L = len(H)-H[1:].index('+')-5 str2 = tabulate(df2.applymap(f'{{:^{L}}}'.format), list(df2), tablefmt='outline', stralign='center', numalign='center') print(str1 + '\n' + '\n'.join(str2.splitlines()[1:]))  Output: +----+---------+-------+----------+ | | Name | Age | Height | +====+=========+=======+==========+ | 0 | Alice | 25 | 165 | | 1 | Bob | 30 | 182 | | 2 | Charlie | 35 | 177 | +----+---------+-------+----------+ | | Summary | +====+============================+ | 0 | Alice is the youngest | | 1 | Bob is the tallest | +----+----------------------------+
Question: This works well and returns 249999500000: sum([x for x in range(1_000_000) if x%2==0])  This is slower but still returns 24999995000000: sum([x for x in range(10_000_000) if x%2==0])  However, larger range of values such as 1_000_000_000 takes a very long time to compute. In fact, this returns an error: sum([x for x in range(10_000_000_000) if x%2==0])  Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 1, in <listcomp> MemoryError.  I've seen posts saying that the built-in sum function is the fastest but this doesn't look like being fast to me. My question then is, do we have a faster Pythonic approach to compute large sums like this? Or is it perhaps a hardware limitation?

Answer: After I posted the below code based on the Gauss Sum, Kelly Bundy pointed out a simplification of the math and a practical simplification making use of properties of a range object. He suggested I add this here. An arbitrary range can be visualized as a trapezoid. That there is the sum for the area of a trapezoid. It also uses the ability of a range to know its length. That simplifies my (stop - start + step-1) // step expression. def sum_range(*args): rg = range(*args) return (rg[0] + rg[-1]) * len(rg) // 2 if rg else 0  Following is my original answer. I've gridsearched both functions for being identical to sum_naive, given argument ranges that cover all the cases.  for arbitrary start, stop, step: #def sum_naive(start, stop, step): return sum(range(start, stop, step)) def sum_gauss(start, stop, step=1): if step == 0: raise ValueError("range() arg 3 must not be zero") if step < 0: step = -step shift = 1 + (start - stop - 1) % step (start, stop) = (stop+shift, start+1) if start >= stop: return 0 # triangle on a base n = (stop - start + step-1) // step base = start * n triangle = n * (n - 1) // 2 * step return base + triangle  A "geometric" interpretation should make these expressions self-evident. ∎ SCNR The negative step case is rewritten into a positive step case using equivalent start and stop. The sums of even and odd integers in a given range start, stop are simply those: one_half = sum_gauss(start, stop, 2) other_half = sum_gauss(start+1, stop, 2) # no need to know which are which assert one_half + other_half == sum_gauss(start, stop, 1)
Question: I am working on a task where I have a Pandas DataFrame using Polars library in Python, containing columns for 'ID' and 'Timestamp'. Each row represents the end of a session identified by the 'Timestamp'. I am trying to create a new column called 'time_since_last_session', which should contain the time duration between sessions for each unique 'ID'. I have been able to compute the time difference between sessions for a specific filtered 'ID' using the following code: import polars as pl # DataFrame: sessions_features # Columns: 'ID', 'Timestamp' filtered_id = "BBIISSIOTNIFSIDYIUSA" time_diff = sessions_features.filter(pl.col("ID") == filtered_id)["Timestamp"].diff().dt.total_seconds()  However, I'm struggling to perform this calculation for each 'ID' in the DataFrame using a group_by() operation or similar. I have attempted to use map_groups() but haven't been successful. Could someone please guide me on how to perform this operation efficiently for each 'ID' using Polars? A minimal reproducible example would be this: import polars as pl import pandas as pd # Creating a sample DataFrame data = { 'ID': ['A', 'A', 'A', 'B', 'B', 'B'], 'Timestamp': ['2023-01-01 10:00:00', '2023-01-01 10:30:00' ,'2023-01-01 11:00:00', '2023-01-01 12:00:00', '2023-01-01 12:30:00', '2023-01-01 13:00:00'] } df = pd.DataFrame(data) # Converting to Polars DataFrame sessions_features = pl.from_pandas(df) sessions_features = sessions_features.with_columns( pl.col("Timestamp").str.to_datetime() ) print(sessions_features.filter(pl.col("ID") == "A")["Timestamp"].diff().dt.total_seconds())  This example creates a sample DataFrame and calculates the time difference between sessions for a specific 'ID'. However, the goal is to perform this calculation for each unique 'ID' in the DataFrame efficiently using Polars. Any help or insights would be greatly appreciated! the expected result for the final df in the minimum example would be: ┌─────┬─────────────────────┬───────────────────────┐ │ ID ┆ Timestamp ┆ time_between_sessions │ │ --- ┆ --- ┆ --- │ │ str ┆ datetime[μs] ┆ i64 │ ╞═════╪═════════════════════╪═══════════════════════╡ │ A ┆ 2023-01-01 10:00:00 ┆ 0 │ │ A ┆ 2023-01-01 10:30:00 ┆ 1800 │ │ A ┆ 2023-01-01 11:00:00 ┆ 1800 │ │ B ┆ 2023-01-01 12:00:00 ┆ 0 │ │ B ┆ 2023-01-01 12:30:00 ┆ 1800 │ │ B ┆ 2023-01-01 13:00:00 ┆ 1800 │ └─────┴─────────────────────┴───────────────────────┘ 

Answer: You can group by ID and then apply the rolling diff to each group: df.group_by("ID").map_groups( lambda g: g.with_columns( pl.col("Timestamp").diff().dt.total_seconds().fill_null(0).alias("Diff") ) )
Question: I have a pytree represented by a list of lists holding parameter tuples. The sub-lists all have the same structure (see example). Now I would like to create a weighted sum so that the resulting pytree has the same structure as one of the sub-lists. The weights for each sub-list are stored in a separate array / list. So far I have the following code that seems to works but requires several steps and for-loop that I would like avoid for performance reasons. import jax import jax.numpy as jnp list_1 = [ [jnp.asarray([[1, 2], [3, 4]]), jnp.asarray([2, 3])], [jnp.asarray([[1, 2], [3, 4]]), jnp.asarray([2, 3])], ] list_2 = [ [jnp.asarray([[2, 3], [3, 4]]), jnp.asarray([5, 3])], [jnp.asarray([[2, 3], [3, 4]]), jnp.asarray([5, 3])], ] list_3 = [ [jnp.asarray([[7, 1], [4, 4]]), jnp.asarray([6, 2])], [jnp.asarray([[6, 4], [3, 7]]), jnp.asarray([7, 3])], ] weights = [1, 2, 3] pytree = [list_1, list_2, list_3] weighted_pytree = [jax.tree_map(lambda tree: weight * tree, tree) for weight, tree in zip(weights, pytree)] reduced = jax.tree_util.tree_map(lambda *args: sum(args), *weighted_pytree) 

Answer: I think this will do what you have in mind: def wsum(*args, weights=weights): return jnp.asarray(weights) @ jnp.asarray(args) reduced = jax.tree_util.tree_map(wsum, *pytree)  For the edited question, where tree elements have more general shapes, you can define wsum like this instead: def wsum(*args, weights=weights): return sum(weight * arg for weight, arg in zip(weights, args))
Question: I'm trying to execute my function using spark_df.foreachPartition(), and I want to pass additional parameter but apparently the function supports only one parameter (the partition). I tried to play with it and do something like this : def my_function(row, index_name) : return True def partition_func(row): return my_function(row, "blabla") spark_df.foreachPartition(partition_func)  However, I'm getting a serialization error : _pickle.PicklingError: Could not serialize object: TypeError: Cannot serialize socket object  How can I make this work? I know I can add parameters to my Spark Dataframe, but I think it's an ugly solution, sending it in function parameter is so much better.

Answer: There might be other ways, but one simple approach could be to create a broadcast variable (or a container that holds any variables you may need), and then pass it to be used in your foreachPartition function. Something like this: def partition_func_with_var(partition, broadcast_var): for row in partition: print(str(broadcast_var.value) + row.desc) df = spark.createDataFrame([(1,"one"),(2,"two")],["id","desc"]) bv = spark.sparkContext.broadcast(" some extra variable ") df.foreachPartition(lambda p: partition_func_with_var(p,bv))  Note that "passing a variable" has a little murky meaning here, as it is actually a broadcast operation, with all its consequences and limitations (read-only, sent once, etc.)
Question: The following is an example of the .csv file of thousands of lines I have for the different bus lines. Table:     trip_id arrival_time departure_time stop_id stop_sequence stop_headsign     107_1_D_1 6:40:00 6:40:00 AREI2 1    107_1_D_1 6:40:32 6:40:32 JD4 2    107_1_D_1 6:41:27 6:41:27 PNG4 3      Raw Data: trip_id,arrival_time,departure_time,stop_id,stop_sequence,stop_headsign 107_1_D_1,6:40:00,6:40:00,AREI2,1, 107_1_D_1,6:40:32,6:40:32,JD4,2, 107_1_D_1,6:41:27,6:41:27,PNG4,3,  I want to create a table or dataframe that creates a line for each road segment and calculates the time between each arrival_time. Expected result:  some other trip_id may share the same RoadSegment

Answer: I think in such cases you can use shift. Here is an example: df = pd.read_csv('...') result = pd.DataFrame() for _, trip_df in df.groupby('trip_id', sort=False): # type: str, pd.DataFrame trip_df = trip_df.sort_values('stop_sequence') trip_df['arrival_time'] = pd.to_timedelta(trip_df['arrival_time']) trip_df['departure_time'] = pd.to_timedelta(trip_df['departure_time']) trip_df['prev_arrival_time'] = trip_df['arrival_time'].shift() trip_df['prev_stop_id'] = trip_df['stop_id'].shift() trip_df['RoadSegment'] = trip_df['prev_stop_id'].str.cat(trip_df['stop_id'], sep='-') trip_df['planned_duration'] = trip_df['departure_time'] - trip_df['prev_arrival_time'] trip_df = trip_df.dropna(subset=['planned_duration']) trip_df['planned_duration'] = ( trip_df['planned_duration'] .apply(lambda x: x.total_seconds()) .astype(int) ) result = pd.concat( [result, trip_df[['RoadSegment', 'trip_id', 'planned_duration']]], sort=False, ignore_index=True, ) print(result)
Question: os package has os.path.islink(path) to check if path is a symlink. How do I determine if a file located at path is a relative symlink? Note that path can be an absolute path (i.e. path == os.path.abspath(path)), e.g. /home/user/symlink that is a relative symlink to ../ which is relative. As opposed to a link to /home/user, which points to the same directory but is absolute.

Answer: I think you just have to read the actual target pathname and check it. def isrellink(path): return os.path.islink(path) and not os.path.isabs(os.readlink(path))