eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Let's say I open a python console inside /home/me/symlink/dir, which is actually a shortcut to /home/me/path/to/the/dir. In this console I execute the following code: from pathlib import Path path = Path.cwd() print(path) # "/home/me/path/to/the/dir"  So cwd() actually resolves the symlinks and get the absolute path automatically and I am searching how to avoid this behavior. If the path with symlink is provided at the Path object declaration, it seems to work as I would like: from pathlib import Path path = Path("/home/me/symlink/dir") print(path) # "/home/me/symlink/dir"  Is there a way to get the current working directory keeping the symlink other than providing the path at object instantiation ?

Answer: you'll need to use os.getcwd() . Path.cwd() internally uses os.getcwdb() which resolves symlinks, while os.getcwd() preserves them. from pathlib import Path import os path = Path(os.getcwd()) print(path)  If you need work with symlinks in general. path.is_symlink() - check if a path is a symlink path.resolve() - resolve symlinks to get the actual path path.readlink() - get the target of a symlink path = Path(os.getcwd()) if path.is_symlink(): real_path = path.resolve() # "/home/me/path/to/the/dir" symlink_target = path.readlink() # Gets the symlink target  Edit 1: For your test case, The PWD environment variable often contains the path with preserved symlinks, as it's typically maintained by the shell. import os from pathlib import Path def get_symlinked_cwd(): return Path(os.getenv("PWD", os.getcwd())) symlink_path = get_symlinked_cwd() print(symlink_path)
Question: While looking for a way to generate spatially varying noise, I came across this answer, which is able to do what I wanted. But I am getting confused about how the code works. From what I understand, the first step of the code generates a gaussian kernel: import numpy as np import scipy.signal import matplotlib.pyplot as plt # Compute filter kernel with radius correlation_scale (can probably be a bit smaller) correlation_scale = 150 x = np.arange(-correlation_scale, correlation_scale) y = np.arange(-correlation_scale, correlation_scale) X, Y = np.meshgrid(x, y) print(X.shape,Y.shape) dist = np.sqrt(X*X + Y*Y) filter_kernel = np.exp(-dist**2/(2*correlation_scale))  which when visualized looks as follows:  The second step of the code generates a random noise grid: n = 512 noise = np.random.randn(n, n)  that looks like:  The third step convolves the random noise generated in step 2 with the filter generated in step 1. noise1 = scipy.signal.fftconvolve(noise, filter_kernel, mode='same')  and the output of this step looks like this:  My question is how does the output of step 3 end up looking like this instead of a smoothed out version of the random noise? Isn't convolution with a Gaussian kernel the same as applying a Gaussian blur? For instance, if I apply a Gaussian filter on the random noise generated, the output would look like this: from scipy.ndimage import gaussian_filter noise = gaussian_filter(noise , sigma=1, radius=10)   Why are the last two images so different from each other?

Answer: I've played a bit with the code you provided, and this seems to be related to the fact that you use standard deviations for your gaussian kernel that are very different, your correlation_scale is 150 in the first example whereas sigma is one in your second example. If I take similar values for both, I get similar results. Unrelated, but feel free to truncate your gaussian kernel a bit (noise_trunc=noise[len(X)//2-50:len(X)//2+50, len(Y)//2-50:len(Y)//2+50]), or at least to set the very low coeffs to 0 (noise[noise<1e-5]=0), to greatly increase the speed of your computations
Question: I am trying to filter my Polars DataFrame for dates that are nearest to a given date. For example: import polars import datetime data = { "date": ["2025-01-01", "2025-01-01", "2025-01-01", "2026-01-01"], "value": [1, 2, 3, 4], } df = polars.DataFrame(data).with_columns([polars.col("date").cast(polars.Date)])  shape: (4, 2) ┌────────────┬───────┐ │ date ┆ value │ │ --- ┆ --- │ │ date ┆ i64 │ ╞════════════╪═══════╡ │ 2025-01-01 ┆ 1 │ │ 2025-01-01 ┆ 2 │ │ 2025-01-01 ┆ 3 │ │ 2026-01-01 ┆ 4 │ └────────────┴───────┘  Given a date, say: date = datetime.date(2024, 12, 31)  I want to filter the DataFrame for rows where the date column only includes records that are closest to my required date. I know that I can do the following: result = df.with_columns( diff=(polars.col("date") - date).abs() ).filter( polars.col("diff") == polars.min("diff") )  shape: (3, 3) ┌────────────┬───────┬──────────────┐ │ date ┆ value ┆ diff │ │ --- ┆ --- ┆ --- │ │ date ┆ i64 ┆ duration[ms] │ ╞════════════╪═══════╪══════════════╡ │ 2025-01-01 ┆ 1 ┆ 1d │ │ 2025-01-01 ┆ 2 ┆ 1d │ │ 2025-01-01 ┆ 3 ┆ 1d │ └────────────┴───────┴──────────────┘  Is there a more succinct way to achieve this (without creating a new column, for example)?

Answer: You don't need to add the temporary column, just filter directly: df.filter((m:=(pl.col('date')-date).abs()).min() == m)  Or, without the walrus operator: diff = (pl.col('date')-date).abs() df.filter(diff.min() == diff)  Output: ┌────────────┬───────┐ │ date ┆ value │ │ --- ┆ --- │ │ date ┆ i64 │ ╞════════════╪═══════╡ │ 2025-01-01 ┆ 1 │ │ 2025-01-01 ┆ 2 │ │ 2025-01-01 ┆ 3 │ └────────────┴───────┘
Question: I have a table like:  | 2022 | 2022 | 2021 | 2021 class | A | B | A | B -----------|------|------|------|------ X | 1 | 2 | 3 | 4 Y | 5 | 6 | 7 | 8  How can I transform it to following form? year | category | class | value ---------------------------------- 2022 | A | X | 1 2022 | A | Y | 5 2022 | B | X | 2 2022 | B | Y | 6 2021 | A | X | 3 2021 | A | Y | 7 2021 | B | X | 4 2021 | B | Y | 8  I tried various combinations of pd.melt with no success.. Thx in advance!

Answer: You could melt with ignore_index=False and rename_axis/rename: out = (df.rename_axis(columns=['year', 'category']) .melt(ignore_index=False) .reset_index() )  Or: out = (df.melt(ignore_index=False) .rename(columns={'variable_0': 'year', 'variable_1': 'category'}) .reset_index() )  Output:  class year category value 0 X 2022 A 1 1 Y 2022 A 5 2 X 2022 B 2 3 Y 2022 B 6 4 X 2021 A 3 5 Y 2021 A 7 6 X 2021 B 4 7 Y 2021 B 8  Reproducible input: df = pd.DataFrame.from_dict({'index': ['X', 'Y'], 'columns': [('2022', 'A'), ('2022', 'B'), ('2021', 'A'), ('2021', 'B')], 'data': [[1, 2, 3, 4], [5, 6, 7, 8]], 'index_names': ['class'], 'column_names': [None, None]}, orient='tight')
Question: I would like to achieve in Python the following semantics which is present in Kotlin: myset = set() for elem in arr: if not myset.add(elem): return 'duplicate found'  I am looking for a way to get a slight performance boost by 'hijacking' add-to-set operation by analysing the returned value. It should be better if compared to the following approach I already know in Python: myset = set() for elem in arr: if elem in myset: return 'duplicate found' myset.add(elem)  Yet as far as I can observe add() method of set returns None no matter what the outcome of adding is. Is there such a method in Python3? Looking into the documentation I have not managed to find such a result returning add-to-set method yet.

Answer: No, and it has been rejected.
Question: I have a simple Click app like this: import click @click.command() @click.argument('message') def main(message: str): click.echo(message) if __name__ == '__main__': main()  When you pass an environment variable in the argument, it expands it: ➜ Desktop python foo.py '$M0/.viola/2025-01-25-17-20-23-307878' M:/home/ramrachum/.viola/2025-01-25-17-20-23-307878  Note that I used single quotes above, so my shell is not expanding the environment variable, Click does. How do I get Click to not expand it?

Answer: The solution is to pass windows_expand_args=False when calling the main command.
Question: My organization has account numbers that are comprised of combining multiple fields. The last field is always 4 characters (typically 0000) Org Account 01 01-123-0000 01 01-456-0000 02 02-789-0000 02 02-456-0000 03 03-987-0000 03 03-123-1234  I also have a dictionary mapping of how many characters the last component should be. MAP = {'01': 4, '02': 3, '03': 3}  However there also special mappings for Org 03: D03_SPECIAL_MAP = {'0000': '012', '1234': '123'}  My code to update the last component is: for i,r in df.iterrows(): updated = False # Keep track if we have updated this row # Split off last component from the rest of the account Acct, last_comp = r['Account'].rsplit('-',1) # Check if we need to update code length and the code length does not match if r['Org'] in MAP and len(last_comp) != MGMT_MAP[r['Org']]: df.at[i,'Account'] = '-'.join(Acct) + "-" + last_comp.zfill(MAP[r['Org']]) updated = True # Special mapping for Org 03 if r['Org'] =='03' and last_comp in D03_SPECIAL_MAP.keys(): df.at[i,'Account'] = '-'.join(Acct) + "-" + D03_SPECIAL_MAP[last_comp] updated = True if not updated: # Join Default if we have not hit either of he conditions above df.at[i,'Account'] = '-'.join(Acct) + "-" + last_comp  The output of this will be: Org Account 01 01-123-0000 01 01-456-0000 02 02-789-000 02 02-456-000 03 03-987-012 03 03-123-123  My code works as expected except this process is a little slow to check every record. Is there a way to perform the same operation without using df.iterrows()?

Answer: Since working with strings is hardly vectorizable, just use a simple python function and a list comprehension: def replace(org, account): a, b = account.rsplit('-', maxsplit=1) if org == '03': return f'{a}-{D03_SPECIAL_MAP[b]}' return f'{a}-{b[:MAP[org]]}' df['Account'] = [replace(o, a) for o, a in zip(df['Org'], df['Account'])]  Output:  Org Account 0 01 01-123-0000 1 01 01-456-0000 2 02 02-789-000 3 02 02-456-000 4 03 03-987-012 5 03 03-123-123
Question: I have to process a huge number of tuples made by k integers, each ranging from 1 to Max_k. Each Max can be different. I need to skip the tuples where an element has reached is max value, in that case keeping only the tuple with "1" in the remaining position. The max is enforced by design, so it cannot be that some item is > of its max For example, if the max of the second element of a triple is 4, i need to keep (1,4,1) but skip (1,4,2) , (1,4,3) ... (2,4,1) etc. I am pretty sure I am missing a much faster way to do that. My typical scenario is tuples with 16 to 20 elements, with maxes in the 50-70 mark. What would be the recommended approach ? In Python, as a toy example with hardcoded Maxes (5,4,2), is the following: from itertools import * def filter_logic(y): if y[0]==5: if y[1] > 1 or y[2] >1: return True if y[1]==4: if y[0] > 1 or y[2] >1: return True if y[2]==2: if y[0] > 1 or y[1] >1: return True return False def tuples_all(max_list): my_iterables = [] for limit in max_list: my_iterables.append(range(1, limit+1)) return product(*my_iterables) def tuples_filtered(max_list): return filterfalse(filter_logic, tuples_all(max_list)) max_list = [5,4,2] print("Original list") for res in tuples_all(max_list): print(res) print("After filtering") for fil in tuples_filtered(max_list): print(fil)  Output of the filtered tuples: After filtering (1, 1, 1) (1, 1, 2) (1, 2, 1) (1, 3, 1) (1, 4, 1) (2, 1, 1) (2, 2, 1) (2, 3, 1) (3, 1, 1) (3, 2, 1) (3, 3, 1) (4, 1, 1) (4, 2, 1) (4, 3, 1) (5, 1, 1) 

Answer: Since you commented that order between tuples isn't important, we can simply produce the tuples with max value and then the tuples without max value: from itertools import * def tuples_direct(max_list): n = len(max_list) # Special case if 1 in max_list: yield (1,) * n return # Tuples with a max. for i, m in enumerate(max_list): yield (1,) * i + (m,) + (1,) * (n-i-1) # Tuples without a max. yield from product(*(range(1, m) for m in max_list)) max_list = [5,4,2] for tup in tuples_direct(max_list): print(tup)  Output (Attempt This Online!): (5, 1, 1) (1, 4, 1) (1, 1, 2) (1, 1, 1) (1, 2, 1) (1, 3, 1) (2, 1, 1) (2, 2, 1) (2, 3, 1) (3, 1, 1) (3, 2, 1) (3, 3, 1) (4, 1, 1) (4, 2, 1) (4, 3, 1)
Question: I'm stuck with a Pytorch problem and could use some help: I've got two tensors:  A 3D tensor (shape: i, j, j) with integer values from 0 to n A 1D tensor (shape: n)  I need to create a new tensor that's the same shape as the first one (i, j, j), but where each value is replaced by the corresponding value from the second tensor. Like using the values in the first tensor as indices for the second one. Here's a quick example: import torch # My tensors big_tensor = torch.randint(0, 256, (10, 25, 25)) small_tensor = torch.rand(256) # What I'm trying to do result = magic_function(big_tensor, small_tensor) # How it should work print(big_tensor[0, 0, 0]) # Let's say this outputs 42 print(small_tensor[42]) # This might output 0.7853 print(result[0, 0, 0]) # This should also be 0.7853  I'm looking for a performant way to do this, preferably not using loops, as both tensors can be quite big. Is there an efficient Pytorch operation or method I can use for this? Any help would be awesome - thanks in advance!

Answer: This should work: small_tensor[big_tensor]  Take note that the type of the big_tensor must be long/int.  Edit: In response to the comment of @simon, I wrote a colab notebook that shows how this solution works without the need to perform any other operation.
Question: I'm trying to understand how type variance works with Python protocols and generics. My test cases seem to contradict what I expect regarding invariant, covariant, and contravariant behavior. Here's a minimal example demonstrating the issue: from typing import TypeVar, Protocol # Type variables T = TypeVar('T') T_co = TypeVar('T_co', covariant=True) T_contra = TypeVar('T_contra', contravariant=True) # Class hierarchy class Animal: pass class Dog(Animal): pass # Protocols class Feeder(Protocol[T]): def feed(self, animal: T) -> T: ... class Adopter(Protocol[T_co]): def adopt(self) -> T_co: ... class Walker(Protocol[T_contra]): def walk(self, animal: T_contra) -> None: ... # Implementations class AnimalFeeder: def feed(self, animal: Animal) -> Animal: ... class DogFeeder: def feed(self, animal: Dog) -> Dog: ... class AnimalAdopter: def adopt(self) -> Animal: ... class DogAdopter: def adopt(self) -> Dog: ... class AnimalWalker: def walk(self, animal: Animal) -> None: ... class DogWalker: def walk(self, animal: Dog) -> None: ...  When testing type assignments, some cases behave differently than expected: # Test cases with expected vs actual behavior feeder1: Feeder[Dog] = DogFeeder() # Expected ✅ Actual ✅ (exact match) feeder2: Feeder[Dog] = AnimalFeeder() # Expected ❌ Actual ❌ (invariant) feeder3: Feeder[Animal] = DogFeeder() # Expected ❌ Actual ✅ (Why does this work?) adopter1: Adopter[Dog] = DogAdopter() # Expected ✅ Actual ✅ (exact match) adopter2: Adopter[Dog] = AnimalAdopter() # Expected ❌ Actual ❌ (return type mismatch) adopter3: Adopter[Animal] = DogAdopter() # Expected ✅ Actual ✅ (covariant, correct) walker1: Walker[Dog] = DogWalker() # Expected ✅ Actual ✅ (exact match) walker2: Walker[Dog] = AnimalWalker() # Expected ✅ Actual ❌ (Should work with contravariance?) walker3: Walker[Animal] = DogWalker() # Expected ❌ Actual ✅ (Why does this work?)  Questions:  Why does feeder3 type-check when it seemingly violates invariance? Why does walker2 fail when it should be valid with contravariance? Are these behaviors correct according to Python's type system, or is this a limitation in PyCharm's type checking?  I'm using Python 3.10 and PyCharm 2024.3.1.

Answer:  feeder3 works because Python's structural typing checks method return type covariance (accepting Dog as Animal), but incorrectly ignores parameter contravariance (should reject Dog input for Animal parameter). This is a type checker limitation (PyCharm/mypy may differ).  walker2 fails due to insufficient contravariance support in some type checkers. AnimalWalker (handles supertype Animal) should satisfy Walker[Dog] (needs subtype input) via contravariance, but PyCharm doesn't recognize  Behaviors are partially incorrect - true variance rules (per PEP 544) would reject feeder3 and accept walker2. PyCharm's checker has limitations in enforcing variance for protocols, unlike stricter tools like mypy.
Question: Can someone help me create a layout for plots with the following structure: A CD A/B EF B GH  where A,B are 3D plots (fig.add_subplot(241, projection='3d')) and C,D,E,F,G,H are regular 2D plots. A/B represents a shared space where half of plot A and half of plot B appear.

Answer: You could use a subplot_mosaic: f, grid = plt.subplot_mosaic('''\ ACD ACD AEF BEF BGH BGH''', per_subplot_kw={('A', 'B'): {'projection': '3d'}} ) plt.tight_layout()  Then plot on grid['A']/grid['B']/grid['C']/... Output:  If you need a more flexible, play with the gridspec: f, grid = plt.subplot_mosaic('''\ ACD ACD AEF BEF BGH BGH''', per_subplot_kw={('A', 'B'): {'projection': '3d'}}, gridspec_kw={'width_ratios':[3, 1, 1], 'height_ratios':[3, 1, 1, 1, 1, 1], }) plt.tight_layout()  Output:
Question: I'm working on creating a time series box plot that spans multiple years, and I want the x-axis to display only the month and year for the 1st of each month. I would like to limit the labels to every month, or maybe every couple of months. import matplotlib.pyplot as plt import seaborn as sns import pandas as pd import matplotlib.dates as mdates import random Dates = pd.date_range(start="2020-01-01", end="2020-02-15", freq='D').date data = {'Date': [], 'Values': []} for d in Dates: data['Date'].extend([d] * 10) data['Values'].extend(random.sample(range(1, 101), 10)) df = pd.DataFrame(data) df['Date'] = pd.to_datetime(df['Date']) sns.boxplot(x='Date', y='Values', data=df) plt.xticks(rotation=45) plt.title('Time Series Box Plot') plt.ylabel('Values') plt.xlabel('Date') plt.tight_layout() plt.show()  I tried adding this to the end, but for some reason it makes the year 1970. ax = plt.gca() ax.xaxis.set_major_locator(mdates.MonthLocator(interval=1)) ax.xaxis.set_major_formatter(mdates.DateFormatter('%b %Y')) plt.show()  result

Answer: You were almost there, just need one more instruction: import matplotlib.dates as mdates mdates.set_epoch('2020-01-01T00:00:00')  It is because in Matplotlib the 0 date is the 01/01/1970. This instruction has to be put at the start of your code and if you are in Jupyter you will need to restart your kernel.
Question: It seems we can use the following types for hinting LinkedLists in Python:  Union[ListNode, None] or Union['ListNode', None] Optional[ListNode] Optional['ListNode'] ListNode | None ...  Which of these should we prefer? Feel free to share any other relevant insights as well. Attempt Here we simply want to merge K sorted LinkedLists: from typing import Union, List, Any, Optional import unittest import gc class ListNode: def __init__(self, val: int = 0, next: Union['ListNode', None] = None): self.val = val self.next = next class Solution: def mergeKLists(self, linkedlists: Optional[List[ListNode]]) -> Union[ListNode, None]: if not linkedlists: return None interval = 1 while interval < len(linkedlists): for i in range(0, len(linkedlists) - interval, interval << 1): linkedlists[i] = self.merge_two_linkedlists(linkedlists[i], linkedlists[i + interval]) interval <<= 1 return linkedlists[0] def merge_two_linkedlists(self, l1: Union[ListNode, None], l2: Union[ListNode, None]) -> Union[ListNode, None]: sentinel = curr = ListNode() while l1 and l2: if l1.val <= l2.val: curr.next = l1 l1 = l1.next else: curr.next = l2 l2 = l2.next curr = curr.next if l1: curr.next = l1 else: curr.next = l2 return sentinel.next def main_test_merge_k_linkedlists(test_case_class): suite = unittest.TestLoader().loadTestsFromTestCase(test_case_class) runner = unittest.TextTestRunner(verbosity=2, failfast=1) return runner.run(suite) class TestMergeKLists(unittest.TestCase): def test_merge_k_sorted_linkedlists(self): l1 = ListNode(10) l1.next = ListNode(20) l2 = ListNode(40) l2.next = ListNode(50) l3 = ListNode(80) l3.next = ListNode(150) self.assertEqual(Solution().mergeKLists([l1, l2, l3]), l1) l1 = l2 = l3 = None gc.collect() if __name__ == '__main__': main_test_merge_k_linkedlists(TestMergeKLists) 

Answer: typing.Optional[T] is just shorthand for typing.Union[T, None]. Would always prefer the former to the latter for its succinctness. Union is of course still useful when using it with something else than None. After Python 3.10, a union of types can simply be written with the | operator (Optional[T] becomes T | None). So nowadays it's unnecessary to import/use either Union or Optional. Also, as of Python 3.9, the built-in collection types support the subscripting [] operator, so a generic typing.List[T] can now just be written as list[T]. The reason for sometimes needing to have a type written in quotes, is that if the type does not exist yet (e.g. using the class as a type within its own body), the string format can still be understood by type checkers and it doesn't cause a runtime NameError. This stringification need can be avoided by using the from __future__ import annotations statement at the top of a module. That statement explicitly opts you into PEP 563 – Postponed Evaluation of Annotations. (That PEP has actually been superseded PEP 649 – Deferred Evaluation Of Annotations Using Descriptors, and it's coming "mandatory" in Python 3.14.)
Question: Say I have this data frame: import pandas as pd x = pd.DataFrame([[1, 'step', 'id', 22, 33], [2, 'step', 'id', 55, 66]], columns=['time', 'head_1', 'head_2', 'value_1', 'value_2']) print(x) time head_1 head_2 value_1 value_2 0 1 step id 22 33 1 2 step id 55 66  Then I use pivot table like below print(x.pivot_table(values=['value_1', 'value_2'], columns='time', index=['head_1', 'head_2'])) value_1 value_2 time 1 2 1 2 head_1 head_2 step id 22 55 33 66  However, I really want to have value_1 and value_2 in rows instead of columns like below (a new header as head_3). That is, put value_1 and value_2 in rows and only time as column. How do I do that? time 1 2 head_1 head_2 head_3 step id value_1 22 55 step id value_2 33 66 

Answer: One straightforward way is to melt your dataframe so value_1 and value_2 become labels in single column and then pivot on that. Like: import pandas as pd x = pd.DataFrame( [ [1, 'step', 'id', 22, 33], [2, 'step', 'id', 55, 66] ], columns=['time','head_1', 'head_2', 'value_1', 'value_2'] ) melted = x.melt( id_vars=['time','head_1', 'head_2'], value_vars=['value_1','value_2'], var_name='head_3', value_name='val' ) result = melted.pivot_table( index=['head_1','head_2','head_3'], columns='time', values='val', aggfunc='first' )  You’ll get: time 1 2 head_1 head_2 head_3 step id value_1 22 55 value_2 33 66  This places value_1 and `value_2 in rows as you wanted with only time going across columns