eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am doing some physics simulations using jax, and this involves a function called the Hamiltonian defined as follows: # Constructing the Hamiltonian @partial(jit, static_argnames=['n', 'omega']) def hamiltonian(n: int, omega: float): """Construct the Hamiltonian for the system.""" H = omega * create(n) @ annhilate(n) return H  and then a bigger function def solve_diff(n, omega, kappa, alpha0): that is defined as follows: @partial(jit, static_argnames=['n', 'omega']) def solve_diff(n, omega, kappa, alpha0): # Some functionality that uses kappa and alpha0 H = hamiltonian(n, omega) # returns an expectation value  When I try to compute the gradient of this function using jax.grad n = 16 omega = 1.0 kappa = 0.1 alpha0 = 1.0 # Compute gradients with respect to omega, kappa, and alpha0 grad_population = grad(solve_diff, argnums=(1, 2, 3)) grads = grad_population(n, omega, kappa, alpha0) print(f"Gradient w.r.t. omega: {grads[0]}") print(f"Gradient w.r.t. kappa: {grads[1]}") print(f"Gradient w.r.t. alpha0: {grads[2]}")  it outputs the following error: ValueError: Non-hashable static arguments are not supported. An error occurred while trying to hash an object of type <class 'jax._src.interpreters.ad.JVPTracer'>, Traced<ShapedArray(float32[], weak_type=True)>with<JVPTrace> with primal = 1.0 tangent = Traced<ShapedArray(float32[], weak_type=True)>with<JaxprTrace> with pval = (ShapedArray(float32[], weak_type=True), None) recipe = LambdaBinding(). The error was: TypeError: unhashable type: 'JVPTracer'  Though, running solve_diff(16,1.0,0.1,1.0) on its own works as expected. Now if I remove omega from the list of static variables for both the hamiltonian function and the solve_diff, the grad is output as expected. This is confusing me, because I no longer know what qualifies as static or dynamic variables anymore, from the definition that static variables does not change between function calls, both n and omega are constants and indeed should not change between function calls.

Answer: The fundamental issue is that you cannot differentiate with respect to a static variable, and if you try to do so you will get the error you observed.  This is confusing me, because I no longer know what qualifies as static or dynamic variables anymore, from the definition that static variables does not change between function calls  In JAX, the term "static" does not have to do with whether the variable is changed between function calls. Rather, a static variable is a variable that does not participate in tracing, which is the mechanism used to compute transformations like vmap, grad, jit, etc. When you differentiate with respect to a variable, it is no longer static because it is participating in the autodiff transformation, and trying to treat it as static later in the computation will lead to an error. For a discussion of transformations, tracing, and related concepts, I'd start with JAX Key Concepts: transformations.
Question: I have a dataset with a schema like: df = pl.DataFrame( { "target": [ [1.0, 1.0, 0.0], [1.0, 1.0, 0.1], [1.0, 1.0, 0.2], [1.0, 1.0, 0.8], [1.0, 1.0, 0.9], [1.0, 1.0, 1.0], ], "feature": ["a", "b", "c", "d", "e", "f"], }, schema={ "target": pl.Array(pl.Float32, 3), "feature": pl.String, }, )  If I make a histogram of the target-z values it looks like:  I want to resample the data so its flat along z. I managed to do it in a hacky-many-steps way (also very slow). I was wondering if people could suggest a cleaner (and more efficient) way? What I am doing is:  Find the bin edges of said histogram:  bins = 2 # Use e.g. 100 or larger in reality z = df.select(z=pl.col("target").arr.get(2)) z_min = z.min() z_max = z.max() breaks = np.linspace(z_min, z_max, num=bins+1)   Find how many counts are in the bin with the fewest counts:  counts = ( df.with_columns(bin=pl.col("target").arr.get(2).cut(breaks)) .with_columns(counter=pl.int_range(pl.len()).over("bin")) .group_by("bin") .agg(pl.col("counter").max()) .filter(pl.col("counter") > 0) # <- Nasty way of filtering the (-inf, min] bin .select(pl.col("counter").min()) ).item()   Choose only "count" elements on each bin:  df = ( df.with_columns(bin=pl.col("target").arr.get(2).cut(breaks)) .with_columns(counter=pl.int_range(pl.len()).over("bin")) .filter(pl.col("counter") <= counts) .select("target", "feature") )  This gives me:  Do people have any suggestions?

Answer: I don't think you can avoid those three steps for resampling (although depending on your use case you could try to transform the data instead) You can optimize that code a bit though, import polars as pl import numpy as np # Some random mocked data rng = np.random.default_rng() df = pl.DataFrame({'z': rng.lognormal(size=100_000) - 0.5}).filter(pl.col('z').is_between(0.0, 1.0)) z = pl.col('z') # Create the bins using polars, and only once cuts = df.select(pl.linear_space(z.min(), z.max(), 99, closed='none'))['z'] df = df.with_columns(bin=z.cut(cuts)) # just use len() instead of range+max() counts = ( df .group_by("bin") .len() .select(pl.col("len").min()) ).item() # take the head of each group or sample result = ( df .group_by('bin') # .head(counts) # You can just use this instead of .map_groups(...sample(counts)), # and head() is closer to what you had in the original, but # taking only the head() may bias the data if the order is not random .map_groups(lambda df: df.sample(counts)) .drop('bin') ) print(result)
Question: I am trying to get user inputs to create a file where users can store website, username, and password in table format whenever users hit a button. I made the function below, and it looks okay to me. However, when a user enters the second and third entries, the data frame structure is broken. Any idea why it happens? You may see the print result each time adding a row to my data. Code: from tkinter import * import pandas as pd import os def save_password(): website_name = input("Website: ") username = input("Username: ") password = input("Password: ") # password_details = f"website: {website_name};username: {username};password: {password}" input_entries_dict = {"Website": [website_name], "Username/Email": [username], "Password": [password]} input_entries_df = pd.DataFrame(input_entries_dict) if not os.path.isfile("MyPassword_test.txt"): input_entries_df.to_csv("MyPassword_test.txt", index=False) print(input_entries_df) else: data = pd.read_csv("MyPassword_test.txt") data = data._append(input_entries_df, ignore_index=True, sort=True) print(data) data.to_csv("MyPassword_test.txt", sep=";", index=False) save_password()  Outputs for each time: First entry: ALL FINE Website Username/Email Password 0 d32d23 f7324f2 f3223f2 Second Entry: Column names are shifted Password Username/Email Website 0 f3223f2 f7324f2 d32d23 1 ddwefddsfds5 32fwefw5 48sfd4s Third Entry:Colum of "Password;Username/Email;Website" created! Password Password;Username/Email;Website Username/Email Website 0 NaN f3223f2;f7324f2;d32d23 NaN NaN 1 NaN ddwefddsfds5;32fwefw5;48sfd4s NaN NaN 2 154152 NaN f32f23f23 2f23f2332 

Answer: The confusion is caused by writing the .CSV with a ; separator but ignoring this on reading. Use:  else: data = pd.read_csv("MyPassword_test.txt") data = pd.concat([data, input_entries_df], ignore_index=True) print(data) data.to_csv("MyPassword_test.txt", index=False)
Question: I want to refactor a big part of my code into a generic descriptor for read only attribute access. The following is an example of property based implementation class A: def __init__(self, n): self._n = n self._top = -1 @property def n(self): return self._n @property def top(self): return self._top def increase(self): self._top += 1  you see I can initialize my A class and increase self._top but not to let user set a.top by omitting property setter method a = A(7) a.top Out[25]: -1 a.increase() a.top Out[27]: 0  if I do a.top = 4, it will give me an error AttributeError: property 'top' of 'A' object has no setter  which is expected. Now, I want to refactor this logic into a descriptor class ReadOnly: def __init__(self): self._name = None def __set_name__(self, owner, name): self._name = name def __get__(self, instance, owner): if instance is None: return self return instance.__dict__[self._name] def __set__(self, instance, value): raise AttributeError("Can't set attribute") def __delete__(self, instance): raise AttributeError("Can't delete attribute") class A: n = ReadOnly() top = ReadOnly() def __init__(self, n): self.n = n self.top = -1 def increase(self): self.top += 1  Well, this doesn't work. I can't even initialize the class A anymore, cause in __init__ it will set n and top immediately and prevent my from initialize. How to write this logic from property into descriptor? P.S. Thank @chepner for this solution. This is what I'm looking for. I made it work. One last thing, if I have a attribute is a list say class Stack: S = ReadOnly() n = ReadOnly() top = ReadOnly() def __init__(self, n): self._S = [None] * n self._n = n self._top = -1 # python offset 1  Now I can't change self.top anymore >>> s = Stack(4) >>> s.S [None, None, None, None]  Nor I can change s >>> s.S = [1, 3] # not allowed anymore. Great! But I can still change an element in the list >>> s.S[3] = 3 [None, None, None, 3]  How can I prevent list element changes?

Answer: Instead of disallowing all modifications, simply check if the attribute exists on instance before creating it. If it already exists, raise the AttributeError. Otherwise, let the attribute be created. def __set__(self, instance, value): if self._name in instance.__dict__: raise AttributeError("Can't set attribute") instance.__dict__[self._name] = value  Also, if I remember correctly, -- Update: I did not. If the class attribute has __get__ and __set__, it takes priority over an instance attribute of the same name. Here is a good reference. However, I would still give the instance attribute a different name for clarity. --you need to use a different name for the underlying private attribute so that you don't shadow the descriptor. def __set_name__(self, owner, name): self._name = "_" + name
Question: What is the best way to create a new file in Python, moving if needed an existing file with the same name to a different path? While you could do if os.path.exists(name_of_file): os.move(name_of_file, backup_name) f = open(name_of_file, "w")  that has TOCTOU issues (e.g. multiple processes could try and create the file at the same time). Can I avoid those issues using only the standard library (preferably), or is there a package which handles this. You can assume a POSIX file system.

Answer: In a loop: Open the file exclusively (open(..., "x") or O_CREAT|O_EXCL). If that fails with FileExistsError (EEXIST), then atomically os.rename the existing file to something else. Try again. If that renaming fails with anything other than FileExistsError (ENOENT, meaning someone else removed or renamed the offending file before you did), break the loop and fail. (It's not clear to me how your competing processes can know it is OK to move the existing file out of the way or not, but presumably you've knowledge of your use case that I do not.)
Question: I have a list of integers with duplicates and I need to sort it by the number of these duplicates. For example: input: n = [2, 4, 1, 2] output: n = [4, 1, 2, 2]  I wrote some code and noticed, that sort() does not change the list. But if I try to use sorted() with the same key argument, then it works just fine. What is the reason behind this? My code: nums = [2, 4, 1, 2] nums.sort(key = lambda x: nums.count(x)) print(nums)  Can it be connected to sort() method using in-place algorithm?

Answer: Unlike sorted, the list.sort method sorts a list in-place, during which time the list is in an interim state where there is no integrity to its internal data structure for the other methods to read from. Since a key function for the sort method is called during a sort, your calling the count method of the same list in the key function does not actually return the count of a given item as you expect, which you can see with a wrapper function: def count(x): n = nums.count(x) print(n) return n nums = [2, 4, 1, 2] nums.sort(key=count)  the above outputs: 0 0 0 0  which explains why the list remains in the same order since the key for each item turns out to be the same value of 0 and since the sorting algorithm used is a stable one. The documentation of list.sort also notes:  CPython implementation detail: While a list is being sorted, the effect of attempting to mutate, or even inspect, the list is undefined. The C implementation of Python makes the list appear empty for the duration, and raises ValueError if it can detect that the list has been mutated during a sort.  So CPython's implementation makes the list appear empty during a sort, which explains why list.count returns 0 when called from a key function.
Question: I am trying to understand the way validation works in pydantic. I create a class and three objects: import pydantic class TestClass(pydantic.BaseModel): id: int = 'text' name: str obj0 = TestClass(id=1, name="test") obj1 = TestClass(name="test") obj2 = TestClass.model_construct(id=2)  One may see that first object is totally valid, second semivalid, and third one definitive not, that's why I omit the validation using model_construct(). Now I want to validate the objects, as pydantic suggests: vobj0= TestClass.model_validate(obj0, strict=True) vobj1= TestClass.model_validate(obj1, strict=True) vobj2= TestClass.model_validate(obj2, strict=True)  The problem is that NONE of the lines rises ValidationError! I found a clumsy way around it, namely transforming objects to dicts: vobj0= TestClass.model_validate(obj0.model_dump(), strict=True) vobj1= TestClass.model_validate(obj1.model_dump(), strict=True) vobj2= TestClass.model_validate(obj2.model_dump(), strict=True)  This code does raise the ValidationError in the second and third lines. I cannot understand why does it behave in such a funny way? I expected something like obj.validate_me(), but can't find anything like this. So, my question: if I create an object without validation (with model_construct()), what is the right way to do the validation afterwards?

Answer: By default Pydantic does not re-validate instances, because it assumes they have been validated. This is configurable, the docs is here In your test program, all you need is to update the TestClass definition: class TestClass(pydantic.BaseModel, revalidate_instances='always'): ...  (There is also another related option you might find interesting: validate_assignment)  Field defaults are also not validated by default. (See the validate_default parameter in Field). Otherwise you would not be able to write: id: int = 'text'
Question: I would like to define a loss function for the CVXPy optimization that minimizes differences from the reference grouped target: import cvxpy as cp import pandas as pd # toy example for demonstration purpose target = pd.DataFrame(data={'a': ['X', 'X', 'Y', 'Z', 'Z'], 'b': [1]*5}) w = cp.Variable(target.shape[0]) beta = cp.Variable(target.shape[0]) def loss_func(w, beta): x = pd.DataFrame(data={'a': target['a'], 'b': w @ beta}).groupby('a')['b'].sum() y = target.groupby('a')['b'].sum() return cp.norm2(x - y)**2 # <<<<<<<<<<<<<< ValueError: setting an array element with a sequence.  but this gives me the following error ValueError: setting an array element with a sequence. What would be the way to cover this use-case using CVXPy?

Answer: Based on my understanding, your code calculates the sum of weighted values (w @ beta) grouped by the unique values in column "a". However, since Pandas cannot handle CVXPY variables, this approach results in errors. The hstack method, on the other hand, uses native CVXPY functions like cp.sum() and cp.hstack(), making it fully compatible and error-free while giving the same result. Therefore, it’s better to use the hstack approach.‍‍‍ for example: import cvxpy as cp import pandas as pd # toy example for demonstration purpose target = pd.DataFrame(data={"a": ["X", "X", "Y", "Z", "Z"], "b": [1] * 5}) w = cp.Variable(target.shape[0]) beta = cp.Variable(target.shape[0])  def loss_func2(w, beta): x = cp.hstack([ cp.sum(w[target["a"] == group] * beta[target["a"] == group]) for group in target["a"].unique() ]) y = target.groupby("a")["b"].sum().values return cp.norm2(x - y) ** 2
Question: I have the array import numpy as np a1 = [["a1", "a2"], ["a3", "a4"], ["a5", "a6"], ["a7", "a8"]] b1 = [["b1", "b2"], ["b3", "b4"], ["b5", "b6"], ["b7","b8"]] c1 = [["c1", "c2"], ["c3", "c4"], ["c5", "c6"], ["c7","c8"]] arr = np.array([a1, b1, c1]) #arr.shape #(3, 4, 2)  Which I want to reshape to a 2D array: ["a1","b1","c1"], ["a2","b2","c2"], ..., ["a8","b8","c8"]  I've tried different things like: # arr.reshape((8,3)) # array([['a1', 'a2', 'a3'], # ['a4', 'a5', 'a6'], # ['a7', 'a8', 'b1'], # ['b2', 'b3', 'b4'], # ['b5', 'b6', 'b7'], # ['b8', 'c1', 'c2'], # ['c3', 'c4', 'c5'], # ['c6', 'c7', 'c8']]) #arr.T.reshape(8,3) # array([['a1', 'b1', 'c1'], # ['a3', 'b3', 'c3'], # ['a5', 'b5', 'c5'], # ['a7', 'b7', 'c7'], # ['a2', 'b2', 'c2'], # ['a4', 'b4', 'c4'], # ['a6', 'b6', 'c6'], # ['a8', 'b8', 'c8']] # arr.ravel().reshape(8,3) # array([['a1', 'a2', 'a3'], # ['a4', 'a5', 'a6'], # ['a7', 'a8', 'b1'], # ['b2', 'b3', 'b4'], # ['b5', 'b6', 'b7'], # ['b8', 'c1', 'c2'], # ['c3', 'c4', 'c5'], # ['c6', 'c7', 'c8']]) 

Answer: What you want is to stack the arrays such that the final 2D shape is (8, 3), where each row contains the same index from each original array. So the key is to use np.array(arr).transpose(1, 2, 0).reshape(-1, 3). Code Example: import numpy as np a1 = [["a1", "a2"], ["a3", "a4"], ["a5", "a6"], ["a7", "a8"]] b1 = [["b1", "b2"], ["b3", "b4"], ["b5", "b6"], ["b7","b8"]] c1 = [["c1", "c2"], ["c3", "c4"], ["c5", "c6"], ["c7","c8"]] arr = np.array([a1, b1, c1]) # shape: (3, 4, 2) result = arr.transpose(1, 2, 0).reshape(-1, 3) print(result)
Question: Is there a clean way to store the values comparison result as dictionary key, naming >, =, and < (not the str format, but the state of "greater" for example). Curious if this could be a way to replace wordy if a > b: print(f'{a} > {b}') elif a == b: print(f'{a} = {b}') else: print(f'{a} < {b}')  with a dict d = { >: '{a} > {b}', # > is something I don't know how to use, not str('>') =: '{a} = {b}', <: '{a} < {b}', }  Then I can use it like compare_result = a > b # I know this returns boolean; I need a way to return a comparison result in one of {>, =, <} print(d[compare_result].format(a, b))  something like that.

Answer: The operator module contains actual function objects for greater-than, less-than, equal, etc. import operator d = { operator.gt: '{a} > {b}', operator.eq: '{a} = {b}', operator.lt: '{a} < {b}', }
Question: I’m working on macOS Sequoia 15.4, using VS Code and Jupyter Notebooks with Conda environments. Everything worked fine until yesterday. Now, when I try to run any of my existing environments, importing NumPy or other scientific libraries results in the following ImportError related to libgfortran.5.dylib. ImportError: dlopen(..._multiarray_umath.cpython-39-darwin.so): Library not loaded: @rpath/libgfortran.5.dylib Referenced from: /Users/.../miniconda3/envs/PARESIS/lib/libopenblasp-r0.3.21.dylib Reason: tried: '/Users/.../libgfortran.5.dylib' (duplicate LC_RPATH '@loader_path'), '/usr/local/lib/libgfortran.5.dylib' (no such file), '/usr/lib/libgfortran.5.dylib' (no such file, not in dyld cache)  This ends with: ImportError: Error importing numpy: you should not try to import numpy from its source directory; please exit the numpy source tree, and relaunch your python interpreter from there.  This happens only in older environments. If I create a new Conda environment, everything works fine. I tried reinstalling NumPy using conda install numpy --force-reinstall  , and also explicitly installing libgfortran=5 in the affected environment. I also updated Conda, restarted my system, and relaunched VS Code multiple times. I expected NumPy to be reinstalled correctly and the missing library issue to be resolved, but the error persists. I was hoping to avoid recreating all my environments from scratch since I have a lot of dependencies already configured.

Answer: As mentioned previously, this is similar to this question. MacOS Sequoia 15.4.1 update raises an error for duplicate R paths, which triggers the error you mentioned for typically 'old' environments. If you don't want to re-create an environment, you can try to install libgfortran5 package to its version which avoids the error (>=14), e.g. run this command with your environment activated: conda install "libgfortran5>=14"  It should fix the error without having to reinstall everything, assuming the version is compatible with other dependancies and that it's the only one causing the error.
Question: When streaming with OpenAI Assistants openai.beta.threads.messages.create( thread_id=thread_id, role="user", content=payload.question ) run = openai.beta.threads.runs.create( thread_id=thread_id, assistant_id=assistant_id, stream=True, tool_choice={"type": "file_search"}, ) streamed_text = "" for event in run: if event.event == "thread.message.delta": delta_content = event.data.delta.content if delta_content and delta_content[0].type == "text": text_fragment = delta_content[0].text.value streamed_text += text_fragment yield {"data": text_fragment} if event.event == "thread.run.completed": break  the citations are coming in the formats like 【4:2†source】 or citeturnXfileY   How to fix it?

Answer: The approach I've used was to get the final message after streaming messages = openai.beta.threads.messages.list(thread_id=thread_id)  and then apply the following regex def replace_placeholder(match): nonlocal citation_index citation_index += 1 return f"[{citation_index}]" pattern = r"(citeturn\d+file\d+|【\d+:\d+†.*?】)" citation_index = 0 assistant_reply_cleaned = re.sub(pattern, replace_placeholder, raw_text)  to replace the placeholders (like 【4:2†source】 or citeturnXfileY) with [1], [2], etc
Question: Executing vscode.executeDocumentSymbolProvider  gives me all symbols in the file. Can I somehow get name of the function that the cursor is currently in?

Answer: If you only want the top-most outer function that the cursor is in, then the answer is fairly straightforward. If you wanted to consider multiple nested functions then it get much trickier. Scan through the symbols. finding the symbol range that contains the cursor: const symbols = await vscode.commands.executeCommand('vscode.executeDocumentSymbolProvider', document.uri); const targetSymbol = Object.values(symbols).find(childSymbol => { return childSymbol.kind === vscode.SymbolKind.Function) && childSymbol.range.contains(selection.active); });  The childSymbol's will be all the top-level symbols in your document.