eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have several numpy timedelta values. I want to convert them to the a format that is better to read for humans without losing information. Let's say I have td = np.timedelta64(10800000000001, 'ns'). Then I can only have it in ns because if I convert it to msor higher it will lose information. If I have td = np.timedelta64(1080000000000, 'ns') I can convert it to 3 hours without losing information. What is a good way to do this automatically? I tried it by taking the number of trailing zeros into account: import numpy as np if __name__ == "__main__": td = np.timedelta64(10800000000001, 'ns') number_of_zeros = len(str(td.item())) - len(str(td.item()).rstrip('0')) if number_of_zeros==0: print("[ns]") elif number_of_zeros<7: print("ms") elif number_of_zeros<10: print("s") elif number_of_zeros<12: print("min") else: print("h")  This is probably not a very elegant way to do it (not to mention that it will be wrong when we get to minutes and higher). Any recommendations?

Answer: If I understand the problem statement, you only want to express the time in terms of one unit: the largest that retains precision using only whole numbers. import numpy as np td = np.timedelta64(10800000000000, 'ns') units = ['h', 'm', 's', 'ms', 'ns'] # Convert to each of the desired units deltas = [td.astype(f'timedelta64[{unit}]') for unit in units] # Retain only the ones that are equivalent to the original representation delta = [delta for delta in deltas if delta == td] delta[0] # extract the zeroth # np.timedelta64(3,'h')  It it a little faster to divide the integer representation of the timedelta by the the number of nanoseconds in each unit and check whether the result is a whole number. # Check whether the number of nanoseconds in each unit is a whole number divisible = np.divmod(td.astype(int), ns_per_unit)[1] == 0 # Use the first unit for which this is true unit = units[np.where(divisible)[0][0]] td.astype(f'timedelta64[{unit}]') # perform the conversion  But the first bit of code is probably a little easier to interpret. Depends on what your goals are.
Question: I'm training a custom model using a script in Amazon SageMaker and launching the job with the Python SDK. I want to pass some environment variables (like API keys or config flags) to the training job so they’re accessible inside the script via os.environ. Here’s a simplified version of my code: from sagemaker.estimator import Estimator estimator = Estimator( image_uri='123456789012.dkr.ecr.us-west-2.amazonaws.com/my-custom-image:latest', role=role, instance_count=1, instance_type='ml.g5.xlarge', entry_point='train.py', source_dir='src', environment={ 'MY_API_KEY': 'abcdef123456', 'DEBUG_MODE': 'true' } )  In my training script, I try to read the variable: import os api_key = os.environ.get('MY_API_KEY') print("API Key:", api_key)  Is this the correct way to pass environment variables to a SageMaker training job using the Python SDK? Are there any limitations or best practices I should be aware of, especially for sensitive information like API keys?

Answer: Yes, your approach is correct. Using the environment parameter in the Estimator and accessing variables with os.environ.get() in your script is the standard way to pass environment variables in SageMaker. As @furas pointed out in their comment, os.environ.get() is the common approach in Python. That said, for handling secrets like API keys, it's better to avoid hardcoding them in your code or environment. A more secure approach is to store them in AWS Secrets Manager and fetch them inside your training script at runtime. You can pass the secret's name as an environment variable and retrieve the value securely using boto3: import boto3 import os secret_name = os.environ.get('API_KEY_SECRET_NAME') region = os.environ.get('AWS_REGION', 'us-west-2') client = boto3.client('secretsmanager', region_name=region) secret_value = client.get_secret_value(SecretId=secret_name) api_key = secret_value['SecretString'] print("API Key:", api_key)  This keeps the actual secret out of your environment config and allows for better access control via IAM.
Question: import sympy as sym # Python 3.13 # Used variables in question x = sym.Symbol("x") I1 = sym.Symbol("I₁") L = sym.Symbol("L") pi = sym.Symbol("Π") v2 = sym.Symbol("v2") P = sym.Symbol("P") E = sym.Symbol("E") I = I1 * (1 - x/L) # For the part a of the question one in Assignment 3. question_part = "a" if question_part == "a": func_v = (1 - sym.cos((pi*x)/(2*L))) * v2 moment = P * L second_derivative_v = sym.diff(func_v, x, x) internal_strain_e_func = 0.5 * moment *second_derivative_v / (E * I) internal_strain_e = sym.integrate(internal_strain_e_func, (x, 0, L)) external_e = P * v2 tip_deflection_eq = sym.Eq(internal_strain_e, external_e) tip_deflection = sym.solve(tip_deflection_eq, v2, exclude=[I1, L, pi, P, E], implicit=True) sym.pprint(tip_deflection)  Code above is for virtual displacement method for a beam deflection question. I can technically do it by hand however, I get a lot of questions that requires for me to solve integrals and I do make mistakes when I do it by hand. So in this one I thought I would do it by sympy and its okey up until the solve() method. For some reason I can't get a symbolically written solution for eq. The eq. I am trying to solve is:  Edit: I am sorry everyone v2 is not supposed be a part in function v. The question gave it to me and I took it without thinking about it although I did learn stuff that I didn't know and would probably ask later so not much of a waste of your time. Thanks!

Answer: Don't declare a symbol called pi. You should use SymPy's sympy.pi if you mean the mathematical number pi. This is your equation as stated: In [42]: x, L, v2, E, I1, P, v2 = symbols('x, L, v2, E, I1, P, v2') In [43]: eq = Eq(Integral((P*(L - x)*(pi/(2*L))**2*cos((pi*x)/(2*L)) * v2)/(E*I1*(1 - x/L)), (x, 0, L ...: )), P*v2) In [44]: eq Out[44]: L ⌠ ⎮ 2 ⎛π⋅x⎞ ⎮ π ⋅P⋅v₂⋅(L - x)⋅cos⎜───⎟ ⎮ ⎝2⋅L⎠ ⎮ ──────────────────────── dx = P⋅v₂ ⎮ 2 ⎛ x⎞ ⎮ 4⋅E⋅I₁⋅L ⋅⎜1 - ─⎟ ⎮ ⎝ L⎠ ⌡ 0  We can use doit to evaluate the integral and then solve for v2: In [45]: eq.doit() Out[45]: π⋅P⋅v₂ ────── = P⋅v₂ 2⋅E⋅I₁ In [46]: solve(eq.doit(), v2) Out[46]: [0]  Apparently this is not the answer you were looking for but in general only v2 = 0 can solve the equation for arbitrary values of the parameters. Maybe you wanted to solve for something else rather than v2? This is the equation rearranged: In [47]: eq2 = eq.doit() In [48]: eq3 = Eq(factor(eq2.lhs - eq2.rhs), 0) In [49]: eq3 Out[49]: -P⋅v₂⋅(2⋅E⋅I₁ - π) ─────────────────── = 0 2⋅E⋅I₁  At least one of the factors in the numerator on the lhs must be zero to satisfy the equation. If you want a value of v2 that can satisfy this then in general it has to be v2 = 0. Maybe the equation has not been put together correctly?
Question: Given an index split list T of length M + 1, where the first element is 0 and the last element is N, generate an array D of length N such that D[T[i]:T[i+1]] = i. For example, given T = [0, 2, 5, 7], then return D = [0, 0, 1, 1, 1, 2, 2]. I'm trying to avoid a for loop, but the best I can do is: def expand_split_list(split_list): return np.concatenate( [ np.full(split_list[i + 1] - split_list[i], i) for i in range(len(split_list) - 1) ] )  Is there a built-in function for that purpose?

Answer: You could combine diff, arange, and repeat: n = np.diff(T) out = np.repeat(np.arange(len(n)), n)  As a one-liner (python ≥3.8): out = np.repeat(np.arange(len(n:=np.diff(T))), n)  Another option with assigning ones to an array of zeros, then cumsum: out = np.zeros(T[-1], dtype=int) out[T[1:-1]] = 1 out = np.cumsum(out)  Output: array([0, 0, 1, 1, 1, 2, 2])
Question: this is my product and category model: class Category(models.Model): name = models.CharField(max_length=100) class Product(models.Model): ... category = models.ForeignKey(Category, related_name="products", on_delete=models.CASCADE)  I want a list of all products with priority order. e.g.  categories_ids = [3,5,1,4,2]  now I want data to order like this  [product_with_category_3, product_with_category_3, product_with_category_3, product_with_category_5, product_with_category_1, product_with_category_1, ...] 

Answer: We can determine the priority based on the category, and a Case-When expression: from django.db.models import Case, Value, When category_ids = [3, 5, 1, 4, 2] Product.objects.filter( category_id__in=category_ids ).alias( priority=Case(*[When(category_id=k, then=Value(i)) for i, k in enumerate(category_ids)]) ).order_by('priority')  This will however result in linear search so if the number of categories is large, it is not a good idea.
Question: I am using scipy's multivariate earth mover distance function wasserstein_distance_nd. I did a quick sanity check that confused me: Given that I draw two Gaussian multivariate samples from the same distribution, I should get an earth mover distance that is close to 0. However, I am getting something large (e.g., 12). Why is this happening? I have tested this with the one dimensional case and I also got something similar (here, the distance produced is always positive). Code that I used is given as follows: '''Multi-dimensional''' import numpy as np from scipy.stats import wasserstein_distance_nd mean = np.zeros(100) cov = np.eye(100) size = 100 sample1 = np.random.multivariate_normal(mean, cov, size) sample2 = np.random.multivariate_normal(mean, cov, size) print("EMD", wasserstein_distance_nd(sample1, sample2)) # output: EMD 12.293968193381374 '''single dimension''' import numpy as np from scipy.stats import wasserstein_distance mean = 0 var = 1 size = 100 sample1 = np.random.normal(mean, np.sqrt(var), size) sample2 = np.random.normal(mean, np.sqrt(var), size) dist = wasserstein_distance(sample1, sample2) print("wasserstein_distance", dist) 

Answer: Your multivariate_normal example is in a 100-dimensional space. You can think of the reason you are getting large distances from an intuitive perspective: there is too much variety possible in a 100-dimensional space for the two samples to be very similar. Some more motivation:  In high dimensional spaces, randomly drawn vectors tend to be nearly perpendicular (see e.g. https://math.stackexchange.com/questions/995623/why-are-randomly-drawn-vectors-nearly-perpendicular-in-high-dimensions). Compare this to a one-dimensional space, in which all random vectors lie on the same line.  Think also about fixing the error in each coordinate to a given level and increasing the number of dimensions: the distance increases according to the square root of the number of dimensions. For instance, in one dimension, if $p_1 = 0$ and $p_2 = 1$, the distance between them is only 1. But in 100 dimensions, $p_1 = (0, ..., 0)$ and $p_2 = (1, ..., 1)$ have a distance of 10.   For a rigorous explanation, seek help on Math Stack Exchange. My point, though, is just that intuition about magnitudes of distances from a 1 dimensional case just won't work well in 100 dimensions. If it helps, you can confirm that wasserstein_distance_nd agrees with wasserstein_distance in 1D. import numpy as np from scipy.stats import wasserstein_distance mean = 0 var = 1 size = 100 sample1 = np.random.normal(mean, np.sqrt(var), size) sample2 = np.random.normal(mean, np.sqrt(var), size) ref = wasserstein_distance(sample1, sample2) res = wasserstein_distance_nd(sample1[:, np.newaxis], sample2[:, np.newaxis]) print("wasserstein_distance", res, ref) # wasserstein_distance 0.2109383092226257 0.21093830922262574
Question: I am trying to plot my pandas df data onto a plotly bar chart, but for some reason, instead of putting the numerical values on the chart, its just putting the order onto it. For example, I am trying to also follow along with the docs, but it also fails on the docs' example. The code: import plotly.express as px fig = px.bar(x=["a", "b", "c"], y=[1, 3, 2]) fig.show()  shows a chart of this:  the bar values are the indexes of the array, rather than their actual values. I presume it might be with an issue with my install, but I am not entirely sure. Any help would be appreciated, as I haven't been able to find other solutions elsewhere.

Answer: I tested this out, and this appears to be a bug that only appears when using plotly v6.0.0 in a jupyter notebook. For now, the issue can be worked around by downgrading the version inside the notebook: %pip install plotly==5.23.0 You'll notice that if you try to plot fig = px.bar(x=["a", "b", "c"], y=["1", "3", "2"]) you get the same, undesired behavior so it's possible that when plotly is run from a jupyter notebook and px.bar is called using lists passed to x and y, the y values are being converted to strings under the hood.
Question: I have a dataframe df. >>> import polars as pl >>> >>> >>> df = pl.DataFrame({"col": ["row1", "row2", "row3"]}) >>> df shape: (3, 1) ┌──────┐ │ col │ │ --- │ │ str │ ╞══════╡ │ row1 │ │ row2 │ │ row3 │ └──────┘  Now I want to create a new column new. It should be the first matched pattern in the col. For example, For the pattern 1|2 it should produce the following output. ┌──────┬───────┐ │ col ┆ new │ │ --- ┆ --- │ │ str ┆ str │ ╞══════╪═══════╡ │ row1 ┆ 1 │ │ row2 ┆ 2 │ │ row3 ┆ null │ └──────┴───────┘  I tried using with the expression API, but it's returning boolean values. >>> df.with_columns(new=pl.col('col').str.contains("1|2")) shape: (3, 2) ┌──────┬───────┐ │ col ┆ new │ │ --- ┆ --- │ │ str ┆ bool │ ╞══════╪═══════╡ │ row1 ┆ true │ │ row2 ┆ true │ │ row3 ┆ false │ └──────┴───────┘ 

Answer: You should modify your pattern to have a capturing group (...) and use Expr.str.extract: df.with_columns(new=pl.col('col').str.extract("(1|2)"))  Output: ┌──────┬──────┐ │ col ┆ new │ │ --- ┆ --- │ │ str ┆ str │ ╞══════╪══════╡ │ row1 ┆ 1 │ │ row2 ┆ 2 │ │ row3 ┆ null │ └──────┴──────┘
Question: Given this dictionary and input strings: d = { 'one': '1', 'two': '2', 'three': '3' } s1 = 'String containing <#one|> or <#two|> numbers. <#one|>, <#two|>, <#three|>' s2 = 'Only replace items which are anchored. <#one|> is replaced, but not this one.'  How can I replace each occurrence of an anchored item <# |> using the dictionary d? The above strings should produce the output: String containing 1 or 2 numbers. 1, 2, 3 Only replace items which are anchored. 1 is replaced, but not this one.  Using single pass multi replacement described here comes close to solving this, but doesn't handle the anchors.

Answer: The regular expression pattern should be <#(\w+)\|?>. The \|? part makes the trailing | optional because the quantifier ? (Zero or One) matches zero or one occurrence of the preceding element. I used the re.sub(pattern, repl, string) function for substituting substrings that match the pattern with a replacement function. For each match found, the replacement function will be called, and its return value will be used as the replacement. import re d = { 'one': '1', 'two': '2', 'three': '3' } def replace_items(text): pattern = r"<#(\w+)\|?>" def replacement(match): # This function will be called for each match key = match.group(1) return d.get(key, match.group(0)) # Return value if key exists, otherwise return original match return re.sub(pattern, replacement, text) s1 = 'String containing <#one|> or <#two|> numbers. <#one|>, <#two|>, <#three>' s2 = 'Only replace items which are anchored. <#one|> is replaced, but not this one.' s3 = 'Number <#four> is not in the dictionary' print(replace_items(s1)) print(replace_items(s2)) print(replace_items(s3))  Output String containing 1 or 2 numbers. 1, 2, 3 Only replace items which are anchored. 1 is replaced, but not this one. Number <#four> is not in the dictionary
Question: I'm trying to let people on my project (including myself) migrate to uv while maintaining compatibility with people who still want to use poetry (including some of our builds). Now that poetry 2.0 has been released, and the more standard [project] fields are supported, this should be possible. It's working well in almost all ways, but the main glitch I'm hitting is the set of dev dependencies. uv and the standard seem to want them as [dependency-groups] dev = [ "bandit ~= 1.8.3", ... ]  while poetry seems to want them as [tool.poetry.group.dev.dependencies] bandit = "^1.8.3" ...  . Without including that section, the dev dependencies get left out of the lockfile when doing poetry lock. Anyone have any techniques for getting the two tools to play nicely in this situation?

Answer: Until poetry adds support for PEP-735, you can (ab)use the project.optional-dependencies to achieve something similar:  [project.optional-dependencies] dev = ["bandit ~= 1.8.3"]  Since this is a standard table, it can be used by both poetry and uv:  $ poetry sync --extras dev $ poetry sync --all-extras $ uv sync --extra dev $ uv sync --all--extras  The caveat here is that, as the name implies, these optional dependencies are still part of the project metadata and as such they'll be part of the published metadata on PyPI and users will be able to install them with $tool install foobar[dev].
Question: I am using tornado to host a server that serves as the backend for a client that needs to be running Jython 2.5.2 (thus urllib2). I have a function that downloads files, that up until now only downloaded text files. I need to add non-text files and download them quickly. To download text files in a timely fashion, I concatenate them into a single string and send them as plain text to the client. class DownloadHandler(web.RequestHandler): def get(self): files_as_text = "" for file in os.listdir("files"): files_as_text += file+"---title_split---"+open("files/"+file).read()+"---file_split---" self.write(files_as_text)  Then, on the other side, I can build them back into files. for uptodate_file in uptodate_files.split("---file_split---")\[:-1\]: uptodate_filename, uptodate_file_contents = uptodate_file.split("---title_split---") new_file = open(dir+uptodate_filename, 'wb') new_file.write(uptodate_file_contents).read()) new_file.close()  This worked great for text files, but when I add any other file into the mix, it no longer works. So, I served each file separately and made individual requests using urllib2.urlopen(url+uptodate_filename).read()  This works, but it is really slow. Is there some way to combine the two and send concatenated files that are not text?

Answer: Turns out you can just stick a b in front of each string to convert it to bytes and everything works great! class DownloadHandler(web.RequestHandler): def get(self): files_as_text = b"" for file in os.listdir("files"): files_as_text += file.encode("utf-8")+b"---title_split---"+open("files/"+file, "rb").read()+b"---file_split---" self.write(files_as_text)
Question: I have difficulties setting my environment variables when launching a python script using Slurm. The following python script import petsc4py without error when launched from a shell session : from petsc4py import PETSc print("petsc4py imported successfully")  However when I try to run it using srun with the following sbatch script (first setting the same environment by sourcing the same python env and loading the same modules) : #!/bin/bash #SBATCH --job-name=petsc4py_import_test #SBATCH --ntasks=2 #SBATCH --cpus-per-task=8 #SBATCH --time=00:30:00 #SBATCH --output=output.log #SBATCH --error=error.log #SBATCH --nodes=1 source $HOME/.bash_profile module purge module load python/3.10.0 openmpi gcc openblas/latest slurm source $HOME/my_env/bin/activate srun --export=ALL --mpi=pmix_v3 python3 petsc4py_test.py  It fails with : ImportError: liblapack.so.3: cannot open shared object file: No such file or directory srun: error: node001: tasks 0-1: Exited with exit code 1 

Answer: This could be because liblapack is available as a operating system package on the node where the shell session is run and not available on the node on which the job runs. You might have to load the lapack module if one exists module load python/3.10.0 openmpi gcc openblas/latest lapack/latest slurm
Question: I'm new to polars and encountering a confusing error. I'm trying to take several array columns and zip them into struct columns. When I try to do this with with_columns I encounter the error: ValueError: can only call `.item()` if the dataframe is of shape (1, 1), or if explicit row/col values are provided; frame has shape (4, 2)  Here is code to reproduce this problem: df = pl.DataFrame( { "a": [[1, 2, 3, 4],[1, 2, 3, 4],[1, 2, 3, 4],[1, 2, 3, 4]], "b": [[1, 2, 3, 5],[1, 2, 3, 5],[1, 2, 3, 5],[1, 2, 3, 5]], "c": [[1, 2, 3, 4],[1, 2, 3, 4],[1, 2, 3, 4],[1, 2, 3, 4]], "d": ['a', 'b', 'c', 'd'] } )  df.with_columns([ (df.explode('a', 'b') .select( "a", "b", "d", pl.struct('a', 'b').alias("test_1")) .group_by("d") .agg("test_1")), (df.explode('b', 'c') .select( "c", "b", "d", pl.struct('b', 'c').alias("test_2")) .group_by("d") .agg("test_2")), ] )  With a single struct column (and no list in the method call) this works just as expected and yields the output:  a b c d test_1 list[i64] list[i64] list[i64] str list[struct[2]] [1, 2, … 4] [1, 2, … 5] [1, 2, … 4] "d" [{1,1}, {2,2}, … {4,5}] [1, 2, … 4] [1, 2, … 5] [1, 2, … 4] "b" [{1,1}, {2,2}, … {4,5}] [1, 2, … 4] [1, 2, … 5] [1, 2, … 4] "c" [{1,1}, {2,2}, … {4,5}] [1, 2, … 4] [1, 2, … 5] [1, 2, … 4] "a" [{1,1}, {2,2}, … {4,5}]  However, even putting this single operation into a list in the method call creates this error: df.with_columns([ (df.explode('a', 'b') .select( "a", "b", "d", pl.struct('a', 'b').alias("test_1")) .group_by("d") .agg("test_1")),] )  I'm sure this is some sort of simple error, but I cant' find any information on the cause and solution to this.

Answer: Compute test_1 and test_2 as separate DataFrames. Use join to combine test_1 and test_2 with the original DataFrame. Avoid passing complete DataFrames to with_columns()  import polars as pl df = pl.DataFrame( { "a": [[1, 2, 3, 4], [1, 2, 3, 4], [1, 2, 3, 4], [1, 2, 3, 4]], "b": [[1, 2, 3, 5], [1, 2, 3, 5], [1, 2, 3, 5], [1, 2, 3, 5]], "c": [[1, 2, 3, 4], [1, 2, 3, 4], [1, 2, 3, 4], [1, 2, 3, 4]], "d": ['a', 'b', 'c', 'd'] } ) test_1 = ( df.explode("a", "b") .select( "a", "b", "d", pl.struct("a", "b").alias("test_1") ) .group_by("d") .agg(pl.col("test_1")) ) test_2 = ( df.explode("b", "c") .select( "c", "b", "d", pl.struct("b", "c").alias("test_2") ) .group_by("d") .agg(pl.col("test_2")) ) result = df.join(test_1, on="d").join(test_2, on="d") print(result)  Result is in graph.
Question: I’m trying to create a Tab in a google doc via the python SDK (as in the new(ish) side navigation/organizational tabs, not a tab character). I see how to read and update existing tabs, but don't see any mention of how to create a tab (or any callouts on that being a pending feature). Is it supported and just not well documented yet, or are Tabs new enough that this feature doesn't exist yet? Searched documentation and couldn't find anything related to creating tabs:  https://developers.google.com/docs/api/how-tos/tabs  https://developers.google.com/docs/api/reference/rest/v1/documents/request  

Answer: It is not currently implemented. See this feature request, https://issuetracker.google.com/375867285.