eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Consider the following MWE: import sympy as sp a,b = sp.symbols("a b", positive=True, real=True) t = sp.symbols("t", real=True) s = sp.symbols("s") T = 1/(1+s*a)/(1+s*b) y = sp.inverse_laplace_transform(T,s,t) tmax = sp.solve(sp.diff(y,t),t)[0] ymax = y.subs(t,tmax) display(ymax.simplify())  This code gives the following output:  How can I force Sympy to simplify this expression? Both the fraction and the sum can be decomposed easily, leaving a term in the form of exp(log(x)), which can be simplified to x. Edit: Here is one example how the simplification of one exp() term could look like (the first one in this case): 

Answer: Notice the denominators in the arguments of the exponentials (which are different than you posted, but that is what I get): >>> from sympy import * ...your code... >>> e=ymax.atoms(exp);e [exp(-log((a/b)**(a*b))/(a**2 - a*b)), exp(-log((a/b)**(a*b))/(a*b - b**2))]  If you assure SymPy that those are non-zero (and real), all can be well: >>> p,q = var('p q',nonzero=True) # nonzero=True implies real >>> reps = {a**2-a*b:p,a*b-b**2:q}; _reps = {v:k for k,v in reps.items()} >>> [i.subs(reps).simplify().subs(_reps) for i in e] [(b/a)**(a*b/(a**2 - a*b)), (b/a)**(a*b/(a*b - b**2))]  Is that what you were hoping would happen to them? If so you can write a handler to be used with replace to target out the exp and make the denominator nonzero: def handler(x): assert isinstance(x, exp) e = x.args[0] n,d = e.as_numer_denom() if d.is_nonzero: return x.simplify() nz=Dummy(nonzero=True) return exp(n/nz).simplify().subs(nz,d) >>> ymax.replace(lambda x:isinstance(x,exp), handler) a*(b/a)**(a*b/(a**2 - a*b))*Heaviside(log((a/b)**(a*b))/(a - b))/(a**2 - a*b) - b*(b/a)**(a*b/(a*b - b**2))*Heaviside(log((a/b)**(a*b))/(a - b))/(a*b - b**2)
Question: I switched from NumPy arrays to Pandas DataFrames (dfs) many years ago because the latter has column names, which  makes programming easier; is robust in order changes when reading data from a .json or .csv file.  From time to time, I need the last row ([-1]) of some column col of some df1, and combine it with the last row of the same column col of another df2. I know the name of the column, not their position/order (I could know, but it might change, and I want to have a code that is robust against changers in the order of columns). So what I have been doing for years in a number of Python scripts is something that looks like import numpy as np import pandas as pd # In reality, these are read from json files - the order # of the columns may change, their names may not: df1 = pd.DataFrame(np.random.random((2,3)), columns=['col2','col3','col1']) df2 = pd.DataFrame(np.random.random((4,3)), columns=['col1','col3','col2']) df1.col2.iloc[-1] = df2.col2.iloc[-1]  but since some time my mailbox gets flooded with cron jobs going wrong, telling me that  You are setting values through chained assignment. Currently this works in certain cases, but when using Copy-on-Write (which will become the default behaviour in pandas 3.0) this will never work to update the original DataFrame or Series, because the intermediate object on which we are setting values will behave as a copy. A typical example is when you are setting values in a column of a DataFrame, like: df["col"][row_indexer] = value  Use df.loc[row_indexer, "col"] = values instead, to perform the assignment in a single step and ensure this keeps updating the original df. See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy df1.col2.iloc[-1] = df2.col2.iloc[-1]   Of course, this error message is incorrect, and replacing the last line in my example with either of df1.loc[-1, 'col2'] = df2.loc[-1, 'col2'] # KeyError: -1 df1.iloc[-1, 'col2'] = df2.iloc[-1, 'col2'] # ValueError (can't handle 'col2')  does not work either, since .iloc[] cannot handle column names and .loc[] cannot handle relative numbers. How can I handle the last (or any other relative number) row and a column with given name of a Pandas DataFrame?

Answer: You can try to use the following snippet. df1.loc[df1.index[-1], 'col1'] = df2.loc[df2.index[-1], 'col1']  On my machine with pandas version 2.2.3, it gives no warnings.
Question: I have source and link paths. I'm trying to create a symlink, but must be misunderstanding how its used. Let's say source = '/var/source/things/' link = '/var/link/'  When I use os.symlink(source, link) Initially I received an error FileNotFoundError: [Errno 129] EDC5129I No such file or directory.: '/var/source/things/' -> '/var/link/'  Ok, so I'll put in something to create the directory if it doesn't exist. if not os.path.exists(link): os.makedirs(link)  Re-run and now receive: FileExistsError: [Errno 117] EDC5117I File exists.: '/var/source/things/' -> '/var/link/'  So if the directory doesn't exist it fails and if the directory does exist it also fails? We have a bash script that uses ln -sf $source/* $link  Which creates symlinks for all folders within 'source' and was hoping python would be just as easy. But as I mentioned before, I'm misunderstanding something here.

Answer: Can you remove "/" at the end under link and try once source = '/var/source/things/' link = '/var/link'  apart from that, I don't see any issues in your code.
Question: I was trying to make a 3D visualization of the joint probability mass function with the following code: import math import numpy as np import matplotlib.pyplot as plt def f(x, y): if(1 <= x + y and x + y <= 4): return (math.comb(3, x) * math.comb(2, y) * math.comb(3, 4 - x - y)) / math.comb(8, 4) else: return 0.0 x_domain = np.array([0, 1, 2, 3]) y_domain = np.array([0, 1, 2]) X, Y = np.meshgrid(x_domain, y_domain) X = np.ravel(X) Y = np.ravel(Y) Z = np.zeros_like(X, dtype=float) for i in range(len(X)): Z[i] = f(X[i], Y[i]) fig = plt.figure(figsize=(8, 4)) ax = fig.add_subplot(1, 1, 1, projection="3d") ax.scatter(X, Y, Z) # plots the induvidual points for i in range(len(X)): # draws lines from xy plane up (x,y,z) ax.plot([X[i], X[i]], [Y[i], Y[i]], [0, Z[i]], color="r") ax.set_xticks(x_domain) ax.set_yticks(y_domain) plt.show()  Which gave the following result:  As you can see, the stems of lines from the xy-plane do not align with the integer coordinates. I have searched through the matplotlib documentation and cant find the cause for this (unless I have overlooked something). Does anyone know how to fix the alignment issue? EDIT It seems stack overflow could not render the mathjax equation, so I removed it.

Answer: Actually, they DO align with the integer grid, as you will see by rotating your plot a bit. It's just that the z axis doesn't naturally start from 0, which is where the end of your whatever-they-are are. Add the line ax.set_zlim( 0.0, 0.25 )  just before plt.show() and you will be good to go.
Question: I have three pyspark dataframes in Databricks: raw_old, raw_new, and master_df. These are placeholders to work out the logic on a smaller scale (actual tables contain billions of rows of data). There is a column in all three called label. I want to calculate the number of labels that appear in:  raw_old and raw_new (the answer is 3: A789, B456, D123) raw_new and master_df (the answer is 2: C456, D123) raw_old and master_df (the answer is 4: A654, B987, C987, D123) raw_old, raw_new, and master_df (the answer is 1: D123)  The three tables are below. How do I calculate the above bullet points? raw_old +---+-----+ | id|label| +---+-----+ | 1| A987| | 2| A654| | 3| A789| | 4| B321| | 5| B456| | 6| B987| | 7| C321| | 8| C654| | 9| C987| | 10| D123| +---+-----+  raw_new +---+-----+ | id|label| +---+-----+ | 1| A123| | 2| A456| | 3| A789| | 4| B123| | 5| B456| | 6| B789| | 7| C123| | 8| C456| | 9| C789| | 10| D123| +---+-----+  master_df +---+-----+ | id|label| +---+-----+ | 1| A999| | 2| A654| | 3| A000| | 4| B111| | 5| B000| | 6| B987| | 7| C999| | 8| C456| | 9| C987| | 10| D123| +---+-----+ 

Answer: You should use an inner join to get the elements in common between the datasets joined_data = raw_old.join( raw_new, on=raw_old["label"] == raw_new["label"], how="inner" )  and then you can collect the result back to Python, keeping all the heavy work in Spark print(joined_data.count())  When joining 3 dataframes, you can do the first 2 and join the resulted dataframe with the third one.
Question: I have a loss function that needs to be minimized def loss(x: np.ndarray[float]) -> float  My problem has nDim=10 dimensions. Loss function works for 1D arrays of shape (nDim,), and with 2D arrays of shape (nSample, nDim) for an arbitrary number of samples. Because of the nature of the implementation of the loss function (numpy), it is significantly faster to make a single call to the loss function with several samples packed into 2D argument than to make several 1D calls. The minimizer I am currently running is sol = scipy.optimize.basinhopping(loss, x0, minimizer_kwargs={"method": "SLSQP"})  It does the job, but is too slow. As of current, the minimizer is making single 1D calls to the loss function. Based on observing the sample points, it seems, SLSQP is performing numerical differentiation, thus sampling 11 points for each 1 sample to calculate the gradient. Theoretically, it should be possible to implement this minimizer with vectorized function calls, requesting all 11 sample points from the loss function simultaneously. I was hoping that there would be a vectorize flag for SLSQP, but it does not seem to be the case, please correct me if I am wrong. Note also that the loss function is far too complicated for analytic calculation of derivatives, so explicit Jacobian not an option. Question: Does Scipy or any other minimization library for python support a global optimization strategies (such as basinhopping) with vectorized loss function and no available Jacobian?

Answer: differential_evolution is a global optimizer that does not require gradients. It has a vectorized keyword to enable many function evaluations in a single call. Alternatively, you could write a function that takes the Jacobian with scipy.differentiate.jacobian, which calls the function at all required points at once, and pass that as the Jacobian callable. However, it is designed for accuracy, not speed, so you should probably set coarse tolerances and low order.
Question: Given an array that looks like this: values [0, 0, 1, 2, 3, 4, 5, 0, 1, 2, 3, 4, 5, 6, 0, 0, 1, 2, 3] index 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18  If I search for index 3, I want to get the indexes of the start and end indexes for that counter, before it is reset again, which is 2 - 6. And for index 10, I want to get 8 - 13. And for index 16, I want to get 16 - 18. How can I achieve this in numpy?

Answer: You can get the start/end coordinates of the non-null stretches with something like: idx = np.nonzero(values == 0)[0] start = idx+1 end = np.r_[idx[1:]-1, len(values)-1] m = start<end indices = np.c_[start, end][m]  indices: array([[ 2, 6], [ 8, 13], [16, 18]])  Then get the position with searchsorted (assuming you only pass non-zeros indices, else you need an additional check (e.g. is values[position] != 0) and explanation of what should be the output): indices[np.searchsorted(indices[:, 1], 2)] # [ 2, 6] indices[np.searchsorted(indices[:, 1], 10)] # [ 8, 13] indices[np.searchsorted(indices[:, 1], 16)] # [16, 18]  And you can get multiple targets at once: target = [2, 6, 10, 16] indices[np.searchsorted(indices[:, 1], target)] array([[ 2, 6], [ 2, 6], [ 8, 13], [16, 18]])  And if you have indices of zero-values you could mask them in the output: target = [1, 2, 6, 7, 10, 16] out = np.ma.masked_array(indices[np.searchsorted(indices[:, 1], target)], np.broadcast_to(values[target, None]==0, (len(target), 2)) ) [[-- --] [ 2 6] [ 2 6] [-- --] [ 8 13] [16 18]]  Used input: values = np.array([0, 0, 1, 2, 3, 4, 5, 0, 1, 2, 3, 4, 5, 6, 0, 0, 1, 2, 3])
Question: I am trying to write Pandas code that would allow me to sample DataFrame using a normal distribution. The most convinient way is to use random_state parameter of the sample method to draw random samples, but somehow employ numpy.random.Generator.normal to draw random samples using a normal (Gaussian) distribution. import pandas as pd import numpy as np import random # Generate a list of unique random numbers temp = random.sample(range(1, 101), 100) df = pd.DataFrame({'temperature': temp}) # Sample normal rng = np.random.default_rng() triangle_df.sample(n=10, random_state=rng.normal())  This obviously doesn't work. There is an issue with random_state=rng.normal().

Answer: Passing a Generator to sample just changes the way the generator is initialized, it won't change the distribution that is used. Random sampling is uniform (choice is used internally [source]) and you can't change that directly with the random_state parameter. Also note that normal sampling doesn't really make sense for discrete values (like the rows of a DataFrame). Now let's assume that you want to sample the rows of your DataFrame in a non-uniform way (for example with weights that follow a normal distribution) you could use the weights parameter to pass custom weights for each row. Here is an example with normal weights (although I'm not sure if this makes much sense): rng = np.random.default_rng() weights = abs(rng.normal(size=len(df))) sampled = df.sample(n=10000, replace=True, weights=weights)  Another example based on this Q/A. Here we'll give higher probabilities to the rows from the middle of the DataFrame: from scipy.stats import norm N = len(df) weights = norm.pdf(np.arange(N)-N//2, scale=5) df.sample(n=10, weights=weights).sort_index()  Output (mostly rows around 50):  temperature 43 94 44 50 47 80 48 99 50 63 51 52 52 1 53 20 54 41 63 3  Probabilities of sampling with a bias for the center (and sampled points):
Question: The following code to converts a one video from mp4 to avi using ffmpeg ffmpeg.exe -i "kingman.mp4" -c copy "kingman.avi" I need to convert multiple videos in a specific path. "outputvideo" with the same video name This is the my code that needs to be modified. from pathlib import Path import subprocess from glob import glob all_files = glob('./video/*.mp4') for filename in all_files: videofile = Path(filename) outputfile = Path(r"./outputvideo/") codec = "copy" ffmpeg_path = (r"ffmpeg.exe") outputfile = outputfile / f"{videofile.stem}" outputfile.mkdir(parents=True, exist_ok=True) command = [ f"{ffmpeg_path}", "-i", f"{videofile}", "-c", f"{codec}", "{outputfile}",] process = subprocess.Popen( command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, bufsize=1, universal_newlines=True) process.communicate() 

Answer: Use ffmpeg.exe if you are using windows, else just ffmpeg will work in case of linux from pathlib import Path import subprocess import os from glob import glob input_folder = "video" output_folder = "outputvideo" abs_path = os.getcwd() all_files = glob(f'{abs_path}/{input_folder}/*.mp4') output_dir = os.path.join(abs_path , output_folder) out_ext = "avi" os.makedirs(output_dir , exist_ok=True) input_dir = os.path.join(abs_path, input_folder) for filepath in all_files: stem = os.path.splitext(os.path.basename(filepath))[0] videofile = Path(filepath) codec = "copy" ffmpeg_path = (r"ffmpeg") outputfile = f"{output_dir}/{stem}.{out_ext}" command = [ f"{ffmpeg_path}", "-i", f"{videofile}", "-c", f"{codec}", f"{outputfile}",] process = subprocess.Popen( command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, bufsize=1, universal_newlines=True) stdout, stderr = process.communicate() # Print output for debugging print(f"Processed: {videofile} -> {outputfile}") if stderr: print(f"FFmpeg Error: {stderr}")
Question: I am trying to solve a mathematical problem related to matrices using numpy as shown below:  I am really finding it hard to represent this kind matrix structure using numpy. I really donot want to type these values because I want to understand how this kind of structures are represented using python. Consider the empty places as zero. Thank you for great help.

Answer: The matrix has to be decomposed into several parts. First, the middle region forms a block diagonal matrix, where each block is a 4x4 Toeplitz matrix. # makes a shifted diagonal matrix def E(n, v, k): return v * np.eye(n, k=k) def toeplitz_block(n, v_list, k_list): return sum(E(n, v, k) for v, k in zip(v_list, k_list))  Then we can do the following: n = 4 # size of block m = 3 # how many blocks # Make block diagonal matrix A A = np.zeros((m, n, m, n)) u, v = np.diag_indices(m) A[u, :, v, :] = toeplitz_block(n, [-1, 3, -1], [-1, 0, 1]) A = A.reshape(m * n, m * n) print(A.astype(int))  # Output: [[ 3 -1 0 0 0 0 0 0 0 0 0 0] [-1 3 -1 0 0 0 0 0 0 0 0 0] [ 0 -1 3 -1 0 0 0 0 0 0 0 0] [ 0 0 -1 3 0 0 0 0 0 0 0 0] [ 0 0 0 0 3 -1 0 0 0 0 0 0] [ 0 0 0 0 -1 3 -1 0 0 0 0 0] [ 0 0 0 0 0 -1 3 -1 0 0 0 0] [ 0 0 0 0 0 0 -1 3 0 0 0 0] [ 0 0 0 0 0 0 0 0 3 -1 0 0] [ 0 0 0 0 0 0 0 0 -1 3 -1 0] [ 0 0 0 0 0 0 0 0 0 -1 3 -1] [ 0 0 0 0 0 0 0 0 0 0 -1 3]]  To get the final desired matrix, we can just add another Toeplitz matrix that has the diagonal of -1s. B = A + E(n * m, -1, -4) print(B.astype(int))  # Output: [[ 3 -1 0 0 0 0 0 0 0 0 0 0] [-1 3 -1 0 0 0 0 0 0 0 0 0] [ 0 -1 3 -1 0 0 0 0 0 0 0 0] [ 0 0 -1 3 0 0 0 0 0 0 0 0] [-1 0 0 0 3 -1 0 0 0 0 0 0] [ 0 -1 0 0 -1 3 -1 0 0 0 0 0] [ 0 0 -1 0 0 -1 3 -1 0 0 0 0] [ 0 0 0 -1 0 0 -1 3 0 0 0 0] [ 0 0 0 0 -1 0 0 0 3 -1 0 0] [ 0 0 0 0 0 -1 0 0 -1 3 -1 0] [ 0 0 0 0 0 0 -1 0 0 -1 3 -1] [ 0 0 0 0 0 0 0 -1 0 0 -1 3]]
Question: I'm using the mpmath plot function (which simply uses pyplot, as far as I understood). Consider the following code: from math import cos, sin import mpmath as mp mp.plot([sin, cos], [0, 3]) # this is fine l = [sin, cos] mp.plot([lambda x: f(2*x) for f in l], [0, 3]) # this only plots sin(2x)!  Is there anything I'm missing here, or it's a bug in the plot function?

Answer: See the relevant documentation here. Here's a quick fix that does what you want. l = [sin, cos] mp.plot([lambda x, f=f: f(2*x) for f in l], [0, 3])  So, what's going on here? The key is that each lambda x: f(2*x) is equivalent to something of the form def func(x): return f(2*x)  Importantly, the f within each lambda function is NOT replaced by the corresponding function from l in the list comprehension, there is literally an f in the function definition. As such, whenever the lambda function is called, Python looks for something called f. f is not defined within the scope of the function, so it uses the value of f from the next level up, which is the scope of the list comprehension (interestingly, f is not a variable in the main scope). Because the list has already been constructed, f within the scope of the list comprehension refers to the last item of the list l, namely cos. For that reason, both functions within the list [lambda x: f(2*x) for f in l] yield cos(2x). The quick fix I provide puts the desired f into the scope of the function through an optional argument. If we write out the first lambda x, f=f: f(2*x) as a long form function definition, we have the following: def func(x, f=sin): return f(2*x)  The key here is that unlike the f of the f(2*x), the f on the right hand side of the = in f=f DOES get replaced by the corresponding function from l in the list comprehension. So, there is no need for Python to go outside of the local scope of the function.  Here's an alternative approach, using the "function factory" method suggested here. def make_func(f): def func(x): return f(2*x) return func l = [sin, cos] mp.plot([make_func(f) for f in l], [0,3])  Or, sticking to lambdas, def make_func(f): return lambda x: f(2*x) l = [sin, cos] mp.plot([make_func(f) for f in l], [0,3])  The key here is that the value of f (i.e. sin or cos) is kept within the scope of the make_func function call. A potential advantage of this method is that the resulting lambda functions do not have f as an optional parameter. In a sense, this amounts to "decorating lambdas", as you suggest in your comment.
Question: Suppose i have a tensor t consisting only zeros and ones: t = torch.Tensor([1, 0, 0, 1]) And a dict with the weights: weights = {0: 0.1, 1: 0.9} I want to form a new tensor new_t, such that every element in tensor t is mapped to the corresponding value in the dict weights: new_t = torch.Tensor([0.9, 0.1, 0.1, 0.9]) Is there an elegant way to do this without iterating over tensor t? I've heard about torch.apply, but it only works if tensor t is on the CPU, is there any other options?

Answer: If you convert your weights dict into a tensor, you can index directly t = torch.tensor([1, 0, 0, 1]) weights = torch.tensor([0.1, 0.9]) new_t = weights[t] new_t >tensor([0.9000, 0.1000, 0.1000, 0.9000])
Question: My SQLAlchemy ORM model is populated by a JSON file that occasionally changes. The JSON file does not provide an integer primary key but has a unique alphanumeric ProductCode. My model: class ProductDescriptor(Base): __tablename__ = 'product_descriptor' id: Mapped[int] = mapped_column(primary_key=True, autoincrement=True) ProductCode: Mapped[str] = mapped_column(String(50), unique=True) DisplayName: Mapped[str] = mapped_column(String(50)) Description: Mapped[str] = mapped_column(String(1000)) ...  This answer makes sense until this line: on_duplicate_stmt = insert_stmt.on_duplicate_key_update(dict(txt=insert_stmt.inserted.txt))  Because the incoming data lacks a key I have nothing to compare. Do I need to change the ProductCode definition? I am using Unique=True. My code: product_list = [] for product in products: # Create filtered list of dicts to be send to the DB product_list.append({ 'ProductName': product.get('ProductName'), 'DisplayName': product.get('DisplayName'), 'Description': product.get('Description'), ... more columns }) insert_stmt = insert(ProductDescriptor).values(product_list) # This is where it goes wrong on_duplicate_stmt = insert_stmt.on_duplicate_key_update() # Trying to deal with integrity errors and rollback requests for product in product_list: try: self.session.add(resource) self.session.commit() except IntegrityError: pass   How can I efficiently create an update or create function for bulk records? Do I need to turn my unique field into a key field? Can I keep an autoincrement key field as well?  In Django I would be using the update_or_create method where I can specify the key field and provide a defaults dictionary: Profile.objects.update_or_create(custid=user_profile.custid, defaults=defaults) 

Answer: SQLite lets us specify the matching columns for ON CONFLICT, like so: from sqlalchemy.dialects.sqlite import insert new_values = json.loads("""\ [ {"ProductCode": "code_1", "DisplayName": "display_1", "Description": "description_1"}, {"ProductCode": "code_2", "DisplayName": "display_2", "Description": "description_2"} ] """) insert_stmt = insert(ProductDescriptor).values(new_values) do_update_stmt = insert_stmt.on_conflict_do_update( index_elements=["ProductCode"], set_=dict( DisplayName=insert_stmt.excluded.DisplayName, Description=insert_stmt.excluded.Description, ), ) engine.echo = True with engine.begin() as conn: conn.execute(do_update_stmt) """ BEGIN (implicit) INSERT INTO product_descriptor ("ProductCode", "DisplayName", "Description") VALUES (?, ?, ?), (?, ?, ?) ON CONFLICT ("ProductCode") DO UPDATE SET "DisplayName" = excluded."DisplayName", "Description" = excluded."Description" [no key 0.00093s] ('code_1', 'display_1', 'description_1', 'code_2', 'display_2', 'description_2') COMMIT """  Note that if "ProductCode" is a unique non-nullable column then it is in fact a natural primary key, so the autoincrement integer "id" column is not really necessary