eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have seen examples where slope, yintercept = numpy.polyfit(x,y,1) is used to return slope and y-intercept, but the documentation does not mention "slope" or "intercept" anywhere. The documentation also states three different return options:  p: ndarray, shape (deg + 1,) or (deg + 1, K) Polynomial coefficients, highest power first. If y was 2-D, the coefficients for k-th data set are in p[:,k],   or  residuals, rank, singular_values, rcond These values are only returned if full == True  residuals: sum of squared residuals of the least squares fit rank: the effective rank of the scaled Vandermonde singular_values: singular values of the scaled Vandermonde rcond: value of rcond,    or  v: ndarray, shape (deg + 1, deg + 1) or (deg + 1, deg + 1, K) Present only if full == False and cov == True   I have also run code with exactly the previous line and the slope and yintercept variables are filled with a single value each, which does not fit any of the documented returns. Which documentation am I supposed to use, or what am I missing?

Answer:  slope, yintercept = numpy.polyfit(x, y, 1)   Both full and cov parameters are False, so it's the first option, "Polynomial coefficients, highest power first". deg is 1, so "p: ndarray, shape (deg + 1,)" is an array of shape (2,). With tuple assignment,  slope is p[0], the xdeg-0 = x1 coefficient, yintercept is p[1], the xdeg-1 = x0 coefficient.
Question: I'm using timeseries-generator from GitHub to get synthetic line graphs. That works good. But now I want to save the plots into a file as separate png images. This is my code so far: i=range(0,4) #first number inclusive, second exclusive for element in i: outpath = "PATH" c=random.random() o=random.random() s=random.random() lt = LinearTrend(coef=c, offset=o, col_name="plot no. {0}".format(element)) g = Generator(factors={lt}, features=None, date_range=pd.date_range(start="01-01-2020", end="12-31-2020")) wn = WhiteNoise(stdev_factor=s) g.update_factor(wn) g.generate() g.plot() g.savefig(path.join(outpath,"graph_{0}.png".format(element)))  The last line get me following error: AttributeError: 'Generator' object has no attribute 'savefig' The type of object g is: print(type(g))  <class 'timeseries_generator.generator.Generator'> I tried following suggestion from an other post: Save image from generator object: with open("output_file.png", "wb") as fp: for chunk in thumb_local: fp.write(chunk)  But that doesn't work for me. Maybe I'm using it wrong, with: with open("graph_{0}.png".format(element), "wb") as fp: fp.write(g)  I get following error: TypeError: a bytes-like object is required, not 'Generator' Is there a way to change the generator object into something else to save the images or is there a way to save the generator objects as png? Or can I use the suggestion above, but do it wrong? Thank you for your help in advance!

Answer: g.plot() simply calls the plot() method of the underlying Pandas dataframe. You should get the dataframe and export it on your own: fig = g.ts.plot(kind='line', figsize=(20, 16)).get_figure() fig.savefig(path.join(outpath,"graph_{0}.png".format(element)))
Question: I am trying to use np.where on a MxN numpy matrix, where I want to return the same number of M rows but the indices in each row where the element exists. Is this possible to do so? For example: a = [[1 ,2, 2] [2, 3, 5]] np.where(a == 2)  I would like this to return: [[1, 2], [0]] 

Answer: One option is to post-process the output of where, then split: a = np.array([[1, 2, 2], [2, 3, 5]]) i, j = np.where(a == 2) out = np.split(j, np.diff(i).nonzero()[0]+1)  Alternatively, using a list comprehension: out = [np.where(x==2)[0] for x in a]  Output: [array([1, 2]), array([0])]  using this output to average another array a = np.array([[1, 2, 2], [2, 3, 5]]) b = np.array([[10, 20, 30], [40, 50, 60]]) m = a == 2 i, j = np.where(m) # (array([0, 0, 1]), array([1, 2, 0])) idx = np.r_[0, np.diff(i).nonzero()[0]+1] # array([0, 2]) out = np.add.reduceat(b[m], idx)/np.add.reduceat(m[m], idx) # array([50, 40])/array([2, 1])  Output: array([25., 40.])  handling NaNs: a = np.array([[1, 2, 2], [2, 3, 5]]) b = np.array([[10, 20, np.nan], [40, 50, 60]]) m = a == 2 i, j = np.where(m) # (array([0, 0, 1]), array([1, 2, 0])) idx = np.r_[0, np.diff(i).nonzero()[0]+1] # array([0, 2]) b_m = b[m] # array([20., nan, 40.]) nans = np.isnan(b_m) # array([False, True, False]) out = np.add.reduceat(np.where(nans, 0, b_m), idx)/np.add.reduceat(~nans, idx) # array([20., 40.])/array([1, 1])  Output: array([20., 40.])
Question: I want to simplify the following expression using sympy: -8*F_p + 8*Omega*u + 6*alpha*u*(u**2 + v**2) - 5*beta*u*(u**4 + 2*u**2*v**2 + v**4) - 8*gamma*omega*v  So it ends up like -8*F_p + 8*Omega*u + 6*alpha*u*r**2 - 5*beta*u*r**4 - 8*gamma*omega*v  with r**2=u**2+v**2 It is not working though, but when I use factor(u**4 + 2*u**2*v**2 + v**4).subs(u**2+v**2, r**2)  The output I get is r**4 as expected. Unfortunately, I don't get this simplification when I try this with the first expression. I wonder if there is a way to tell sympy to factor only the expressions inside the innermost parentheses. Here is the full code from sympy import symbols u, v, r = symbols('u v r') # parameters alpha = symbols('alpha', positive=True) beta = symbols('beta', positive=True) F_p = symbols('F_p', positive=True) gamma = symbols('gamma', positive=True) omega = symbols('omega', positive=True) Omega = symbols('Omega', positive=True) f=-8*F_p + 8*Omega*u + 6*alpha*u**3 + 6*alpha*u*v*v - 5*beta*u**5 - 10*beta*u**3*v**2 - 5*beta*u*v**4 - 8*gamma*omega*v print(f) f = collect(f, [alpha, beta]) print(simplify(f)) # Substitute u^2 + v^2 = r^2 subs_dict = {u**2 + v**2: r**2} # Simplify the expression simplified_expr = simplify(f.subs(subs_dict)) print(simplified_expr) print(factor(u**4 + 2*u**2*v**2 + v**4).subs(u**2+v**2, r**2)) 

Answer: Sometimes the easiest thing to do is to re-arrange the expression you are trying to replace: >>> f.subs(v**2, r**2 - u**2).expand() -8*F_p + 8*Omega*u + 6*alpha*r**2*u - 5*beta*r**4*u - 8*gamma*omega*v  For an older version of SymPy, that doesn't replace the v**2 in the way you want, you might have to fall back to using v, instead and then cleaning up residual v-expressions. >>> s=sqrt(var('r',positive=True)**2-u**2) >>> f.subs(v,s).expand().subs(s,v) -8*F_p + 8*Omega*u + 6*alpha*r**2*u - 5*beta*r**4*u - 8*gamma*omega*v
Question: If I have 2 .csv files stored locally data/file_1.csv and data/file_2.csv which both have the same schema, it is easy to polars-read both of them in to 1 concatenated data frame like so: pl.read_csv('data/file_*.csv')  But if I am storing these same 2 files within Google Drive (not a GCS bucket), and I am using GDriveFileSystem from pydrive2.fs as my fsspec file system, I cannot find a way to make use of the glob pattern and have to read them in separately, e.g. fs = GDriveFileSystem(ROOT_FOLDER_ID, client_id = CLIENT_ID, client_secret = CLIENT_SECRET) dfs = [] for i in range(1, 3): with fs.open(f'{ROOT_FOLDER_ID}/data/file_{i}.csv', 'rb') as f: dfs += pl.read_csv(f) df = pl.concat(dfs)  Not only does this mean I need to know and specify the amount of files and their exact file paths in advance, but the code also just feels a lot less cleaner than before. Is there any way I can still read these multiple files with a glob path but using the fsspec file system?

Answer: Although pydrive2 has an fsspec interface, it doesn't seem to declare a protocol or register itself with fsspec, so calls like fsspec.open("gdrive://...", ) are not automatically recognised. This is the intended usage, so I suggest an issue should be raised with them to make sure this gets implemented. You could call fsspec.register_implementation manually to assign a protocol to the PyDrive2 fsspec class. The older, less complete and unreleased gdrivefs does support this usage, because it is listed in fsspec.registry.known_implementations.
Question: jaxtyping provides type annotations that use a str as parameter (as opposed to a type), e.g.: Float[Array, "dim1 dim2"]  Let's say I would like to create a type alias which combines the Float and Array part. This means I would like to be able to write MyOwnType["dim1 dim2"]  instead. To my understanding I cannot use TypeAlias/TypeVar, as the generic parameter (here "dim1 dim2") is not a type but actually an instance of a type. Is there a concise Pythonic way to achieve this? EDIT: I tried the following and it does not work: class _Singleton: def __getitem__(self, shape: str) -> Float: return Float[Array, shape] MyOwnType = _Singleton()  Using MyOwnType["dim1 dim2"]as function parameter annotation gives the mypy complaint Variable "MyOwnType" is not valid as a type Solution: Based on the answer of @chepner this was the final solution: class MyOwnType(Generic[Shape]): def __class_getitem__(cls, shape: str) -> Float: return Float[Array, shape] 

Answer: You need to override/define __class_getitem__, not __getitem__ (which applies to instances of _Singleton, not _Singleton itself). class _Singleton: def __class_getitem__(self, shape: str) -> Float: return Float[Array, shape]
Question: Let's say I have an array of permutations perm which could look like: perm = np.array([[0, 1, 2], [1, 2, 0], [0, 2, 1], [2, 1, 0]])  If I want to apply it to one axis, I can write something like: v = np.arange(9).reshape(3, 3) print(v[perm])  Output: array([[[0, 1, 2], [3, 4, 5], [6, 7, 8]], [[3, 4, 5], [6, 7, 8], [0, 1, 2]], [[0, 1, 2], [6, 7, 8], [3, 4, 5]], [[6, 7, 8], [3, 4, 5], [0, 1, 2]]])  Now I would like to apply it to two axes at the same time. I figured out that I can do it via: np.array([v[tuple(np.meshgrid(p, p, indexing="ij"))] for p in perm])  But I find it quite inefficient, because it has to create a mesh grid, and it also requires a for loop. I made a small array in this example but in reality I have a lot larger arrays with a lot of permutations, so I would really love to have something that's as quick and simple as the one-axis version.

Answer: How about: p1 = perm[:, :, np.newaxis] p2 = perm[:, np.newaxis, :] v[p1, p2]  The zeroth axis of p1 and p2 is just the "batch" dimension of perm, which allows you to do many permutations in one operation. The other dimension of perm, which corresponds with the indices, is aligned along the first axis in p1 and the second in p2. Because the axes are orthogonal, the arrays get broadcasted, basically like the arrays you got using meshgrid - but these still have the batch dimension. That's the best I can do from my cell phone : ) I can try to clarify later if needed, but the key idea is broadcasting. Comparison: import numpy as np perm = np.array([[0, 1, 2], [1, 2, 0], [0, 2, 1], [2, 1, 0]]) v = np.arange(9).reshape(3, 3) ref = np.array([v[tuple(np.meshgrid(p, p, indexing="ij"))] for p in perm]) p1 = perm[:, :, np.newaxis] p2 = perm[:, np.newaxis, :] res = v[p1, p2] np.testing.assert_equal(res, ref) # passes %timeit np.array([v[tuple(np.meshgrid(p, p, indexing="ij"))] for p in perm]) # 107 µs ± 20.6 µs per loop %timeit v[perm[:, :, np.newaxis], perm[:, np.newaxis, :]] # 3.73 µs ± 1.07 µs per loop   A simpler (without batch dimension) example of broadcasting indices: import numpy as np i = np.arange(3) ref = np.meshgrid(i, i, indexing="ij") res = np.broadcast_arrays(i[:, np.newaxis], i[np.newaxis, :]) np.testing.assert_equal(res, ref) # passes  In the solution code at the top, the broadcasting is implicit. We don't need to call broadcast_arrays because it happens automatically during the indexing.
Question: I am attempting to suppress a message that is printed to stdout by a library implemented in C. My specific usecase is OpenCV, so I will use it for the MCVE below. The estimateChessboardSharpness function has a printout when the grid size is too small (which happens here). I made a PR to fix it, but in the meantime, I'd like to suppress the message. For example: import cv2 import numpy as np img = np.zeros((512, 640), dtype='uint8') corners = [] for i in range(10): for j in range(8): corner = (30 + 3 * j, 70 + 3 * i) if i and j: corners.append(corner) if (i % 2) ^ (j % 2): img[corner[0]:corner[0] + 3, corner[1]:corner[1] + 3] = 255 corners = np.array(corners)  >>> cv2.estimateChessboardSharpness(img, (9, 7), corners) calcEdgeSharpness: checkerboard too small for calculation. ((9999.0, 9999.0, 9999.0, 9999.0), None)  The line appears to be a simple std::cout << ..., so I have tried all of the following: from contextlib import redirect_stdout from os imoprt devnull import sys with redirect_stdout(None): cv2.estimateChessboardSharpness(img, (9, 7), corners) with open(devnull, "w") as null, redirect_stdout(null): cv2.estimateChessboardSharpness(img, (9, 7), corners) sys.stdout = open(devnull, "w") cv2.estimateChessboardSharpness(img, (9, 7), corners)  I've even tried redirect_stderr instead of redirect_stdout just in case. I've also tried setting OPENCV_LOG_LEVEL=SILENT in bash and os.environ["OPENCV_LOG_LEVEL"] = "SILENT" in python before importing cv2, not that I expected stdout to be conflated with logging in this case. In all cases, the message prints. How do I make it stop?

Answer: Assuming a UNIX-like platform, (you'll get an OSError if not) You can save the underlying fd with backup = os.dup(sys.stdout.fileno())  point it to a different file (eg. different) with os.dup2(different.fileno(), sys.stdout.fileno())  and resume normal service with os.dup2(backup, sys.stdout.fileno())  Do note that this won't behave at all nicely if the file you redirect to is also being used for buffered output (ie, you have writes to different mixed with C++ library writes to std::cout which is using different.fileno()...)
Question: Current Workflow Right now, me, as an ML engineer, am using a jupyter notebook to plot some pandas dataframes. Basically, inside the jupyter notebook, I am setting some hard-coded configurations like k_1:float=2.3 date_to_analyse:str='2024-06-17' # 17th June region_to_analyse:str='montana' ...  Based on the hardcoded parameters like above, the notebook (along with some helper functions) runs a query to my company's data warehouse in Google Big Query, crunches a few numbers, applies some business logic and generates a pandas dataframe for transaction numbers minute by minute throughout the day. The plot of the pandas dataframe is what's important to the business user, which I show to them as part of our model evaluation metric. The logic is straightforward enough, and captured in the figure below.  Objective I am tasked with making this into a self-service application directly usable by the business team, where a user can  input the above values via a simple UI the plot appears on the screen.  I don't have a dedicated frontend guy, neither do I have frontend experience myself. So was just wondering whether there is a simple enough solution (best if serverless, and part of the GCP eco-system) that can accomplish this? I know about Google colab notebooks (basically, jupyter notebook, right?), but have not tried them. Can they provide a simple way to expose the notebook's functionality (generate the plots from user provided configs)

Answer: If you can expose the notebook to users (load credentials via env vars etc.) you can use IPython widgets. Google collab has forms and allows you to hide the code, if you think filling out a form an pressing the play button left of the field is managable for your userbase that is an option as well. While not totally serverless gradio, huggingfaces ui library makes it simple enough to create simple webinterfaces. You can get an internal and a public address for it. You could run the gradio code in google collab as well.
Question: Description: I have been using SQL-like padding in my Polars DataFrame queries for better readability and easier commenting of conditions. This approach involves using True (analogous to 1=1 in SQL) to pad the conditional logic. SQL Example In SQL, padding with 1=1 makes it easy to add or remove conditions: SELECT * FROM employees WHERE 1=1 AND department = 'Sales' AND salary > 50000 AND hire_date > '2020-01-01';  SELECT * FROM employees WHERE 1=1 AND department = 'Sales' -- AND salary > 50000 AND hire_date > '2020-01-01';  (>>>> in my opinion <<<<) This makes adding, removing, and replacing operators very easy. Also it simplifies the git diff because adding and removing conditions appear as a single diff. Polars Example Here's the Polars equivalent that I have been using: # Example data data = { "name": ["Alice", "Bob", "Charlie", "David"], "department": ["Sales", "HR", "Sales", "IT"], "salary": [60000, 45000, 70000, 50000], "hire_date": ["2021-06-01", "2019-03-15", "2020-08-20", "2018-11-05"] } # Create a DataFrame df = pl.DataFrame(data) # Filter with padding filtered_df = df.filter( ( True & pl.col("department").eq("Sales") # & pl.col("salary").gt(50000) & pl.col("hire_date").gt("2020-01-01") & True ) ) print(filtered_df)  Concern I am concerned about the potential performance impact of this approach in Polars. Does padding the filter conditions with True significantly affect the performance of query execution in Polars? Also is there a better way to do this ? Thank you for your assistance.

Answer: Personally for SQL i switched to “and at the end of the line” cause i think it’s much more readable like that. It does make commenting out the last line trickier, but i just don’t like 1=1 dummy condition. Luckily, in modern dataframe library like polars you don’t need to do that anymore. Just pass multiple filter condutions instead of combined one. filtered_df = df.filter( pl.col("department").eq("Sales"), # pl.col("salary").gt(50000), pl.col("hire_date").gt("2020-01-01"), )
Question: I am trying to horizontally concatenate a pair of data frames with identical indices, but the result is always a vertical concatenation with NaN values inserted into every column. dct_l = {'1':'a', '2':'b', '3':'c', '4':'d'} df_l = pd.DataFrame.from_dict(dct_l, orient='index', columns=['Key']) dummy = np.zeros((4,3)) index = np.arange(1,5) columns = ['POW', 'KLA','CSE'] df_e = pd.DataFrame(dummy, index, columns) print(df_l)   Key 1 a 2 b 3 c 4 d  print(df_e)   POW KLA CSE 1 0.0 0.0 0.0 2 0.0 0.0 0.0 3 0.0 0.0 0.0 4 0.0 0.0 0.0  pd.concat([df_l, df_e], axis=1)  Actual Result  Key POW KLA CSE 1 a NaN NaN NaN 2 b NaN NaN NaN 3 c NaN NaN NaN 4 d NaN NaN NaN 1 NaN 0.0 0.0 0.0 2 NaN 0.0 0.0 0.0 3 NaN 0.0 0.0 0.0 4 NaN 0.0 0.0 0.0  Expected Result  Key POW KLA CSE 1 a 0.0 0.0 0.0 2 b 0.0 0.0 0.0 3 c 0.0 0.0 0.0 4 d 0.0 0.0 0.0  What is happening here?

Answer: You have different dtypes for your two indexes: df_e.index Index([1, 2, 3, 4], dtype='int64') df_l.index Index(['1', '2', '3', '4'], dtype='object')  which will break the alignment (1 != '1'). Make sure they are identical. For example: pd.concat([df_l.rename(int), df_e], axis=1) Key POW KLA CSE 1 a 0.0 0.0 0.0 2 b 0.0 0.0 0.0 3 c 0.0 0.0 0.0 4 d 0.0 0.0 0.0
Question: I am finding it difficult to write regex expression. Need your help in getting expected output. import re pattern1=r'(\d)\1{1,}' s = '1222311' c=re.finditer(pattern,s) for i in c: print(i.__getitem__(0))  I managed to get repeating digit but not able to get non repeating digit. How to frame regex to get non repeating digits and repeating digits. s = '1222311'  Expected output Repeating digit : 222 , 11 Non Repeating digit : 1,3 

Answer: Changing your pattern to r'(\d)(\1*)' will select all digits, and group together each set of consecutive digits. Then with your list of matches, simply sort them based on length: import re s = '1222311' pattern = r'(\d)\1*' repeating_digits = [] non_repeating_digits = [] # finditer produces the following matches `['1', '222', '3', '11']` for match in re.finditer(pattern, s): digits = match.group(0) if len(digits) > 1: repeating_digits.append(digits) else: non_repeating_digits.append(digits) print(f"Repeating digit : {' , '.join(repeating_digits)}") print(f"Non Repeating digit : {' , '.join(non_repeating_digits)}")  Pattern Explanation (\d) matches any single digit (0 - 9), and is the first capture group. \1 is a back reference to the first capture group, (\d). It will match whatever was matched by the first capture group, which in this case is a single digit. Adding the asterisk to the back reference \1* will match the digit from the first capture group when repeated 1 or more times.
Question: I'm using an Arch Linux machine and trying to run the following code from a Python file. import socket import sys if __name__ == "__main__": print(sys.platform) server = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)  and it keeps telling me that AttributeError: module 'socket' has no attribute 'AF_UNIX'  Things tried  Some posts claim this error occurs on Windows but obviously that isn't the case.  sys.platform prints linux   Code works on my Mac which was running Python3.9  Downgraded from Python3.12 to Python3.9 on the Linux machine and still no luck   socket.AF_INET has the same issue running python -c "import socket; socket = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) returns the same error error when using system binary python, conda python and venv python running it in the interactive shell has no issue however. 

Answer: The error is caused by an unlucky choice of variable name. The variable socket shadows the module of the same name. The first time the line is run, it works fine: import socket socket = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) # works fine, but now socket points to the object returned by the socket.socket() call socket = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) # error!  Solution: use a different variable name, e.g. sock: sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
Question: I want to make a code which works like this: after certain seconds I want to terminate the working function in the main function. I've tried the below code but when the code in the loop function isn't an async function(like await asyncio.sleep(1)) it doesn't work. import asyncio async def loop(): while True: pass async def main(): task = asyncio.Task(loop()) await asyncio.sleep(3) print('1223') task.cancel() print('x') asyncio.run(main()) 

Answer: Use multiprocessing. When you want a function to be terminated after certain seconds, then all the function should start running simultaneously. My comments inside the code explains the rest. import multiprocessing as mp import time as t # function which needs to be terminated after certain seconds def func1(): # Do something def func2(): # Do something # A function to terminate func1 def stop(func): # terminate the func after 1 sec t.sleep(1) func.terminate() # Add execution guard in case "spawn"-multiprocessing mode is used if __name__ == "__main__": # Create two processes pro_1 = mp.Process(target=func1) pro_2 = mp.Process(target=func2) # third process being stop() and the args being pro_1 pro_3 = mp.Process(target=stop, args=(pro_1,)) # Start the processes pro_1.start() pro_2.start() pro_3.start() #As the pro_1 is terminated use process.join for only pro_2 and pro_3 pro_2.join() pro_3.join()