eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I need to group by multiple columns on a dataframe and calculate the rolling mean in the group. But the original index needs to be preserved. Simple python code below : data = {'values': [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15], 'type':['A','B','A','B','A','B','A','B','A','B','A','B','A','B','A'], 'type2':['C','D','C','D','C','D','C','D','C','D','C','D','C','D','C']} df = pd.DataFrame(data) window_size = 3 df['mean'] = (df.groupby(['type','type2'])['values'].rolling(window=3).mean().reset_index(drop=True)) print(df)  Output :  values type type2 mean 0 1 A C NaN 1 2 B D NaN 2 3 A C 3.0 3 4 B D 5.0 4 5 A C 7.0 5 6 B D 9.0 6 7 A C 11.0 7 8 B D 13.0 8 9 A C NaN 9 10 B D NaN 10 11 A C 4.0 11 12 B D 6.0 12 13 A C 8.0 13 14 B D 10.0 14 15 A C 12.0  What I need :  values type type2 mean 0 1 A C 1 1 2 B D 2 2 3 A C 2 3 4 B D 3 4 5 A C 3 5 6 B D 4 6 7 A C 5 7 8 B D 6 8 9 A C 7 9 10 B D 8 10 11 A C 9 11 12 B D 10 12 13 A C 11 13 14 B D 12 14 15 A C 13  The requirement is very simple. The mean has to be calcluated in the groups. So last row is group (A,C) . So it is 15+ 13(previous) + 11 (previous to previous because window is 3) = 39 /3 =13 Same for other rows. But when I do this with level = 0 with below code I get df['mean'] = (df.groupby(['type','type2'])['values'].rolling(window=3).mean().reset_index(level=0,drop=True))   raised in MultiIndex.from_tuples, see test_insert_error_msmgs 12690 if not value.index.is_unique: 12691 # duplicate axis 12692 raise err 12693 > 12694 raise TypeError( 12695 "incompatible index of inserted column with frame index" 12696 ) from err 12697 return reindexed_value, None TypeError: incompatible index of inserted column with frame index  How to go about this simple requirement ?

Answer: You should use droplevel: cols = ['type', 'type2'] df['mean'] = (df.groupby(cols)['values'] .rolling(window=3).mean() .droplevel(cols) )  Output:  values type type2 mean 0 1 A C NaN 1 2 B D NaN 2 3 A C NaN 3 4 B D NaN 4 5 A C 3.0 5 6 B D 4.0 6 7 A C 5.0 7 8 B D 6.0 8 9 A C 7.0 9 10 B D 8.0 10 11 A C 9.0 11 12 B D 10.0 12 13 A C 11.0 13 14 B D 12.0 14 15 A C 13.0  And to avoid the NaNs, add min_periods=1: cols = ['type', 'type2'] df['mean'] = (df.groupby(cols)['values'] .rolling(window=3, min_periods=1).mean() .droplevel(cols) )  Output:  values type type2 mean 0 1 A C 1.0 1 2 B D 2.0 2 3 A C 2.0 3 4 B D 3.0 4 5 A C 3.0 5 6 B D 4.0 6 7 A C 5.0 7 8 B D 6.0 8 9 A C 7.0 9 10 B D 8.0 10 11 A C 9.0 11 12 B D 10.0 12 13 A C 11.0 13 14 B D 12.0 14 15 A C 13.0
Question: I am defining a piecewise function for some data, def fit_jt(x, e1, e2, n1, E1, E2, N1, N2): a = 1.3 return np.piecewise(x, [x <= a, x > a], [ lambda x: 1 / e1 + (1 - np.float128(np.exp(-e2 * x / n1))) / e2, lambda x: 1 / E1 + (1 - np.float128(np.exp(-E2 * x / N1))) / E2 + x / N2 ])  which is called in main as:  popt_jt, pcov_jt = optimize.curve_fit(fit_jt, time.values, jt.values, method='trf')  Now, the problem here is the a is hardcoded in the function fit_jt. Is it possible to supply the value of a from the main (without making a lot of changes)?

Answer: Use a factory function that returns a fit_it function with the desired value of a "baked in" via a closure: def make_fit_it(a): def fit_jt(x, e1, e2, n1, E1, E2, N1, N2): return np.piecewise(x, [x <= a, x > a], [ lambda x: 1 / e1 + (1 - np.float128(np.exp(-e2 * x / n1))) / e2, lambda x: 1 / E1 + (1 - np.float128(np.exp(-E2 * x / N1))) / E2 + x / N2 ]) return fit_it  To use it: popt_jt, pcov_jt = optimize.curve_fit(make_fit_jt(1.3), time.values, jt.values, method='trf')
Question: Given a numpy array arr = np.array([1, 2, 3, 4, 5])  I need to construct a binary mask according to a (arbitrary, potentially long) list of values, i.e. given values = np.array([2, 4, 5])  mask should be mask = np.array([False, True, False, True, True])  So I want to avoid condition = (arr==2) or (arr==4) or (arr==5) mask = np.where(condition, arr)  to get something like mask = np.in(values, arr)  Or, if it is not possible, how to construct a condition from an arbitrary list of values to feed into np.where?

Answer: In [64]: arr = np.array([1, 2, 3, 4, 5]) ...: values = np.array([2, 4, 5])  While isin is easy to use, it isn't the only option: In [66]: np.isin(arr, values) Out[66]: array([False, True, False, True, True])  We could compare the whole arrays: In [67]: values[:,None]==arr Out[67]: array([[False, True, False, False, False], [False, False, False, True, False], [False, False, False, False, True]]) In [68]: (values[:,None]==arr).any(axis=0) Out[68]: array([False, True, False, True, True])  Your use of or does not work: In [69]: (arr==values[0]) or (arr==values[1]) or (arr==values[2]) --------------------------------------------------------------------------- ValueError Traceback (most recent call last) Cell In[69], line 1 ----> 1 (arr==values[0]) or (arr==values[1]) or (arr==values[2]) ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()  We have to use the array logical_or: In [70]: (arr==values[0]) | (arr==values[1]) | (arr==values[2]) Out[70]: array([False, True, False, True, True])  Depending on the relative size of the two array isin may actually do this kind of logical_or. Let's check the times (actual times will depend on the sizes): In [71]: timeit (arr==values[0]) | (arr==values[1]) | (arr==values[2]) 13.8 μs ± 75.8 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each) In [72]: timeit np.isin(arr, values) 97 μs ± 242 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each) In [73]: timeit (values[:,None]==arr).any(axis=0) 17 μs ± 101 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)  isin is convenient, but slowest in this example.
Question: I'm calculating rotation quaternions from Euler angles in Python using SciPy and trying to validate against an external source (Wolfram Alpha). This Scipy code gives me one answer: from scipy.spatial.transform import Rotation as R rot = R.from_euler('xyz', [30,45,60], degrees=1) quat = rot.as_quat() print(quat[3], quat[0], quat[1], quat[2]) # w, x, y, z (w,x,y,z) = 0.8223, 0.0222, 0.4396, 0.3604  while Wolfram Alpha gives a different answer (w,x,y,z) = 0.723, 0.392, 0.201, 0.532   Why the difference? Is it a difference in paradigm for how the object is rotated (ex. extrinsic rotations vs. intrinsic rotations)?

Answer: Indeed, when you switch from small "xyz", which denotes extrinsic rotations, to capital "XYZ", which denotes intrinsic rotations in from_euler(), the results will match those of WolframAlpha: from scipy.spatial.transform import Rotation import numpy as np np.set_printoptions(precision=3) rot = Rotation.from_euler("XYZ", [30, 45, 60], degrees=True) print(rot.as_matrix().T) # [[ 0.354 0.927 0.127] # [-0.612 0.127 0.78 ] # [ 0.707 -0.354 0.612]] print(np.roll(rot.as_quat(), shift=1)) # [0.723 0.392 0.201 0.532]
Question: I have the following heatmap: import seaborn as sns import matplotlib.pyplot as plt import numpy as np # Create a sample heatmap data = np.random.rand(10, 12) ax = sns.heatmap(data) plt.show()  How can I add a black edge to the colormap? Is there any way to do this without needing to use subplots?

Answer: Looks like seaborn turns the edges off by default. Here's an approach to get a reference to the colorbar Axes, and then re-apply the edge: ax = sns.heatmap(data) cax = ax.figure.get_children()[-1] cax.spines['outline'].set_linewidth(0.5) # adjust as desired  Output:
Question: I am trying to create a dictionary from a string. In this case I have posted my sample code (sorry its not that clean, just hardcoded values), the first str1 works fine and is able generate a corresponding dictionary by splitting correctly ; and associating key value at = sign. However, the second string (str5) is not working. I am assuming its because there is an extra "=" at: 1 = '< 24 hours'; 2 = '> 24 hours, <**=** 30 days'; Can you please tell me how I can resolve this issue of ignoring the = sign and continue creating the dictionary. I get the following error: 0=Blank;1=<24hours;2=>24hours,<=30days;3=>30days(i.e.,permanent) Traceback (most recent call last): File xmlread.py:303 in <module> main() File xmlread.py:299 in main stringParseTest() File xmlread.py:283 in stringParseTest key, value = pair.split("=") ValueError: too many values to unpack (expected 2)  I could try parsing it manually and not create a dictionary; however I would like to see if there is a condition I could add in there to recognize this issue. Don't know how to do that though. What if I add an if statement to check how many times a string has been splitted. def stringParseTest(): str1 = str("0 = Blank; 1 = Surface Device: Skin; 2 = Surface Device: Mucosal Membrane; 3 = Surface Device: Breached or Compromised Surfaces; 4 = External Communicating Device: Blood Path, Indirect; 5 = External Communicating Device: Tissue/Bone/Dentin; 6 = External Communicating Device: Circulating Blood; 7 = Implant Device: Tissue/Bone; 8 = Implant Device: Blood") str5 = str("0 = Blank; 1 = '< 24 hours'; 2 = '> 24 hours, <= 30 days'; 3 = '> 30 days (i.e., permanent)'") dictionary = {} #Working string split str1 = str1.replace(" ", "") str1 = str1.replace("'", "") print(str1) for pair in str1.split(";"): key, value = pair.split("=") dictionary[key] = value print(dictionary) #Not working split str5 = str5.replace(" ", "") str5 = str5.replace("'", "") print(str5) for pair in str5.split(";"): key, value = pair.split("=") dictionary[key] = value print(dictionary) 

Answer: In split function you can pass additional parameter like maxsplit=1, so that it will split only based on first occurence of delimiter. i.e.g pair.split("=", 1) for pair in str5.split(";"): key, value = pair.split("=", 1) dictionary[key] = value
Question: I was learning boolean indexing in numpy and came across this. How is the indexing below not producing a Index Error as for axis 0 as there are only two blocks? x = np.arange(30).reshape(2, 3, 5) x array([[[ 0, 1, 2, 3, 4], [ 5, 6, 7, 8, 9], [10, 11, 12, 13, 14]], [[15, 16, 17, 18, 19], [20, 21, 22, 23, 24], [25, 26, 27, 28, 29]]]) x[[[True, True, False], [False, True, True]]] array([[ 0, 1, 2, 3, 4], [ 5, 6, 7, 8, 9], [20, 21, 22, 23, 24], [25, 26, 27, 28, 29]]) 

Answer: You are performing boolean array indexing, which is fine. You would have an indexing error with: # first dimension second dimension x[[True, True, False], [False, True, True]] # IndexError: boolean index did not match indexed array along dimension 0; # dimension is 2 but corresponding boolean dimension is 3  However, in your case, you have an extra set of brackets, which makes it index only the first dimension, using an array: # [ first dimension ] # [ second dimension], [ second dimension] x[[[True, True, False], [False, True, True]]]  This means, that using a 2x3 array, you request in the second dimension, for the first "row" [True, True, False], and for the second "row" [False, True, True]. Since your array shape matches the first two dimensions, this is valid, and more or less equivalent to: np.concatenate([x[0][[True, True, False]], x[1][[False, True, True]]])
Question: I have the following list: plates = [[], [], [{'plate ID': '193a', 'ra': 98.0, 'dec': 11.0, 'sources': [[3352102441297986560, 99.28418829069784, 11.821604434173034], [3352465726807951744, 100.86164898224092, 12.756149587760696]]}], [{'plate ID': '194b', 'ra': 98.0, 'dec': 11.0, 'sources': [[3352102441297986560, 99.28418829069784, 11.821604434173034], [3352465726807951744, 100.86164898224092, 12.756149587760696]]}], [], [], [], [], [], [], [], [], [], []]  I'd need to loop plates, find the key 'sources' and store some data to another list. import pandas as pd matched_plates = [] matches_sources_ra = [] matches_sources_dec =[] plates = [[], [], [{'plate ID': '193a', 'ra': 98.0, 'dec': 11.0, 'sources': [[3352102441297986560, 99.28418829069784, 11.821604434173034], [3352465726807951744, 100.86164898224092, 12.756149587760696]]}], [{'plate ID': '194b', 'ra': 98.0, 'dec': 11.0, 'sources': [[3352102441297986560, 99.28418829069784, 11.821604434173034], [3352465726807951744, 100.86164898224092, 12.756149587760696]]}], [], [], [], [], [], [], [], [], [], []] plates_df = pd.DataFrame(plates) for idx, row in plates_df.iterrows(): if 'sources' in row.keys(): print(row["plate ID"]) matched_plates.append([row["plate ID"],len(row["sources"])]) matches_sources_ra.append(row["sources"][0][1]) matches_sources_dec.append(row["sources"][0][2])  This code never enters the if, what am I doing wrong? Thank you for your help

Answer: There is absolutely no need for pandas here. You're just using it as an expensive and slow container. Just loop over the lists and dictionaries, in pure python: matched_plates = [] matches_sources_ra = [] matches_sources_dec = [] for lst in plates: for dic in lst: if 'sources' in dic: print(dic['plate ID']) matched_plates.append([dic['plate ID'], len(dic['sources'])]) matches_sources_ra.append(dic['sources'][0][1]) matches_sources_dec.append(dic['sources'][0][2])  Output: 193a 194b  Output lists: matched_plates # [['193a', 2], ['194b', 2]] matches_sources_ra # [99.28418829069784, 99.28418829069784] matches_sources_dec # [11.821604434173034, 11.821604434173034]
Question: There is quite an old game (that no longer works) that has to make some API calls in order to be playable. I am creating a Flask mock server to handle those requests, however it turned out that the requests are not compliant with HTTP standard and are malformed. For example: Get /config.php http/1.1  to which flask reports code 400, message Bad request version ('http/1.1'). After searching for various solutions, here is what I tried (and none worked):  before_request Python decorator wsgi_app middleware  Here is my code: from flask import Flask, request from functools import wraps app = Flask(__name__) class RequestMiddleware: def __init__(self, app): self.app = app def __call__(self, environ, start_response): print('Middleware', environ['REQUEST_METHOD'], environ['SERVER_PROTOCOL']) return self.app(environ, start_response) def RequestDecorator(view): @wraps(view) def decorated(*args, **kwargs): print('Decorator', args, kwargs) return view(*args, **kwargs) return decorated @app.before_request def RequestHook(): print('Before request: url: %s, path: %s' % (request.url, request.path)) app.wsgi_app = RequestMiddleware(app.wsgi_app) @app.route("/config.php", methods=["GET"]) @RequestDecorator def get_config(): return ("{}", 200)  Example output: Middleware GET HTTP/1.1 Before request: url: http://localhost/config.php, path: /config.php Decorator () {} "GET /config.php HTTP/1.1" 200 - code 400, message Bad request version ('http/1.1') "Get /config.php http/1.1" 400 -  Malformed request is not getting output from any of the solutions. My goal was to intercept the request before it is rejected in order to string replace Get to GET and http/1.1 to HTTP/1.1. Is it even possible?

Answer: Flask uses Werkzeug which in turn uses Python's http.server to parse HTTP requests. So the error is actually thrown by Python's http.server.BaseHTTPRequestHandler @ Line 311. Only way to get around this error is by monkey patching certain methods. If you let me know the version number of Werkzeug that you're running, I'll update my answer with a patch.
Question: I found Nodriver, which is the successor Undetected-Chromedriver. I am trying to run in headless mode but am having problems. import nodriver as uc async def main(): browser = await uc.start(headless=True) page = await browser.get('https://bot.sannysoft.com/') if __name__ == '__main__': uc.loop().run_until_complete(main())  However I get error Traceback (most recent call last): File "C:\no_drive_test.py", line 21, in <module> uc.loop().run_until_complete(main()) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python313\Lib\asyncio\base_events.py", line 721, in run_until_complete return future.result() ~~~~~~~~~~~~~^^ File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python313\no_drive_test.py", line 5, in main browser = await uc.start(headless=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python313\Lib\site-packages\nodriver\core\util.py", line 95, in start return await Browser.create(config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python313\Lib\site-packages\nodriver\core\browser.py", line 90, in create await instance.start() File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python313\Lib\site-packages\nodriver\core\browser.py", line 393, in start await self.connection.send(cdp.target.set_discover_targets(discover=True)) File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python313\Lib\site-packages\nodriver\core\connection.py", line 413, in send await self._prepare_headless() File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python313\Lib\site-packages\nodriver\core\connection.py", line 492, in _prepare_headless response, error = await self._send_oneshot( ^^^^^^^^^^^^^^^ TypeError: cannot unpack non-iterable NoneType object  I tried to create issue on Nodriver github page but it looks like it's only available to collaborators of the project

Answer: The issue was created on the repo link This was suggested: browser_args.append(f"--headless=new")
Question: I'd like to use a function like cumsum, but that would create a set of all values contained in the column up to the point, and not to sum them df = pl.DataFrame({"a": [1, 2, 3, 4]}) df["a"].cum_sum() shape: (4,) Series: 'a' [i64] [ 1 3 6 10 ]  but I'd like to have something like df["a"].cum_sum() shape: (4,) Series: 'a' [i64] [ {1} {1, 2} {1, 2, 3} {1, 2, 3, 4} ]  also note that I'm working on big (several Millions of rows) df, so I'd like to avoid indexing and map_elements (as I've read that it slows down a lot)

Answer: This can be achieved using pl.Expr.cumulative_eval together with pl.Expr.unique and pl.Expr.implode as follows. df.with_columns( res=pl.col("a").cumulative_eval(pl.element().unique().implode()) )  shape: (4, 2) ┌─────┬─────────────┐ │ a ┆ res │ │ --- ┆ --- │ │ i64 ┆ list[i64] │ ╞═════╪═════════════╡ │ 1 ┆ [1] │ │ 2 ┆ [1, 2] │ │ 3 ┆ [1, 2, 3] │ │ 4 ┆ [1, 2, … 4] │ └─────┴─────────────┘
Question: I have a data frame collected from a CSV in the following format: Book Name,Languages "Book 1","['Portuguese','English']" "Book 2","['English','Japanese']" "Book 3","[Spanish','Italian','English']" ...  I was able to convert the string array representation on the column Languages to a python array using transform, but now i'm struggling to find a way to group Books by language. I would like to produce from this data set a dict like this: { 'Portuguese': 'Book 1' 'English': ['Book 1', 'Book 2', 'Book 3'], 'Spanish': 'Book 3', 'Italian': 'Book 3', 'Japanese': 'Book 2' }  I tried to look into groupby on the array column but could not figure out how to make each entry on the array a key to be used as grouping. Any pointers would be really apreciated.

Answer: You can do this by iterating through the DataFrame and updating a dictionary dynamically. import pandas as pd import ast data = { "Book Name": ["Book 1", "Book 2", "Book 3"], "Languages": ["['Portuguese','English']", "['English','Japanese']", "['Spanish','Italian','English']"] } df = pd.DataFrame(data) df["Languages"] = df["Languages"].apply(ast.literal_eval) language_dict = {} for _, row in df.iterrows(): book_name = row["Book Name"] for lang in row["Languages"]: if lang in language_dict: if isinstance(language_dict[lang], list): language_dict[lang].append(book_name) else: language_dict[lang] = [language_dict[lang], book_name] else: language_dict[lang] = book_name print(language_dict)  Output will be { 'Portuguese': 'Book 1', 'English': ['Book 1', 'Book 2', 'Book 3'], 'Japanese': 'Book 2', 'Spanish': 'Book 3', 'Italian': 'Book 3' }
Question: As the question mentions, does clickhouse_connect.get_client in the python client return a new client instance every time it is called? I can't seem to find if it is explicitly mentioned as such in the documentation, but it seems implied. I'm a little confused because of the name get_client (instead of say create_client).

Answer: Yes. You can find this in init.py from clickhouse_connect.driver import create_client, create_async_client driver_name = 'clickhousedb' get_client = create_client get_async_client = create_async_client  Permalink to the line of importance.
Question: I need to work with arrays that can have zeros in their shapes. However, I am encountering an issue. Here's an example: import numpy as np arr = np.array([[]]) assert arr.shape == (1,0) arr.reshape((1,0)) # No problem (nothing changes) arr.reshape((-1,0)) # ValueError: cannot reshape array of size 0 into shape (0)  I always thought that -1 for a reshape operation means the product of all the remaining dimensions, i.e., 1 in this case. Is this a bug, or am I not understanding how this should work?

Answer: If you read the documentation:  One shape dimension can be -1. In this case, the value is inferred from the length of the array and remaining dimensions.  As furas says, it can't automatically calculate the remaining dimension because of undefined division by 0. Any number times 0 is 0. arr.reshape((1,0)) # Works arr.reshape((3,0)) # Works too! arr.reshape((42,0)) # Works too!  Arguably, the error message thrown by NumPy is not the clearest though, and it is factually wrong: arr.reshape(0) # Works as well  So you actually can "reshape array of size 0 into shape (0)". And it should say something along the lines of "undefined reshape requested" instead... So it is not a "bug", but definitely a wart, IMO. It might be worth reporting if you feel like it :)
Question: The goal is to compare two pandas dataframes considering a margin of error. To reproduce the issue: Importing pandas import pandas as pd  Case one - same data dataframes df1 = pd.DataFrame({"A": [1,1,1], "B": [2,2,2], "C": [3,3,3]}) df2 = pd.DataFrame({"A": [1,1,1], "B": [2,2,2], "C": [3,3,3]}) print(df1.compare(df2, result_names=('df1', 'df2'))) # The result is an empty dataframe Empty DataFrame Columns: [] Index: []  Case two - different data dataframes df1 = pd.DataFrame({"A": [1,1,1], "B": [2,2,2], "C": [3,3,3]}) df2 = pd.DataFrame({"A": [1,1,1], "B": [2,2.2,2], "C": [3,3,3]}) # Note that the second B value is 2.2 print(df1.compare(df2, result_names=('df1', 'df2'))) # The result is a dataframe showing differences B df1 df2 1 2.0 2.2  The issue is that I want that it only considers differences greater than 0.5 How I achieved it. threshold = 0.5 df3 = df1.melt().reset_index().merge(df2.melt().reset_index(), on="index") df3["diff"] = (df3["value_x"] - df3["value_y"]).abs() print(df3.loc[df3["diff"] > threshold]) # The result is an empty dataframe Empty DataFrame Columns: [index, variable_x, value_x, variable_y, value_y, diff] Index: []  Is there a better way to do this? It takes a lot of time for a huge DF. In time:  This is only a reproducible example. I am opened to use other libraries as Numpy. 

Answer: Depending on your ultimate goal, assert_frame_equal with the atol parameter may work. from pandas.testing import assert_frame_equal # specify dtypes for the reproducible example # otherwise assert_frame_equal flags different dtypes (int vs. float) df1 = pd.DataFrame({"A": [1,1,1], "B": [2,2,2], "C": [3,3,3]}, dtype=float) df2 = pd.DataFrame({"A": [1,1,1], "B": [2,2.2,2], "C": [3,3,3]}, dtype=float) assert_frame_equal(df1, df2, atol=0.5)