eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I've finished this tutorial about async list comprehension. Now I want to try an async dict comprehension. This is the code to replicate: import asyncio async def div7_tuple(x): return x, x/7 async def main(): lost = [4, 8, 15, 16, 23, 42] awaitables = asyncio.gather(*[div7_tuple(x) for x in lost]) print({k: v for k, v in awaitables}) if __name__ == '__main__': asyncio.run(main())  However, this results in an exception: > RuntimeError: await wasn't used with future > sys:1: RuntimeWarning:coroutine 'div7_tuple' was never awaited  How to do this with asyncio.gather()? It's weird this does not work in unsorted way for contruct an unsorted object, because it works if I try in a sorted way: async def div7(x): return x/7 async def main2(): lost = [4, 8, 15, 16, 23, 42] print({k: await v for k, v in [(x, div7(x)) for x in lost]}) 

Answer: gather() gives you a Future object back(this is that Future object that the error message says - await wasn't used with future). If you need the result of the object(in order to iterate over it) you need to await it first: async def main(): lost = [4, 8, 15, 16, 23, 42] awaitables = asyncio.gather(*[div7_tuple(x) for x in lost]) print({k: v for k, v in await awaitables})  or: async def main(): lost = [4, 8, 15, 16, 23, 42] awaitables = await asyncio.gather(*[div7_tuple(x) for x in lost]) print(dict(awaitables))   The relevant code in the source code is here:  def __await__(self): if not self.done(): self._asyncio_future_blocking = True yield self if not self.done(): raise RuntimeError("await wasn't used with future") return self.result() # May raise too. __iter__ = __await__ # make compatible with 'yield from'.  That __iter__ = __await__ is what makes it possible to iterate over a Future(and not to get a TypeError instead which says "TypeError: 'Future' object is not iterable") but since you didn't use await to make that Future done(It's result get set) it shows you that error message.
Question: I'm plotting a network using Dash-Cytoscape, using the breadthfirst layout, as documented here. I would like to control the order of appearance of elements on the same level. The JS API has the depthSort argument to achieve this, but I couldn't figure out how to pass a callback in Python that the frontend can live with. Things I've tried:  "depthSort": lambda a,b: a - b "depthSort": "(a,b) => a - b" "depthSort": "function (a,b) { return a - b}"  Minimal example: I would like to get this:  1 3 2  but what I'm getting is this:  1 2 3  from dash import Dash import dash_cytoscape as cyto app = Dash(__name__) app.layout = cyto.Cytoscape( elements=[ {"data": {"id": "1", "label": "1"}}, {"data": {"id": "2", "label": "2"}}, {"data": {"id": "3", "label": "3"}}, {"data": {"source": "1", "target": "2"}}, {"data": {"source": "1", "target": "3"}}, ], layout={ "name": "breadthfirst", "roots": ["1"], # "depthSort": ? }, ) app.run_server(debug=True) 

Answer: Options that expect a JS function are not supported in Python, because it implies passing the function as a string, and thus it would require Cytoscape.js to evaluate arbitrary strings, which is probably something the maintainers don't want for security reasons. That said, Dash supports clientside callbacks (JS) so we can still assign the function within a callback : from dash import Dash, Output, Input, State import dash_cytoscape as cyto app = Dash(__name__) app.layout = cyto.Cytoscape( id="cyto", elements=[ {"data": {"id": "1", "label": "1"}}, {"data": {"id": "2", "label": "2"}}, {"data": {"id": "3", "label": "3"}}, {"data": {"source": "1", "target": "2"}}, {"data": {"source": "1", "target": "3"}}, ], layout={ "name": "breadthfirst", "roots": ["1"] } ) app.clientside_callback( """ function (id, layout) { layout.depthSort = (a, b) => b.data('id') - a.data('id'); cy.layout(layout).run(); return layout; } """, Output('cyto', 'layout'), # update the (dash) cytoscape component's layout Input('cyto', 'id'), # trigger the function when the Cytoscape component loads (1) State('cyto', 'layout'), # grab the layout so we can update it in the function prevent_initial_call=False # ensure (1) (needed if True at the app level) ) app.run_server(debug=True)  NB. We need to execute cy.layout(layout).run() because no element are added/removed so it won't run automatically.
Question: I have a table:    Object Col1 Col2 Col3 Col4     reference 10 14 7 29   Obj1 0 9 1 30   Obj2 1 16 0 17   Obj3 9 21 3 0   Obj4 11 0 4 22    I want to transform it by condition: if any cell (except the cells of the 1st row) is =0, then it must be replaced with an incremented (X+1) value from the 1st row of this column. The resulting table is:    Object Col1 Col2 Col3 Col4     reference 10 14 7 29   Obj1 11 9 1 30   Obj2 1 16 8 17   Obj3 9 21 3 30   Obj4 11 15 4 22    I've tried this variant: df = np.where(df[df == 0] == 0, df.iloc[0] + 1, df)  but the result is ndarray, not DataFrame and the performance is not well enough. Is there a way to do this using only pandas' utils?

Answer: Use DataFrame.mask: out = df.mask(df == 0, df.iloc[0] + 1, axis=1) print (out) Col1 Col2 Col3 Col4 Object reference 10 14 7 29 Obj1 11 9 1 30 Obj2 1 16 8 17 Obj3 9 21 3 30 Obj4 11 15 4 22
Question: I have a Pandas's dataframe with 9 columns (called just dataframe), 5 of them called: [['DEG1','DEG2','DEG3','DEG4','DEG5']], it has almost 2000 rows. I have a list from which I build another dataframe with 5 columns called [['deg1','deg2','deg3','deg4','deg5']], this dataframe only has 1 row with the data series that needs to be checked in the other dataframe. I need to check if the entire series (the 5 columns) from the little dataframe (n_serie_df), if repeated in the big dataframe, here my actual state, which gives me:  "ValueError: Item wrong length 1 instead of 1709.".  This is my actual code: import pandas as pd import numpy as np def check_repeated_deg(n_serie_list, dataframe): n_serie_dict = { "DEG1": [n_serie_list[0]], "DEG2": [n_serie_list[1]], "DEG3": [n_serie_list[2]], "DEG4": [n_serie_list[3]], "DEG5": [n_serie_list[4]], } n_serie_df = pd.DataFrame(n_serie_dict) repeated = dataframe[np.all(n_serie_df.values == n_serie_df.values, 1)].any().any() if repeated: return f"This deg serie is already measured" else: return None dataframe = pd.read_csv(r"data_deg.csv") n_serie_list = [2, 11, 21, 27, 41] result = check_repeated_deg(n_serie_list, dataframe) print(result) 

Answer: Use: #create sample data - for test rewrite last row by n_serie_list values np.random.seed(56) dataframe = pd.DataFrame(np.random.randint(10, size=(5,9)), columns=['DEG1','DEG2','DEG3','DEG4','DEG5','a','b','c','d']) n_serie_list = [2, 11, 21, 27, 41] dataframe.iloc[-1, :5] = n_serie_list print (dataframe) DEG1 DEG2 DEG3 DEG4 DEG5 a b c d 0 5 4 0 2 9 7 6 4 9 1 7 1 8 2 0 5 6 1 9 2 5 5 2 9 3 5 9 2 1 3 0 4 6 2 0 8 6 4 0 4 2 11 21 27 41 2 1 3 8  Compare filtered rows of DataFrame by list n_serie_list with DataFrame.all and test at least one match by Series.any: cols = ['DEG1','DEG2','DEG3','DEG4','DEG5'] repeated = (dataframe[cols].values == n_serie_list).all(axis=1).any() print (repeated) True  How it working: print ((dataframe[cols].values == n_serie_list)) [[False False False False False] [False False False False False] [False False False False False] [False False False False False] [ True True True True True]]
Question: I want to create absolute maximum of all Polars data frame columns. Here is a way, but surely could be improved. import numpy as np import polars as pl df = pl.DataFrame({ "name": ["one", "one", "one", "two", "two", "two"], "val1": [1.2, -2.3, 3, -3.3, 2.2, -1.3], "val2": [1,2,3,-4,-3,-2] }) absVals = [] for col in df.columns: try: absVals.append((lambda arr: max(arr.min(), arr.max(), key=abs)) (df[col])) except: absVals.append(np.NaN) df_out= pl.DataFrame(data=absVals).transpose() df_out.columns=df.columns print(df_out)  Outputs - 

Answer: You can use import numpy as np import polars as pl df = pl.DataFrame({ "name": ["one", "one", "one", "two", "two", "two"], "val1": [1.2, -2.3, 3, -3.3, 2.2, -1.3], "val2": [1,2,3,-4,-3,-2] })  polars.DataFrame.schema schema has the Column name as well as the Column datatype. pl.NUMERIC_DTYPES contains all the numeric datatypes. for col,typ in df.schema.items(): if typ in pl.NUMERIC_DTYPES: print(max(df[col],key=abs)) else: print(np.NaN)  Using List comprehension in one line: [max(df[col],key=abs) if typ in pl.NUMERIC_DTYPES else np.NaN for col,typ in df.schema.items()] output #[nan, -3.3, -4]   Bonus: Your code without lambda: lst=[] for col in df.columns: try: lst.append(max(df[col],key=abs)) except: lst.append(np.NaN) print(lst) [nan, -3.3, -4]
Question: I am trying to make my code more elegant/less verbose with list comprehension. I have the following so far: a = [47, 67, 22] b = [26, 47, 12] at = 0 bt = 0 for a, b in [c for c in zip(a, b)]: if a>b: at+=1 elif a<b: bt+=1  I've been trying to do something like this to remove the if elif statements: [at+=1 if (a>b) else bt+=1 if (a<b) for a, b in [c for c in zip(a, b)]]  Which gives me a invalid syntax error highlighting the at+=1 (so I've gone down a rabbit hole: [at+=1 if (a>b) else bt+=1 if (a<b) else hello=None for a, b in [c for c in zip(a, b)]] ^^ SyntaxError: invalid syntax 

Answer: I'm not sure that list comprehension is more elegant (because it can be harder to read for people new to the language). However, you can write something like: a = [47, 67, 22] b = [26, 47, 12] at = sum(1 for x,y in zip(a,b) if x > y) bt = sum(1 for x,y in zip(a,b) if x < y)  This uses a generator expression to count all the pairs which are either truly greater or truly smaller. You are running into syntax errors because the trinary operator (bar if foo else baz) needs all three parts. It also must produce an output, so you can't use it to filter during a comprehension. If you wish to filter during a comprehension you need to use the (<expr> for <iterator> if <filter_expr>) variant of a comprehension which will skip elements for which the filter_expr evaluates to false. Edit: Deceze pointed out that this will iterate over the list twice. This is true and could be fixed by sacrificing readability: a = [47, 67, 22] b = [26, 47, 62] # changed the array for demo purposes result = sum(1 if x > y else 1j if x < y else 0 for x,y in zip(a,b)) at, bt = int(result.real), int(result.imag)  This solution is, however, specific to this particular example.
Question: I have 2 3D numpy arrays: a = np.array([[[1, 2, 3], [4, 5, 6]], [[7, 8, 9], [10, 11, 12]], [[13, 14, 15], [16, 17, 18]]]) b = a + 100  and 1 1D array: c = np.array([0, 1, 0])  I want to create another 3D array which elements are from a and b based on c, i.e. if c is 0 take from a, if 1 take from b. The result should be: array([[[ 1, 2, 3], [ 4, 5, 6]], [[107, 108, 109], [110, 111, 112]], [[ 13, 14, 15], [ 16, 17, 18]]])  As I want to use just numpy and nothing else, I was trying with np.where, but the result is not what I want: >>> np.where(c==0, a, b) array([[[ 1, 102, 3], [ 4, 105, 6]], [[ 7, 108, 9], [ 10, 111, 12]], [[ 13, 114, 15], [ 16, 117, 18]]])  Any suggestion?

Answer: You just need to up c to the desired dimension: np.where(c[:, None, None]==0, a, b)  [[[ 1 2 3] [ 4 5 6]] [[107 108 109] [110 111 112]] [[ 13 14 15] [ 16 17 18]]]
Question: I'm trying for a project to integrate acceleration data in order to have an approximation of the position. I used a real simple set of data to start with, with a constant acceleration. from scipy.integrate import cumtrapz t = [0, 0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45, 0.5, 0.55, 0.6, 0.65, 0.7] a = [-9.8, -9.8, -9.8, -9.8, -9.8, -9.8, -9.8, -9.8, -9.8, -9.8, -9.8, -9.8, -9.8, -9.8, -9.8] v = cumtrapz(a, t, initial=0) z = cumtrapz(v, t, initial=5)  The result is quite satisfying, apart from the fact that the initial condition for the position is only respected for the first value, and I don't understand how I can change this ? 

Answer: First of all, scipy.integrate.cumtrapz is left for backward compatibility, you should use the newer scipy.integrate.cumulative_trapezoid function. Secondly, if you read the documentation you'll see that initial is not an initial condition but simply a value prepended onto the array, which would normally be one element shorter than the original data.  initial : scalar, optional If given, insert this value at the beginning of the returned result. 0 or None are the only values accepted. Default is None, which means res has one element less than y along the axis of integration.  There is also a deprecation warning that, starting version 1.12.0, providing anything but 0 or None will result in a warning This shows the sizes of the result: from scipy.integrate import cumulative_trapezoid t = [0, 0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45, 0.5, 0.55, 0.6, 0.65, 0.7] a = [-9.8, -9.8, -9.8, -9.8, -9.8, -9.8, -9.8, -9.8, -9.8, -9.8, -9.8, -9.8, -9.8, -9.8, -9.8] print(len(t), len(cumulative_trapezoid(a, t)), len(cumulative_trapezoid(a, t, initial=0))) # 15, 14, 15  To enforce your initial condition, you should set initial=0 and then add the initial condition to the result. v0 = 0. z0 = 5. v = cumulative_trapezoid(a, t, initial=0) + v0 z = cumulative_trapezoid(v, t, initial=0) + z0
Question: We want to type hint a function def f(*args:float)->tuple[float,...]: ... return tuple(args)  such that it is specified that the number of elements in the tuple matches the number of args. Of course, the return here is a placeholder for more complicated logic. We would like to use mypy or pylance to check if we always return a) the correct number of elements and b) the correct tyoe of all elements. Using TypeVarTuple would allow to specify that we return the same number of elements, but not the type. Is there in current python (3.12) way to do it besides writing many overloads for 1-parameter, 2-parameter, 3-parameters etc?

Answer: Yes, you can write a no-op decorator to make the signature of f reject attempts at passing a type that you don't want. The following example makes f reject any attempts at passing values which aren't compatible with float. Demo: mypy Playground, Pyright Playground import typing_extensions as t if t.TYPE_CHECKING: import collections.abc as cx F = t.TypeVar("F", bound=cx.Callable[..., t.Any]) Ts = t.TypeVarTuple("Ts") class _FloatOnlyCallable(t.Protocol): def __call__(self, /, *args: float) -> t.Any: ... def asFloatOnlyCallable(f: F, /) -> F | _FloatOnlyCallable: """Decorate a function to make it only accept variadic positional float arguments""" return f @asFloatOnlyCallable def f(*args: *Ts) -> tuple[*Ts]: return args  >>> a, b = f(1.0, 2.0) # OK >>> c, d = f("3.0", 4.0) # Error: Incompatible type "str", expected "float" >>> e, g, h = f(5.0, 6.0) # Error: Need more values to unpack
Question: Suppose we have a sample dataframe like the one below: df = pd.DataFrame({'A': [np.nan, 0.5, 0.5, 0.5, 0.5], 'B': [np.nan, 3, 4, 1, 2], 'C': [10, np.nan, np.nan, np.nan, np.nan]}) >>> df A B C 0 NaN NaN 10.0 1 0.5 3.0 NaN 2 0.5 4.0 NaN 3 0.5 1.0 NaN 4 0.5 2.0 NaN  Col 'D' is calculated with the following operation: >>> df A B C D 0 NaN NaN 10.0 10.0 1 0.5 3.0 NaN 8.0 = (10 x 0.5) + 3 2 0.5 4.0 NaN 8.0 = (8 x 0.5) + 4 3 0.5 1.0 NaN 5.0 = (8 x 0.5) + 1 4 0.5 2.0 NaN 4.5 = (5 x 0.5) + 2  Calculating col 'D' reflects a folding process that recalls the previous row of col 'C' and current row of col 'A' and 'B' in each row operation. I've tried using for loops, functools.reduce() and iterators to do this, but I want to know if there's another method that uses vectorization as much as possible in order to make this operation more efficient in a larger dataset.

Answer: I'm not aware of pure vectorized pandas/numpy solution, but you can try to use numba to speed up the computation: from numba import njit @njit def calculate(A, B, starting_value=10): out = np.empty_like(A, dtype=np.float64) out[0] = starting_value for i, (a, b) in enumerate(zip(A[1:], B[1:]), 1): out[i] = (out[i - 1] * a) + b return out df["D"] = calculate(df["A"].values, df["B"].values, 10) print(df)  Prints:  A B C D 0 NaN NaN 10.0 10.0 1 0.5 3.0 NaN 8.0 2 0.5 4.0 NaN 8.0 3 0.5 1.0 NaN 5.0 4 0.5 2.0 NaN 4.5
Question: I have two simple code examples of the same calculation, which creates load on the CPU. The sequential one is significantly slower then the multi-threaded one. If I understand the concept of the GIL correctly, that should not be the case, as just one thread can run at the same time. So the overall time should be the same, as there is not wait time in I/O. What could be an explanation for this? I have another example with the multiprocessing library, which runs even faster then the multi-threaded one, which is expected, as it creates load on 3 cores in parallel. Sequential code - ~31 seconds: import time start_time = time.time() my_threads = [100000000, 200000000, 300000000] for x in my_threads: print(f"start {x}") for y in range(x): y*y print(f"finished input {x}: {time.time() - start_time:.2f} seconds") print(f"{time.time() - start_time:.2f} seconds")  Multi-threaded code - ~15 seconds execution time: EDIT: removed all unnecessary code, to make it simpler (does not change the outcome) import threading import time def my_threaded_function(s, x): with s: start_time = time.time() for y in range(x): y*y print(f"finished input {x}: {time.time() - start_time:.2f} seconds") s = threading.Semaphore(25) my_threads = [100000000, 200000000, 300000000] threads = [] start_time = time.time() for x in my_threads: t = threading.Thread( target=my_threaded_function, name=f"Thread-{x}", args=(s, x,) ) t.daemon = True threads.append(t) t.start() [thread.join() for thread in threads] print(f"{time.time() - start_time:.2f} seconds")  I expect the two examples to have the same execution time. EDIT: full output: ❯ python3 code/sequential.py finished input 100000000: 4.57 seconds finished input 200000000: 13.61 seconds finished input 300000000: 27.09 seconds 27.09 seconds ❯ python3 code/threading_example.py finished input 100000000: 7.84 seconds finished input 200000000: 12.44 seconds finished input 300000000: 14.65 seconds 14.68 seconds 

Answer: The reason is that in the sequential code, you have global variables for reading and writing but in the threaded code, the variables are local. Local variables are stored(STORE_FAST) and loaded(LOAD_FAST) much faster because the namespace of the functions are actually an array, instead of the dictionary and Python accesses them by index. Put all the code in the sequential code in a function and run that again. You'll see that they are almost the same. Having: for i in range(n): x = i * i  In the first line of the for loop, you store the returned number in i. In the second line, you look up i twice (i * i) and store x. These are all slower in global namespace.
Question: def permutation(str): #str = string input if len(str) == 0: return [""] result = [] for i, char in enumerate(str): for p in permutation(str[:i] + str[i + 1 :]): result.append(char + p) return result  I am confused in asymptotic analysis (time complexity) in this code does it have an O(n!) or O(n*n!) because the recursive call is in the loop. I am still confused does the big O notation only based on regular notation like logarithmic, linear, quadratic, exponential, factorial, etc. Or could beyond that like a combination not only based on the regular notation I already tried to calculate it and see some explanation in youtube but still couldn't understand well.

Answer: Let the time complexity of the function be T(n), where n is the length of the input string. For the case where n is not 0, we need to consider the following parts:  For the outer loop, it needs to be repeated n times.  For the outer loop body, the time complexity of each recursive call is T(n - 1). (I omitted the O(n) time complexity of constructing the input, considering that T(n - 1) is at least (n - 1)!, which should not affect the results.)  For inner loop, since the output size of the permutation is n!, it needs to be repeated (n - 1)! times (the input length for recursive calls is n - 1).  For the inner loop body, due to the length of p being n - 1, the time complexity of char + p is O(n).   In summary, we can conclude that: T(n) = n * (T(n - 1) + O(n) * (n - 1)!) = n * T(n - 1) + n * O(n!) = n * ((n - 1) * T(n - 2) + (n - 1) * O((n - 1)!)) + n * O(n!) = n * (n - 1) * T(n - 2) + (n - 1) * O(n!) + n * O(n!) ... = n! * T(0) + 1 * O(n!) + 2 * O(n!) + ... + n * O(n!) = O(n!) + (1 + 2 + ... + n) * O(n!) = O((1/2 * n**2 + 1/2 * n + 1) * n!) = O(n**2 * n!)
Question: I'm trying to compute the aggregate metrics (e.g. mean, median, sum) of column subsets in a numpy array. Take this array for example: 1 6 3 4 2 3 4 5 1 4 5 6 3 5 6 7  I have a set of clusters, which are lists of column indices like this: clusters = [[0, 1], [2], [3]]  Both the array and the list of cluster indices can be large and I can guarantee that each column in the array belongs to exactly one cluster, i.e. there are no duplicates in the clusters list. The indexes in the list aren't necessarily ordered, i.e. [[0, 3], [2, 1]] would also be a valid cluster. What I'm looking for is for example summing up the values per cluster - the result for the example above would look like this: [25, 18, 22]  A simple implementation in python could look like this: import numpy as np arr = np.array([ [1, 6, 3, 4], [2, 3, 4, 5], [1, 4, 5, 6], [3, 5, 6, 7], ]) clusters = [[0, 1], [2], [3]] result = np.array([arr[:,c_indices].sum() for c_indices in clusters]) # array([25, 18, 22])  My problem is that the matrix and number of clusters can grow quite large and I'd like to avoid looping in Python and instead keeping as much of this as possible in numpy's C-implementation for performance reasons. Are there more efficient ways of doing this? (Ideally compatible with min, max, median, mean, and sum)

Answer: A start would be to sum the columns in numpy and then only sum up the results: n = 10**4 arr = np.random.rand(n,n) clusters = [[random.randint(0,n-1) for _ in range(random.randint(0,n))] for _ in range(n//100)]  %%timeit arr_sumed = arr.sum(0) [arr_sumed[c_indices].sum() for c_indices in clusters]  38.6 ms ± 1.08 ms per loop (mean ± std. dev. of 7 runs, 10 loops each) %timeit [arr[:,c_indices].sum() for c_indices in clusters]  27.3 s ± 382 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) What if I want a median instead of a sum? Ok I hope somebody else will come up with a better solution BUT one thing you can do is sort the columns in advance. Then you can find the index of a number with adding up the indices in the sorted lists. Finally you do binary search for the median: n = 10**4 arr = np.random.rand(n,n) clusters = [np.array([random.randint(0,n-1) for i in range(random.randint(0,n))]) for j in range(n//100)]  %%time def median_from_sorted_arrays(arr_sorted, arr_min, arr_max, c_indices): n = len(arr_min)*len(c_indices) a = min(arr_min[c_indices]) b = max(arr_max[c_indices]) while True: k = sum(np.searchsorted(arr_sorted[i], (a+b)/2) for i in c_indices) if k/n < 0.499: a = (a+b)/2 elif k/n > 0.501: b = (a+b)/2 else: return (a+b)/2 arr_sorted = np.sort(arr, 0) arr_min = np.min(arr, 0) arr_max = np.max(arr, 0) [median_from_sorted_arrays(arr_sorted, arr_min, arr_max,c_indices) for c_indices in clusters]  This took 13.7 seconds compared to 1min 43s for the naive approach. Note that though the binary search should always work and give a perfect solution I didn't take the time to work out the correct stopping criterion making this median an approximation.