eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: As stated above, I need to efficiently count the number of distinct triplets of the form (a, b, b). In addition, the triplet is only valid if and only if it can be formed by deleting some integers from the array, only leaving behind that triplet in that specific ordering. What this is saying is that the triplets need to be in chronological order, I believe, but don't have to consist of consecutive elements. The solution needs to be really efficient as N (the length of the array) can go upto 10^6 (or a million). For example, if the array was [5, 6, 7, 3, 3, 3], then the answer would be 3 as the triplets would be: (5, 3, 3), (6, 3, 3), and (7, 3, 3). This was my first brute force (just to start off, O(n^3)): n = int(input()) arr = list(map(int, input().split())) ans = set() for i in range(n): for j in range(i + 1, n): if arr[i] != arr[j]: for k in range(j + 1, n): if arr[j] == arr[k]: ans.add((arr[i], arr[j], arr[k])) print(len(ans))  Then, I unsuccessfully tried optimizing this to an O(n^2), which is still too slow, but I can't even seem to get this right: def solve(): n = int(input()) arr = list(map(int, input().split())) freq = Counter(arr) ans = set() for a in freq: if freq[a] < 1: continue for b in freq: if b != a and freq[b] >= 2: ans.add((a, b, b)) return len(ans) print(solve())  I can't fix the logic for the O(n^2) and optimize this further to fully solve the problem under the given constraints. Assistance would be much appreciated.

Answer: At the second-to-last occurrence of each b-value, add the number of different values that came before it. Takes about 1.5 seconds for array length 10^6. from collections import Counter def linear(arr): ctr = Counter(arr) A = set() result = 0 for b in arr: if ctr[b] == 2: result += len(A) - (b in A) ctr[b] -= 1 A.add(b) return result  Testing your small example and five larger arrays: import random from time import time def original(arr): n = len(arr) ans = set() for i in range(n): for j in range(i + 1, n): if arr[i] != arr[j]: for k in range(j + 1, n): if arr[j] == arr[k]: ans.add((arr[i], arr[j], arr[k])) return len(ans) def test(arr): expect = original(arr) result = linear(arr) print(result == expect, expect, result) # Correctness test([5, 6, 7, 3, 3, 3]) for _ in range(5): test(random.choices(range(100), k=100)) # Speed n = 10**6 arr = random.choices(random.sample(range(10**9), n), k=n) t = time() print(linear(arr)) print(time() - t)  Sample output (Attempt This Online!): True 3 3 True 732 732 True 1038 1038 True 629 629 True 754 754 True 782 782 80414828386 1.4968228340148926
Question: Since python3.11 we are able to use builtin library tomllib before we had access to third party library tomli and a few others. I did not have analyzed both packages deeply but came to the conclusion, that I am able to replace tomli with tomllib for my purposes. My issue I have with the situation: How to handle the change while also supporting down to 3.9? Is there a solution to provide different dependencies in pyproject.toml for different versions of python (with additons to codebase changes accordingly)? This Question could be probably generalized, but I think the example gives a good picture of the general issue I face.

Answer: You can use PEP 496 – Environment Markers and PEP 508 – Dependency specification for Python Software Packages; they're usable in setup.py, setup.cfg, pyproject.toml, requirements.txt. In particular see PEP 631 – Dependency specification in pyproject.toml for pyproject.toml: [project] dependencies = [ 'tomli ; python_version < "3.11"' ]  After installing or skipping tomli you test presence of tomllib/tomli in the code: try import tomllib except ImportError: try import tomli except ImportError: sys.exit('Error: This program requires either tomllib or tomli but neither is available')
Question: Given the following dataFrame: pl.DataFrame({ 'A': ['a0', 'a0', 'a1', 'a1'], 'B': ['b1', 'b2', 'b1', 'b2'], 'x': [0, 10, 5, 1] })  I want to take value of column B with max value of column x within same value of A (taken from this question). I know there's solution with pl.Expr.get() and pl.Expr.arg_max(), but I wanted to use pl.Expr.top_k_by() instead, and for some reason it doesn't work for me with k = 1: df.with_columns( pl.col.B.top_k_by("x", 1).over("A").alias("y") )  ComputeError: the length of the window expression did not match that of the group Error originated in expression: 'col("B").top_k_by([dyn int: 1, col("x")]).over([col("A")])'  It does work for k = 2 though. Do you think it's a bug?

Answer: The error message produced when running your code without the window function gives is a bit more explicit and hints at a solution. df.with_columns( pl.col("B").top_k_by("x", 1) )  InvalidOperationError: Series B, length 1 doesn't match the DataFrame height of 4 If you want expression: col("B").top_k_by([dyn int: 1, col("x")]) to be broadcasted, ensure it is a scalar (for instance by adding '.first()').  Especially, pl.Expr.first can be used to allow for proper broadcasting here. df.with_columns( pl.col("B").top_k_by("x", 1).first().over("A").alias("y") )  shape: (4, 4) ┌─────┬─────┬─────┬─────┐ │ A ┆ B ┆ x ┆ y │ │ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ str ┆ i64 ┆ str │ ╞═════╪═════╪═════╪═════╡ │ a0 ┆ b1 ┆ 0 ┆ b2 │ │ a0 ┆ b2 ┆ 10 ┆ b2 │ │ a1 ┆ b1 ┆ 5 ┆ b1 │ │ a1 ┆ b2 ┆ 1 ┆ b1 │ └─────┴─────┴─────┴─────┘
Question: I have a large FastAPI application. There are many different endpoints, including one that is used to proxy video streams. The usage is something like this: the endpoint receives the video stream URL, opens it and returns it through streaming response. If I proxy 5 video streams, then everything is fine. If I proxy 6 streams, then my FastAPI app stops accepting any further requests, until I close one of the opened streams in the browser. I will need to proxy a much larger number of video streams, but the problem occurs already when proxying 6. How fix this? I am attaching a minimally reproducible example, but I cannot attach the links to the video streams themselves. I will note that there are no problems with the original streams, they work directly in any quantity. import uvicorn import httpx from fastapi import FastAPI from fastapi.responses import StreamingResponse app = FastAPI() async def proxy_mjpeg_stream(url: str): async with httpx.AsyncClient() as client: try: async with client.stream("GET", url) as stream: async for chunk in stream.aiter_bytes(): if chunk: yield chunk except httpx.ReadTimeout as ER: print(f'ERROR ReadTimeout - {url=} - {ER}') @app.get("/get_stream") async def get_stream(url: str): if url: headers = { "Content-Type": 'multipart/x-mixed-replace; boundary=frame', } return StreamingResponse(proxy_mjpeg_stream(url), headers=headers) if __name__ == "__main__": uvicorn.run(app) 

Answer: As you noted that you are testing the application through a web browser, you should be aware that every browser has a specific limit for parallel connections to a given hostname (as well as in general). That limit is hard coded—in Chrome and FireFox, for instance, that is 6—have a look here. So, if one opened 6 parallel connections to the same domain name through Chrome/FireFox, all subsequent requests would get stalled and queued, until a connection is available (that is, when the request is completed/a response is received). You could use a Python client using httpx, as demonstrated in this answer, in order to test your application instead. Also, as noted in the linked answer, when opening a new connection to the same endpoint, you should do that from a tab that is isolated from the browser's main session; otherwise, succeeding requests might be blocked by the browser (i.e., on client side), as the browser might be waiting for a response to the previous request from the server, before sending the next request (e.g., for caching purposes). Finally, please have a look at this answer on how to create a reusable HTTP client at application startup, instead of creating a new instance every time the endpoint is called.
Question: Is it possible to convert the following filter, which uses two conditions, to something that uses expression expansion or a custom function in order to apply the DRY priciple (avoid the repetition)? Here is the example: import polars as pl df = pl.DataFrame( { "a": [1, 2, 3, 4, 5], "val1": [1, None, 0, 0, None], "val2": [1, None, None, 0, 1], } ) df.filter((~pl.col("val1").is_in([None, 0])) | (~pl.col("val2").is_in([None, 0])))  Results in: ┌─────┬──────┬──────┐ │ a ┆ val1 ┆ val2 │ │ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ i64 │ ╞═════╪══════╪══════╡ │ 1 ┆ 1 ┆ 1 │ │ 5 ┆ null ┆ 1 │ └─────┴──────┴──────┘ 

Answer: .any_horizontal() and .all_horizontal() can be used to build | and & chains. .not_() can also be used instead of ~ if you prefer. df.filter( pl.any_horizontal( pl.col("val1", "val2").is_in([None, 0]).not_() ) )  shape: (2, 3) ┌─────┬──────┬──────┐ │ a ┆ val1 ┆ val2 │ │ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ i64 │ ╞═════╪══════╪══════╡ │ 1 ┆ 1 ┆ 1 │ │ 5 ┆ null ┆ 1 │ └─────┴──────┴──────┘
Question: Is there a way to use list of indices to simultaneously access the modules of nn.ModuleList in python? I am working with pytorch ModuleList as described below, decision_modules = nn.ModuleList([nn.Linear(768, 768) for i in range(10)])  Our input data is of the shape x=torch.rand(32,768). Here 32 is the batch size and 768 is the feature dimension. Now for each input data point in a minibatch of 32 datapoints, we want to select 4 decision modules from the list of decision_modules. The 4 decision engines from decision_engine are selected using an index list as explained below. I have a index matrix of dimensions ind. The ind matrix is of dimension torch.randint(0,10,(32,4)). I want to us a solution without use of loops as loops slows down the xecution significantly. But the following code throws and error. import torch import torch.nn as nn linears = nn.ModuleList([nn.Linear(768, 768) for i in range(10)]) ind=torch.randint(0,10,(32,4)) input=torch.rand(32,768) out=linears[ind](input)  The following error was observed  File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\nn\modules\container.py:334, in ModuleList.getitem(self, idx) 332 return self.class(list(self._modules.values())[idx]) 333 else: --> 334 return self._modules[self._get_abs_string_index(idx)] File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\nn\modules\container.py:314, in ModuleList._get_abs_string_index(self, idx) 312 def _get_abs_string_index(self, idx): 313 """Get the absolute index for the list of modules.""" --> 314 idx = operator.index(idx) 315 if not (-len(self) <= idx < len(self)): 316 raise IndexError(f"index {idx} is out of range") TypeError: only integer tensors of a single element can be converted to an index  The expected output shape is (32,4,768). Any help will be highly useful.

Answer: The ind tensor is of size (bs, n_decisions), which means we're choosing a different set of experts for each item in the batch. With this setup, the most efficient way to compute the output is to compute all experts for all batch items, then gather the desired choices after. This will be more performant in GPU compared to looping over the individual experts. Since we're looking at a linear layer, you can compute all the experts using a single linear layer of size n_experts * dim. d_in = 768 n_experts = 10 bs = 32 n_choice = 4 # Create a single large linear layer fused_linear = nn.Linear(d_in, d_in * n_experts) indices = torch.randint(0, n_experts, (bs, n_choice)) x = torch.randn(bs, d_in) # Forward pass through the fused layer y = fused_linear(x) # Shape: [bs, d_in * n_experts] # Reshape to separate the experts dimension ys = y.reshape(bs, n_experts, d_in) # Shape: [bs, n_experts, d_in] # Gather the chosen experts ys = torch.gather(ys, 1, indices.unsqueeze(-1).expand(-1, -1, d_in))  The output ys will be of shape (bs, n_choice, d_in)
Question: I have a dataset with 'tag-like' groupings:  Id tags 0 item1 ['friends','family'] 1 item2 ['friends'] 2 item3 [] 3 item4 ['family','holiday']  So a row can belong to several groups. I want to create an object similar to groupby, so that I can use agg etc. df.groupby('tags').count()  expected result  tags count 0 'friends' 2 1 'family' 2 2 'holiday' 1  But of course it won't work because it treats the whole list as the key, rather than the individual tags. Here's an attempt tagset = set(df.tags.explode()) grpby = { t: df.loc[df.tags.str.contains(t, regex=False)] for t in tagset }  From what I understand, groupby objects are structured a bit like this. But how to make it a groupby object? So that I can do things like grpby.year.mean() etc?

Answer: You can't have a row belong to multiple groups like your grpby object. Thus what you want to do is impossible in pure pandas, unless you duplicate the rows with explode, then you will be able to groupby.agg: out = (df.explode('tags') .groupby('tags', as_index=False) .agg(**{'count': ('tags', 'size')}) )  Output:  tags count 0 family 2 1 friends 2 2 holiday 1  With a more meaningful aggregation: out = (df.explode('tags') .groupby('tags', as_index=False) .agg({'Id': frozenset}) )  Output:  tags Id 0 family (item4, item1) 1 friends (item2, item1) 2 holiday (item4)  Note however that explode is quite expensive, so if you just want to count the tags, better use pure python: from collections import Counter from itertools import chain out = Counter(chain.from_iterable(df['tags']))  Output: Counter({'friends': 2, 'family': 2, 'holiday': 1})  And if you want to split the DataFrame like your grpby object: tmp = df.assign(group=df['tags']).explode('group') group = tmp.pop('group') out = dict(list(tmp.groupby(group)))  Output: {'family': Id tags 0 item1 [friends, family] 3 item4 [family, holiday], 'friends': Id tags 0 item1 [friends, family] 1 item2 [friends], 'holiday': Id tags 3 item4 [family, holiday]}
Question: I have sorted start indices (included) and end indices (excluded) of intervals (obtained by using seachsorted), for instance: import numpy as np # Both arrays are of same size, and sorted. # Size of arrays is number of intervals. # Intervals do not overlap. # interval indices: 0 1 2 3 4 5 interval_start_idxs = np.array([0, 3, 3, 3, 6, 7]) interval_end_excl_idxs = np.array([2, 4, 4, 4, 7, 9])  An empty interval is identified when interval_start_idxs[interval_idx] == interval_end_excl_idxs[interval_idx]-1  I would like to identify the starts and ends of each region where intervals are empty. A region is made with one or several intervals sharing the same start indices and end excluded indices. With previous data, expected result would then be: empty_interval_starts = [1, 4] # start is included empty_intervals_ends_excl = [4, 5] # end is excluded  This result is to be understood as:  intervals from index 1 to 3, these intervals are a same region of empty intervals and interval at index 4 is a separate region on its own 

Answer: import numpy as np interval_start_idxs = np.array([0, 3, 3, 3, 6, 7]) interval_end_excl_idxs = np.array([2, 4, 4, 4, 7, 9]) is_region_start = np.r_[True, np.diff(interval_start_idxs) != 0] is_region_end = np.roll(is_region_start, -1) is_empty = (interval_start_idxs == interval_end_excl_idxs - 1) empty_interval_starts = np.nonzero(is_region_start & is_empty)[0] empty_interval_ends_excl = np.nonzero(is_region_end & is_empty)[0] + 1  Explanation:  is_region_start marks the starts of all potential regions, i.e. indices where the current index differs from its predecessor the index of the end of a potential region is right before the start of a new region, which is why we roll back all markers in is_region_start by one to get is_region_end; the rollover in the roll-back from index 0 to index -1 works in our favor here: the marker, previously at index 0, which is always True, used to mark the start of the first potential region in is_region_start and now marks the end of the last potential region in is_region_end is_empty marks all indices that are actually empty, according to your definition empty_interval_starts is the combination of two criteria: start of a potential region and actually being empty (since np.nonzero() returns tuples, we need to extract the first element, …[0], to get to the actual array of indices) empty_interval_ends_excl, likewise, is the combination of two criteria: end of a potential region and actually being empty; however, since empty_interval_ends_excl should be exclusive, we need to add 1 to get the final result  At present, the results (empty_interval_starts and empty_interval_ends_excl) are Numpy arrays. If you prefer them as lists, as written in the question, you might want to convert them with empty_interval_starts.tolist() and empty_interval_ends_excl.tolist(), respectively.
Question: Given the following polars dataframe: pl.DataFrame({'A': ['a0', 'a0', 'a1', 'a1'], 'B': ['b1', 'b2', 'b1', 'b2'], 'x': [0, 10, 5, 1]}) shape: (4, 3) ┌─────┬─────┬─────┐ │ A ┆ B ┆ x │ │ --- ┆ --- ┆ --- │ │ str ┆ str ┆ i64 │ ╞═════╪═════╪═════╡ │ a0 ┆ b1 ┆ 0 │ │ a0 ┆ b2 ┆ 10 │ │ a1 ┆ b1 ┆ 5 │ │ a1 ┆ b2 ┆ 1 │ └─────┴─────┴─────┘  I want to add a column y which groups by A and selects the value from B with the maximum corresponding x. The following dataframe should be the result: ┌─────┬─────┬─────┬─────┐ │ A ┆ B ┆ x ┆ y │ │ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ str ┆ i64 ┆ str │ ╞═════╪═════╪═════╪═════╡ │ a0 ┆ b1 ┆ 0 ┆ b2 │ │ a0 ┆ b2 ┆ 10 ┆ b2 │ │ a1 ┆ b1 ┆ 5 ┆ b1 │ │ a1 ┆ b2 ┆ 1 ┆ b1 │ └─────┴─────┴─────┴─────┘  I've tried various versions of df.with_columns(y=pl.col('B').?.over('A')) without any luck.

Answer: You can use pl.Expr.get and pl.Expr.arg_max to obtain the value of B with maximum corresponding value of x. This can be combined with the window function pl.Expr.over to perform the operation separately for each group defined by A. df.with_columns( pl.col("B").get(pl.col("x").arg_max()).over("A").alias("y") )  shape: (4, 4) ┌─────┬─────┬─────┬─────┐ │ A ┆ B ┆ x ┆ y │ │ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ str ┆ i64 ┆ str │ ╞═════╪═════╪═════╪═════╡ │ a0 ┆ b1 ┆ 0 ┆ b2 │ │ a0 ┆ b2 ┆ 10 ┆ b2 │ │ a1 ┆ b1 ┆ 5 ┆ b1 │ │ a1 ┆ b2 ┆ 1 ┆ b1 │ └─────┴─────┴─────┴─────┘
Question: I have a pandas dataframe which looks like this: 1_2 1_3 1_4 2_3 2_4 3_4 1 5 2 8 2 2 4 3 4 5 8 5 8 8 8 9 3 3 4 3 4 4 8 3 8 0 7 4 2 2  where the columns are the 4C2 combinations of 1,2,3,4. And I would like to generate 4 new columns f_1, f_2, f_3, f_4 where the values of the columns are defined to be df['f_1'] = df['1_2']+df['1_3']+df['1_4'] df['f_2'] = df['1_2']+df['2_3']+df['2_4'] df['f_3'] = df['1_3']+df['2_3']+df['3_4'] df['f_4'] = df['1_4']+df['2_4']+df['3_4']  In other words, the column f_i are defined to be the sum of columns i_j and k_i. So I can brute force my way in this case. However, my original dataframe is a lot bigger and there are 20C2 = 190 columns instead and hence a brute force method wouldn't work. So the desired outcome looks like 1_2 1_3 1_4 2_3 2_4 3_4 f_1 f_2 f_3 f_4 1 5 2 8 2 2 8 11 15 6 4 3 4 5 8 5 11 17 13 17 8 8 8 9 3 3 24 20 20 14 4 3 4 4 8 3 11 16 10 15 8 0 7 4 2 2 15 14 6 11  Thank you so much.

Answer: Build a dictionary of the columns with str.split+explode+Index.groupby, and process them in a loop: s = df.columns.to_series().str.split('_').explode() d = s.index.groupby(s) for k, v in d.items(): df[f'f_{k}'] = df[v].sum(axis=1)  You could also use eval instead of the loop once you have d: query = '\n'.join(f'f_{k} = {"+".join(map("`{}`".format, v))}' for k,v in d.items()) out = df.eval(query)  Output:  1_2 1_3 1_4 2_3 2_4 3_4 f_1 f_2 f_3 f_4 0 1 5 2 8 2 2 8 11 15 6 1 4 3 4 5 8 5 11 17 13 17 2 8 8 8 9 3 3 24 20 20 14 3 4 3 4 4 8 3 11 16 10 15 4 8 0 7 4 2 2 15 14 6 11  Intermediate d: {'1': ['1_2', '1_3', '1_4'], '2': ['1_2', '2_3', '2_4'], '3': ['1_3', '2_3', '3_4'], '4': ['1_4', '2_4', '3_4'], }  Pure python approach to build d: d = {} for c in df: for k in c.split('_'): d.setdefault(k, []).append(c)  You could also imagine a pure pandas approach based on reshaping with melt+pivot_table, but this is most likely much less efficient: out = df.join(df .set_axis(df.columns.str.split('_'), axis=1) .melt(ignore_index=False).explode('variable') .reset_index() .pivot_table(index='index', columns='variable', values='value', aggfunc='sum') .add_prefix('f_') )
Question: I have a series with four categories A,B,C,D and their current value s1 = pd.Series({"A": 0.2, "B": 0.3, "C": 0.3, "D": 0.9}) And a threshold against which I need to compare the categories, threshold = {"custom": {"A, B": 0.6, "C": 0.3}, "default": 0.4} But the threshold has two categories summed together: A & B And it has a "default" threshold to apply to each category that hasn't been specifically named. I can't quite work out how to do this in a general way I can solve two separate sub problems, but not the whole problem.  I can solve the problem with single categories and a default threshold. Or, I can solve combined categories, but not apply the default threshold.  What I need to evaluate is: s1[A]+s1[B] < threshold["custom"]["A,B"] :: 0.2 + 0.3 < 0.6 s1[C] < threshold["custom"]["C"] :: 0.3 < 0.3 s1[D] < threshold["default"] :: 0.9 < 0.4  And return this Series: # A,B True # C False # D False  Here is what I've got for the subproblems 1. To apply the default threshold, I reindex and fillna with the default value: aligned_threshold = ( pd.Series(threshold.get("custom")) .reindex(s1.index) .fillna(threshold.get("default")) ) # A 0.4 # B 0.4 # C 0.3 # D 0.4  then I can compare: s1 < aligned_threshold # A True # B True # C False # D False # dtype: bool  2. To combine categories threshold_s = pd.Series(threshold.get("custom")) s1_combined = pd.Series(index=threshold_s.index) for category, threshold in threshold["custom"].items(): s1_combined[category] = sum([s1.get(k, 0) for k in category.split(", ")]) # now s1_combined is: # A,B 0.6 # C 0.3 s1_combined < threshold_s # A,B True # C False # dtype: bool  but I've lost category D To recap, what I need is: s1[A]+s1[B] s1[C] s1[D]  So that I can compare thus: s1 < threshold And return this Series: # A,B True # C False # D False 

Answer: You could build a mapper to rename, then groupby.sum and compare to the reference thresholds: mapper = {x: k for k in threshold['custom'] for x in k.split(', ')} # {'A': 'A, B', 'B': 'A, B', 'C': 'C'} s2 = (s1.rename(mapper) .groupby(level=0).sum() ) out = s2.lt(s2.index.to_series() .map(threshold['custom']) .fillna(threshold['default']) )  Alternative for the last step if you don't have NaNs: out = s2.lt(s2.index.map(threshold['custom']).values, fill_value=threshold['default'])  Output: A, B True C False D False dtype: bool