eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Given an array of values, I want to select multiple sequences of consecutive elements that satisfy a condition. The result should be one array for each sequence of elements. For example I have an array containing both negative and positive numbers. I need to select sequences of negative numbers, with each sequence in a separate array. Here is an example : import numpy as np # Example data values = np.array([1, 2, 3, -1, -2, -3, 4, 5, 6, -7, -8, 10]) mask = values < 0  Here is how the output should look like : Array 1: [-1 -2 -3] Array 2: [-7 -8]  I tried to do it using numpy.split, but it became more like spaghetti code. I was wondering is there a Pythonic way to do this task?

Answer: clustering groups of negative numbers If you only want to group the chunks of negative values, irrespective of their relative values, then simply compute a second mask to identify the starts of each negative chunk: mask = values < 0 mask2 = np.r_[True, np.diff(mask)] out = np.array_split(values[mask], np.nonzero(mask2[mask])[0][1:])  Output: [array([-1, -7, -3]), array([-7, -8])] clustering groups of negative numbers if they are successive in value If you want to cluster the negative values that also a successively decreasing (e.g. -1, -2, -3, -5, -6 would form 2 clusters: -1, -2, -3 and -5, -6. Then I would use pandas:  convert to Series identify the negative values create a grouper for consecutive negative values ((~mask).cumsum()) add the index (or a range) to group the successive groupby  import pandas as pd s = pd.Series(values) # mask to keep negative values mask = s<0 # group consecutive negatives group1 = (~mask).cumsum() # group successive decrementing values s2 = s+s.index group2 = s2.ne(s2.shift()).cumsum() out = [g.to_numpy() for k, g in s[mask].groupby([group1, group2])]  Output: [array([-1, -2, -3]), array([-7, -8]), array([-7])] Intermediates:  s mask s2 group1 group2 0 1 False 1 1 1 1 2 False 3 2 2 2 3 False 5 3 3 3 -1 True 2 3 4 # out 1 4 -2 True 2 3 4 # 5 -3 True 2 3 4 # 6 4 False 10 4 5 7 5 False 12 5 6 8 6 False 14 6 7 9 -7 True 2 6 8 # out 2 10 -8 True 2 6 8 # 11 -7 True 4 6 9 # out 3 12 10 False 22 7 10
Question: In Sympy I want to get the real part of an expression, but unable to separate the real part. The steps I have done: from sympy import * a, b = symbols('a b') z1 = a + b*I z2 = simplify(expand(z1**3))  The z2 output is: a**3 + 3*I*a**2*b - 3*a*b**2 - I*b**3  When I try to get the real part, I get this: >>> re(z2) re(a)**3 - 3*re(a)*im(a)**2 + 3*re(b)**2*im(b) - 3*re(a*b**2) - im(b)**3 - 3*im(a**2*b)  I expected a**3 - 3*a*b**2.

Answer: Sympy doesn't realize that you want a and b to be real variables, so its output assumes they are complex. If a and b are real, then terms containing im(a) and/or im(b) are 0 since there are no imaginary parts to a or b. To fix this, you should define your variables with your assumptions, i.e. tell sympy that a and b are real. Then the result will be as expected. from sympy import * a, b = symbols("a b", real=True) z1 = a + b*I z2 = simplify(expand(z1**3)) print(re(z2))  Output: a**3 - 3*a*b**2
Question: Below is my outlier detection code in pandas. I am doing rolling over window of 15, what I want is to do over window of 5 where this window is based on the day of week of the centered date i.e. if the centre is Monday take 2 backwards monday and 2 forward monday. Rolling doesn't have any support for this. How to do? import pandas as pd import numpy as np np.random.seed(0) dates = pd.date_range(start='2022-01-01', end='2023-12-31', freq='D') prices1 = np.random.randint(10, 100, size=len(dates)) prices2 = np.random.randint(20, 120, size=len(dates)).astype(float) data = {'Date': dates, 'Price1': prices1, 'Price2': prices2} df = pd.DataFrame(data) r = df.Price1.rolling(window=15, center=True) price_up, price_low = r.mean() + 2 * r.std(), r.mean() - 2 * r.std() mask_upper = df['Price1'] > price_up mask_lower = df['Price1'] < price_low df.loc[mask_upper, 'Price1'] = r.mean() df.loc[mask_lower, 'Price1'] = r.mean() 

Answer: One option using a groupby.rolling and the dayofweek as grouper to ensure only using the identical days in the rolling: r = (df.set_index('Date') .groupby(df['Date'].dt.dayofweek.values) # avoid index alignment .rolling(f'{5*7}D', center=True) ['Price1'] ) avg = r.mean().set_axis(df.index) # restore correct index std = r.std().set_axis(df.index) price_up, price_low = avg + 2 * std, avg - 2 * std mask_upper = df['Price1'] > price_up mask_lower = df['Price1'] < price_low df.loc[mask_upper, 'Price1'] = avg df.loc[mask_lower, 'Price1'] = avg  Example output:  Date Price1 Price2 0 2022-01-01 54.0 86.0 1 2022-01-02 57.0 117.0 2 2022-01-03 74.0 32.0 3 2022-01-04 77.0 35.0 4 2022-01-05 77.0 53.0 .. ... ... ... 725 2023-12-27 44.0 37.0 726 2023-12-28 60.0 65.0 727 2023-12-29 30.0 116.0 728 2023-12-30 53.0 82.0 729 2023-12-31 10.0 42.0 [730 rows x 3 columns]
Question: I am trying to find the value for an inlet flow rate that keeps a gravity drained tank at a required level. After the solution is found using IMODE 3, I want to simulate the system at this state for a period of time using IMODE 4. Below is the code I am using: from gekko import GEKKO import numpy as np m = GEKKO(remote = False) F_in = m.MV(value = 1, name= 'F_in') level = m.CV(value = 1, name = 'level') F_out = m.Var(value = 1, name = 'F_out') level.STATUS = 1 level.SPHI = 6 level.SPLO = 4 m.Equation(level.dt() == (F_in - F_out)/5) m.Equation(F_out == 2*(level)**0.5) # Find the steady state. F_in.STATUS = 1 m.options.IMODE = 3 m.options.SOLVER = 3 m.solve(disp = False) print(F_in.value) # Run at steady state for 5 time units. F_in.STATUS = 0 # Turn STATUS off for 0 DOF. m.time = np.linspace(0,5,10) m.options.IMODE = 4 m.solve(disp = False) print(level.value)  However, the code produces the error stating that IMODE 4 requires 0 DOF. Does changing the F_in status to 0 not result in 0 DOF? Below is the full error message: --------------------------------------------------------------------------- Exception Traceback (most recent call last) Cell In[4], line 27 25 m.time = np.linspace(0,5,10) 26 m.options.IMODE = 4 ---> 27 m.solve(disp = False) 28 print(level.value) File ~\VRMEng\VRMEng\Lib\site-packages\gekko\gekko.py:2140, in GEKKO.solve(self, disp, debug, GUI, **kwargs) 2138 print("Error:", errs) 2139 if (debug >= 1) and record_error: -> 2140 raise Exception(apm_error) 2142 else: #solve on APM server 2143 def send_if_exists(extension): Exception: @error: Degrees of Freedom * Error: DOF must be zero for this mode STOPPING...  Any help will be appreciated greatly!

Answer: Set m.options.SPECS = 0 to ignore the calculated / fixed specifications when changing modes (see documentation). Here is a complete script that solves successfully. from gekko import GEKKO import numpy as np m = GEKKO(remote = False) F_in = m.MV(value = 1, name= 'F_in') level = m.CV(value = 1, name = 'level') F_out = m.Var(value = 1, name = 'F_out') level.STATUS = 1 level.SPHI = 6 level.SPLO = 4 m.Equation(level.dt() == (F_in - F_out)/5) m.Equation(F_out == 2*(level)**0.5) # Find the steady state. F_in.STATUS = 1 m.options.IMODE = 3 m.options.SOLVER = 3 m.solve(disp = False) print(level.value) print(F_in.value) print(F_out.value) # Run at steady state for 5 time units. m.options.SPECS = 0 F_in.value = F_in.value.value F_in.STATUS = 0 # Turn STATUS off for 0 DOF. m.time = np.linspace(0,5,10) m.options.IMODE = 4 m.solve(disp = True) print(level.value) print(F_in.value) print(F_out.value)  A few more details: The t0 file in the run directory m._path stores whether a variable should be calculated or fixed. The rto.t0 file is generated with IMODE=3 and stores the steady-state values and the fixed/calculated information. Setting SPECS=0 ignores the specifications from the t0 file and uses the default specifications. For IMODE=4 the STATUS for each of the FVs and MVs are over-written to turn off. Set F_in.value = F_in.value.value to retain the F_in value from the steady state simulation.
Question: I am trying to install pytorch using either of below command and I got a lot of error. I am using windows CPU only. conda install pytorch::pytorch or conda install pytorch torchvision torchaudio cpuonly -c pytorch  some of the error are CondaVerificationError: The package for pytorch located at C:\Users\test\miniconda3\pkgs\pytorch-2.2.1-py3.10_cpu_0 appears to be corrupted. The path 'Lib/site-packages/torchgen/static_runtime/gen_static_runtime_ops.py' specified in the package manifest cannot be found. CondaVerificationError: The package for pytorch located at C:\Users\test\miniconda3\pkgs\pytorch-2.2.1-py3.10_cpu_0 appears to be corrupted. The path 'Lib/site-packages/torchgen/static_runtime/generator.py' specified in the package manifest cannot be found. CondaVerificationError: The package for pytorch located at C:\Users\test\miniconda3\pkgs\pytorch-2.2.1-py3.10_cpu_0 appears to be corrupted. The path 'Lib/site-packages/torchgen/utils.py' specified in the package manifest cannot be found. CondaVerificationError: The package for pytorch located at C:\Users\test\miniconda3\pkgs\pytorch-2.2.1-py3.10_cpu_0 appears to be corrupted. The path 'Lib/site-packages/torchgen/yaml_utils.py' specified in the package manifest cannot be found. CondaVerificationError: The package for pytorch located at C:\Users\test\miniconda3\pkgs\pytorch-2.2.1-py3.10_cpu_0 appears to be corrupted. The path 'Scripts/convert-caffe2-to-onnx-script.py' specified in the package manifest cannot be found. CondaVerificationError: The package for pytorch located at C:\Users\test\miniconda3\pkgs\pytorch-2.2.1-py3.10_cpu_0 appears to be corrupted. The path 'Scripts/convert-onnx-to-caffe2-script.py' specified in the package manifest cannot be found. CondaVerificationError: The package for pytorch located at C:\Users\test\miniconda3\pkgs\pytorch-2.2.1-py3.10_cpu_0 appears to be corrupted. The path 'Scripts/torchrun-script.py' specified in the package manifest cannot be found. 

Answer: One of your packages seems to be corrupt. You can try cleaning the packages from cache and reinstalling pytorch: conda clean -p
Question: This is my DataFrame: import pandas as pd import numpy as np df = pd.DataFrame( { 'a': [2, 2, 2, -4, np.nan, np.nan, 4, -3, 2, -2, -6], 'b': [2, 2, 2, 4, 4, 4, 4, 3, 2, 2, 6] } )  I want to add a plus sign for positive numbers only for column a when exporting to Excel. For example 1 becomes +1. Note that I have NaN values as well. I want them to be empty cells in Excel similar to the default behavior of Pandas when dealing with NaN values in to_excel. I have tried many solutions. This is one of them. But it didn't work in Excel. df.style.format({'a': '{:+g}'}).to_excel(r'df.xlsx', sheet_name='xx', index=False) 

Answer: What about using a callable: (df.style.format({'a': lambda x: f'{x:+g}' if np.isfinite(x) else ''}) .to_excel(r'/tmp/df.xlsx', sheet_name='xx', index=False) )  Alternative callable: lambda x: '' if np.isnan(x) else f'{x:+g}' Output (in jupyter):
Question: Suppose my data looks like this: data = { 'value': [1,9,6,7,3, 2,4,5,1,9] }  For each row, I would like to find the row number of the latest previous element larger than the current one. So, my expected output is: [None, 0, 1, 2, 1, 1, 3, 4, 1, 0]   the first element 1 has no previous element, so I want None in the result the next element 9 is at least as large than all its previous elements, so I want 0 in the result the next element 6, has its previous element 9 which is larger than it. The distance between them is 1. So, I want 1 in the result here.  I'm aware that I can do this in a loop in Python (or in C / Rust if I write an extension). My question: is it possible to solve this using entirely dataframe operations? pandas or Polars, either is fine. But only dataframe operations. So, none of the following please:  apply map_elements map_rows iter_rows Python for loops which loop over the rows and extract elements one-by-one from the dataframes 

Answer: This iterates only on the range of rows that this should look. It doesn't loop over the rows themselves in python. If your initial bound_range covers all the cases then it won't ever actually do a loop. lb=0 bound_range=3 df=df.with_columns(z=pl.lit(None, dtype=pl.UInt64)) while True: df=df.with_columns( z=pl.when(pl.col('value')>=pl.col('value').shift(1).cum_max()) .then(pl.lit(0, dtype=pl.UInt64)) .when(pl.col('z').is_null()) .then( pl.coalesce( pl.when(pl.col('value')<pl.col('value').shift(x)) .then(pl.lit(x, dtype=pl.UInt64)) for x in range(lb, lb+bound_range) ) ) .otherwise(pl.col('z')) ) if df[1:]['z'].drop_nulls().shape[0]==df.shape[0]-1: break lb+=bound_range  For this example I set bound_range to 3 to make sure it loops at least once. I ran this with 1M random integers between 0 and 9(inclusive) and I set the bound_range to 50 and it took under 2 sec. You could make this smarter in between loops by checking things more explicitly but the best approach there would be data dependent.
Question: A basic use of value_counts in a DataFrame is to obtain the count of a specific value. If I have a df such as: DataFrame({"color": ["red", "blue", "red", "green", "blue", "blue"]})  then if I want the count for color = 'red' in Pandas I could simply use: df['color'].value_counts()['red']  which is clear and obvious. In Polars, value_counts() produces a DF with a column of struct values: ┌─────────────┐ │ color │ │ --- │ │ struct[2] │ ╞═════════════╡ │ {"green",1} │ │ {"blue",3} │ │ {"red",2} │ └─────────────┘  which could be split into a DF with separate columns using counts = df.select(pl.col("color").value_counts()).unnest('color')  and then the required value can be obtained using counts.select(pl.col('count').filter(pl.col('color') == 'red')).item()  similarly group_by('color').len() could be used instead of value_counts This all seems rather complicated for such a frequent requirement. Is there a simpler way of extracting a single count value using Polars and more generally to access struct values.

Answer: If you don't want to use unnest, you can do df.select(pl.col("color").value_counts()).filter( pl.col("color").struct["color"] == "red" ).item()["count"]  which gives 2  There's tradeoffs - not having an Index opens up more doors for scalability, but admittedly some operations to become more verbose
Question: My understanding is that SQLModel uses pydantic BaseModel (checked the _mro_. Why is it that the bellow fails the type comparison. (err provided). class SomeModel(SQLModel,table=True): timestamp: datetime id: UUID = Field(default_factory=lambda: str(uuid4()), primary_key=True) def test_some_model(): m=SomeModel(**{'timestamp':datetime.utcnow().isoformat()}) assert type(m.timestamp)==datetime  E AssertionError: assert <class 'str'> == datetime FastAPI/SQLModel experts explain yourselves :D . Note - I tried using Field with default factories etc... as well.

Answer: Adding validate_assignment fixes the issue. Picked this solution up from https://github.com/tiangolo/sqlmodel/issues/52 after more than a couple days of breaking my head. Hopefully this helps other poor souls.  class SomeModel(SQLModel,table=True): class Config: validate_assignment = True timestamp: datetime age:int id: UUID = Field(default_factory=lambda: str(uuid4()), primary_key=True)
Question: I'm upgrading some code from python2 to python3 and the modern pandas version (I now have pandas 2.0.3 and numpy version 1.26.4) My dataframe is :  N NE E SE S SW W NW H12 NaN NaN NaN NaN NaN NaN NaN NaN H13 0.7 NaN NaN NaN NaN NaN 1.0 1.4 H14 0.3 NaN NaN NaN NaN NaN 0.8 1.1 H15 NaN NaN NaN NaN NaN NaN NaN NaN H16 NaN NaN NaN NaN NaN NaN NaN NaN  And I want to leave out all the NaNs such that i get a new df:  N W NW H13 0.7 1.0 1.4 H14 0.3 0.8 1.1  My old code has df.any(1) or something very similar used working, but now I get and error message NDFrame._add_numeric_operations.<locals>.any() takes 1 positional argument but 2 were given Maybe there is a better way to do it, I'm not fussed about using any().

Answer: any is now keyword-only, you have to use axis=1. Although you have not shown your original code, the following should work: out = df.loc[df.any(axis=1), df.any()]  Output:  N W NW H13 0.7 1.0 1.4 H14 0.3 0.8 1.1
Question: I have a DataFrame like this: import pandas as pd df = pd.DataFrame({ 'minutes':[1,2,3,4,5,6,7,8,9,10], 'score1': [0,0,1,1,0,0,0,1,1,2], 'score2': [0,1,1,1,1,2,1,1,2,2], 'sum_score': [0,1,2,2,1,2,1,2,3,4] })  Output:  minutes score1 score2 sum_score 0 1 0 0 0 1 2 0 1 1 2 3 1 1 2 3 4 1 1 2 4 5 0 1 1 5 6 0 2 2 6 7 0 1 1 7 8 1 1 2 8 9 1 2 3 9 10 2 2 4  The columns score1 and score2 are not sequentially increasing. There are rows in which its values decrease. When loop through each row, I would like to remove all previous rows where values are higher than the current one considering columns score1 and score2. I tried to apply masks: m1 = (df['score1'] > df['score1'].shift(-1)) m2 = (df['score2'] > df['score2'].shift(-1)) df = df.loc[~(m1 & m2)]  But it does not work as expected. I expect to get this: Output:  minutes score1 score2 sum_score 0 1 0 0 0 1 2 0 1 1 4 5 0 1 1 6 7 0 1 1 7 8 1 1 2 8 9 1 2 3 9 10 2 2 4 

Answer: Try: x = df[["score1", "score2"]][::-1].cummin()[::-1] mask = (df.score1 == x.score1) & (df.score2 == x.score2) print(df[mask])  Prints:  minutes score1 score2 sum_score 0 1 0 0 0 1 2 0 1 1 4 5 0 1 1 6 7 0 1 1 7 8 1 1 2 8 9 1 2 3 9 10 2 2 4
Question: I'm trying to count the number of non-overlapping pairs given a list of intervals. For example: [(1, 8), (7, 9), (3, 10), (7, 12), (11, 13), (13, 14), (9, 15)]  There are 8 pairs: ((1, 8), (11, 13)) ((1, 8), (13, 14)) ((1, 8), (9, 15)) ((7, 9), (11, 13)) ((7, 9), (13, 14)) ((3, 10), (11, 13)) ((3, 10), (13, 14)) ((7, 12), (13, 14))  I can't seem to figure out a better solution other than to just brute force it by comparing everything with virtually everything else, resulting in a O(n^2) solution. def count_non_overlapping_pairs(intervals): intervals = list(set(intervals)) # deduplicate any intervals intervals.sort(key=lambda x: x[1]) pairs = 0 for i in range(len(intervals)): for j in range(i+1, len(intervals)): if intervals[i][1] < intervals[j][0]: pairs += 1 return pairs  Is there a more optimal solution than this?

Answer: Sort the intervals, once by start point, once by end point. Now, given an interval, perform a binary search using the start point in the intervals sorted by end point. The index you get tells you how many non-overlapping intervals come before: All intervals that end before your interval starts are non-overlapping. Do the same for the end point: Do a binary search in the array of intervals sorted by start point. All intervals that start after your interval ends are non-overlapping. All other intervals either start before your interval ends, but after it has started, or end after your interval starts, but start before it. Do this for every interval, sum the results. Make sure to halve, to not count intervals twice. This looks as follows: Overall you get O(n log n): Two sorts, O(n) times two O(log n) binary searches. Now observe that half of this is not even needed - if A and B are two non-overlapping intervals, it suffices if B counts the intervals before it, including A; A doesn't need to count the intervals after it. This lets us simplify the solution further; you just need to sort the end points to be able to count the intervals before an interval, and of course we now don't need to halve the resulting sum anymore: # Counting the intervals before suffices def count_non_overlapping_pairs(intervals): ends = sorted(interval[1] for interval in intervals) def count_before(interval): return bisect_left(ends, interval[0]) return sum(map(count_before, intervals))  (Symmetrically, you could also just count the intervals after an interval.)
Question: I was exploring the random module and can't find a correct answer to whether b is inclusive or exclusive in random.uniform(a, b). In a code like random.uniform(0, 1), some answers say 1 is included while others say 1 is never produced. What's the correct answer?

Answer: The documentation for random.uniform(a, b) suggests that you may encounter cases where the upper limit b is included in the range due to how floating-point numbers are represented, but it's not guaranteed:  Return a random floating point number N such that a <= N <= b for a <= b and b <= N <= a for b < a. The end-point value b may or may not be included in the range depending on floating-point rounding in the equation a + (b-a) * random().  So sometimes the upper limit b may be included, and sometimes it may not be.
Question: I am learning python after having some java in school back in the day. If there are 2 lists, one with single digit integers and one with objects with a bunch of attributes, they appear to be a different speed to step through them? I was under the understanding that the objects are a memory pointer. Example: int_list = [1] * 1000 obj_list = [CustObject(0,1,2,3)] * 1000 total = 0 for i in int_list: total += i print(total) total = 0 for o in obj_list: total += o.int_variable print(total)  I ran this with some tile objects that have about 20 attributes, one of them being an image. Obviously the tile list creation takes a lot more time, which is expected. But stepping through them is slower as seen from the output: print("starting test") ints = [[1 for i in range(100)] for j in range(80)] print("created ints") tiles = [[Tile(i, j, 30, 1) for i in range(100)] for j in range(80)] print("created tiles") # step through the ints start_time = time.perf_counter() total = 0 for i in range(0, len(ints)): for j in range(0, len(ints[i])): total += ints[i][j] end_time = time.perf_counter() elapsed_time = end_time - start_time print("Ints Elapsed time: ", elapsed_time, total) # step through the Tiles start_time = time.perf_counter() total = 0 for i in range(0, len(tiles)): for j in range(0, len(tiles[i])): total += tiles[i][j].tile_type end_time = time.perf_counter() elapsed_time = end_time - start_time print("Tiles Elapsed time: ", elapsed_time, total)  Output starting test created ints created tiles Ints Elapsed time: 0.0011542000574991107 8000 Tiles Elapsed time: 0.002249700017273426 8000 

Answer: Accessing ints[i][j] needs:  lookup for ints variable lookup for i & j variables array boundary check  Accessing tiles[i][j].tile_type needs the above plus name lookup for tile_type, so it's slower. Anyway, instead of trying to figure out why all this is slower, you can optimize this code a lot by forgetting about indices. This code: for i in range(0, len(tiles)): for j in range(0, len(tiles[i])): total += tiles[i][j].tile_type  becomes (after directly iterating on the objects, not the indices) for ta in tiles: for e in ta: total += e.tile_type  or even faster probably with sum and double flat comprehension: total = sum(e.tile_type for ta in tiles for e in ta)