eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a dataframe, where I want to forward / backward fill, based on a boolean series, df['condition']. A single group consists of a streak of True values including the breaking False entry that separates a streak from the next one. What I mean is very clear when looking at my dataframes. My input looks like this: condition_values = [True, True, True, False, True, True, False, True, True, False, True, True, True] value_values = [0.1, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, 0.5, np.nan, 0.9, np.nan] data = {'condition': condition_values, 'value': value_values} df = pd.DataFrame(data) condition value 0 True 0.1 1 True NaN 2 True NaN 3 False NaN 4 True NaN 5 True NaN 6 False NaN 7 True NaN 8 True NaN 9 False 0.5 10 True NaN 11 True 0.9 12 True NaN  My desired output looks like this: condition_values = [True, True, True, False, True, True, False, True, True, False, True, True, True] value_values = [0.1,0.1, 0.1, 0.1, np.nan, np.nan, np.nan, 0.5, 0.5, 0.5, 0.9, 0.9, 0.9] data = {'condition': condition_values, 'value': value_values} df2 = pd.DataFrame(data) condition value 0 True 0.1 1 True 0.1 2 True 0.1 3 False 0.1 4 True NaN 5 True NaN 6 False NaN 7 True 0.5 8 True 0.5 9 False 0.5 10 True 0.9 11 True 0.9 12 True 0.9  I have tried to make a bunch of dataframes, split by the Falses, ffill and bfill, and then re-concatenated. There has to be a faster way. I am very willing to take hints instead of the full solution - I'm trying to figure this out one way or another.

Answer: You were quite close indeed! You can try as follows: Option 1: ffill().bfill() df['value'] = ( df.groupby( df['condition'].shift().eq(0).cumsum() .where(df['condition']).ffill() )['value'] .apply(lambda x: x.ffill().bfill()) .droplevel(0) )  Option 2: transform(first) If you only have one value per group (as in your example), this can be done even simpler: df['value'] = ( df.groupby( df['condition'].shift().eq(0).cumsum() .where(df['condition']).ffill() )['value'].transform('first') )  Output  condition value 0 True 0.1 1 True 0.1 2 True 0.1 3 False 0.1 4 True NaN 5 True NaN 6 False NaN 7 True 0.5 8 True 0.5 9 False 0.5 10 True 0.9 11 True 0.9 12 True 0.9  Explanation  Start with Series.shift for df['condition'] and check Series.eq to 0 (as you did).  Now, chain Series.where to "reset" False rows to NaN and apply Series.ffill, thus adding them to the previous group.  Pass the result to df.groupby, select column 'value'.  Option 1: use groupby.apply to apply Series.ffill and Series.bfill to each group. In this case, you need to get rid of the prepended index level with df.droplevel before assignment to df['value'].  Option 2: if you only have 1 value, you can simply use groupby.transform and get groupby.first. With default setting, this will return the first non-NaN value.
Question: I have a dict of lists such as here, though the inner data could be any level of complexity (perhaps strings, perhaps dicts, perhaps multi layer nested complex object). my_dict: my_list_a: - a - b - c my_list_b: - a - d - e list_c: - f   How do I flatten this structure / join the inner items into a list in the following format through jinja2 (preserving order is not important to me in my current use case but it's always nice when possible).  my_combined_list: - a - b - c - a - d - e - f  Unfortunately as list comprehensions are out of the question due to lack of support in Jinja, I haven't been able to piece a solution together so far. There are also some extra things I need to do, though most are easily doable on the final list (deduplicate, sort, etc). Something that needs to be done during the join is:  select or reject based on a test (e.g. only include lists that start with string: my_ meaning list_c is not added to the end result)  I've found many ways for list of dicts, or various more complex special cases, but I can't seem to find anything on making a list of all inner items of a dict of lists of items.

Answer: Q: Flatten the inner items into a list. A: The simplest option is my_combined_list: "{{ my_dict.values() | flatten }}"  gives my_combined_list: [a, b, c, a, d, e, f]  Q: Select or reject based on a test (start string: my_ , e.g. list_c is omitted). A: For example, use the test match allow_match: [my_] my_combined_list: "{{ my_dict | dict2items | selectattr('key', 'match', allow_match|join('|')) | map(attribute='value') | flatten }}"  gives my_combined_list: [a, b, c, a, d, e]    Example of a complete playbook for testing - hosts: localhost vars: my_dict: my_list_a: [a, b, c] my_list_b: [a, d, e] list_c: [f] my_combined_list: "{{ my_dict.values() | flatten }}" allow_match: [my_] my_combined_lis2: "{{ my_dict | dict2items | selectattr('key', 'match', allow_match|join('|')) | map(attribute='value') | flatten }}" tasks: - debug: var: my_combined_list | to_yaml - debug: var: my_combined_lis2 | to_yaml
Question: I have a dataframe that looks like this: Put/Call StrikePrice fixedprice floatprice fixedspread floatspread Put 10 0 20 0 0 Put 10 0 20 0 0 nan 0 0 0 13 15 nan 0 0 0 14 16  If the put/call column has the value 'Put', I need to take the value from the strike price column and place it in the fixedspread column, and I need to take the value from the float price column and place it in the float spread column. Once the values are in the correct places I can get rid of the Put/Call column, strike price column, float price column and fixed price column.  the output should look something like this:  fixedspread floatspread 10 20 10 20 13 15 14 16 

Answer: You can directly combine the columns with mask using the underlying numpy array for the right-hand side: out = (df[['fixedspread', 'floatspread']] .mask(df['Put/Call'].eq('Put'), df[['StrikePrice', 'floatprice']].values) )  Output:  fixedspread floatspread 0 10 20 1 10 20 2 13 15 3 14 16
Question: The following code works in numpy 1.26.4, but not in numpy 2.0.0rc1, giving  OverflowError: Python integer -256 out of bounds for int8  import numpy as np np.array([1], dtype=np.int8) + (-256)  Is that expected? Is it due to NEP 50 adoption? If so, which part of NEP 50 mandates this new behavior?

Answer: Yeah, that's NEP 50 behavior. Quoting NEP 50:  This proposal uses a “weak scalar” logic. This means that Python int, float, and complex are not assigned one of the typical dtypes, such as float64 or int64. Rather, they are assigned a special abstract DType, similar to the “scalar” hierarchy names: Integral, Floating, ComplexFloating. When promotion occurs (as it does for ufuncs if no exact loop matches), the other DType is able to decide how to regard the Python scalar. E.g. a UInt16 promoting with an Integral will give UInt16.   At no time is the value used to decide the result of this promotion. The value is only considered when it is converted to the new dtype; this may raise an error.  Before, the value of the Python int -256 would have been examined to determine what result dtype to use. Now, adding a Python int to data of int8 dtype always converts the Python int to int8 dtype. If the value doesn't fit in int8 dtype, you get this error.
Question: This is the code I took from the API docs in order to start multiturn chat model = genai.GenerativeModel("gemini-pro") messages = [{'role':'user', 'parts': ['hello']}] response = model.generate_content(messages) # "Hello, how can I help" messages.append(response.candidates[0].content) messages.append({'role':'user', 'parts': ['How does quantum physics work?']}) response = model.generate_content(messages)  But I'm getting the following error: TypeError: Could not create Blob, expected Blob, dict or an Image type(PIL.Image.Image or IPython.display.Image). Got a: <class 'google.ai.generativelanguage_v1beta.types.content.Content'> Value: parts { text: "hello" } I also tried using the send_message feature, but I got the same error after I tried to use send_message the second time.

Answer: As a simple modification, how about the following modification? Modified script 1: genai.configure(api_key=apiKey) model = genai.GenerativeModel("gemini-pro") messages = [{"role": "user", "parts": ["hello"]}] response = model.generate_content(messages) messages.append({"role": "model", "parts": [response.text]}) print("--- response 1") print(response.text) messages.append({"role": "user", "parts": ["How does quantum physics work?"]}) response = model.generate_content(messages) print("--- response 2") print(response.text)  Modified script 2: As another modification, how about the following modification? Ref import google.generativeai as genai apiKey = "###" # If you use API key, please set your API key. genai.configure(api_key=apiKey) model = genai.GenerativeModel("gemini-pro") chat = model.start_chat() def get_chat_response(chat: genai.ChatSession, prompt: str) -> str: response = chat.send_message(prompt) return response.text prompt = "hello" print(get_chat_response(chat, prompt)) prompt = "How does quantum physics work?" print(get_chat_response(chat, prompt))
Question: Sharing one sample code . For column b as u can see both values{1,2} has same frequency{2} so mode is returning both values but I want null . Which logically means that it is unable to find one unique value which is most occuring. import polars as pl if __name__ == '__main__': df = pl.DataFrame( { "a": [1, 1, 2, 3], "b": [1, 1, 2, 2],}) # print(df) print(df.select(pl.col("b").mode())) 

Answer: You could try using a pl.when().then() construct. df.select( pl.when( pl.col("b").mode().len() == 1 ).then( pl.col("b").mode() ) )  shape: (2, 1) ┌──────┐ │ b │ │ --- │ │ i64 │ ╞══════╡ │ null │ │ null │ └──────┘  If you prefer a single null, you can append .unique() to the closing parenthesis of .then().
Question: How do I show an equation like this:  I can show the left side using display(binomial(n,k))  and the right side using y= factorial(n)/(factorial(n-k) * factorial(k)) display(y)  but how do I show the equation? Also, how do I make it show (n-k) instead of (-k+n)?

Answer: Make it into an Equality. from sympy import init_session init_session(quiet=True) import sympy as smp n, k = smp.symbols("n k") eq = smp.Eq(smp.binomial(n, k), smp.factorial(n)/(smp.factorial(n-k) * smp.factorial(k))) eq   The variable ordering in displayed result is determined internally by sympy (I believe it is alphabetical). There is a way to adjust this ordering by writing your own printer methods, but it is arguably more work than necessary. So, unless you have a specific reason why -k + n is an issue, I recommend leaving it as it is (mathematically, it is equivalent).
Question: I'm trying to use new type hints from Python 3.12, and suddenly PyCharm highlights some obscure problem about new usage of ParamSpec. import functools from dataclasses import dataclass from typing import Callable from typing import Concatenate @dataclass class Message: text: str type CheckFunc[**P] = Callable[Concatenate[Message, P], None] def check_is_command_not_answer[**P](func: CheckFunc[P]) -> CheckFunc[P]: @functools.wraps(func) def inner(message: Message, *args: P.args, **kwargs: P.kwargs) -> None: if message.text: ... else: return func(message, *args, **kwargs) return inner  On return inner PyCharm blames:  Expected type '(Message, ParamSpec("P")) -> None', got '(message: Message, ParamSpec("P"), ParamSpec("P")) -> None' instead  Here is the screenshot. BTW, I'm using this decorator here and there, and the code works perfectly. Pyright doesn't complain for any issues here either. I cannot spot the problem. Can you?

Answer: Your code has no problem. The problem is with PyCharm. Its support for Concatenate et al. is incomplete, as discussed in PY-51766. I'm unable to find a good existing issue for this bug, however. Put a type: ignore or noqa comment there and move on.
Question: I have a function which runs in a loop performing calculations on a list of arrays. At a point during the first iteration of the function, a polars lazyframe is initialised. On the following iterations, a new dataframe is specified using the same schema, and the two dataframes are joined row-wise using pl.vstack, and then specified as a lazyframe again.  import numpy as np import polars as pl def my_func(): array_list = [np.zeros((1,19))]*2 #this is just for example and not representative of shape of real array. for i, _ in enumerate(array_list): #calculations are done here result = np.zeros((1,19)) #result of calculations (correct shape of real result) if i < 1: result_df = pl.DataFrame(data = result, schema = { 'MDL', 'MVL', 'MWVL', 'RR', 'DET', 'ADL', 'LDL', 'DIV', 'EDL', 'LAM', 'TT', 'LVL', 'EVL', 'AWVL', 'LWVL', 'LWVLI', 'EWVL', 'Ratio_DRR', 'Ratio_LD'}, orient='row').lazy() else: new_df = pl.DataFrame(data=data, schema = { 'MDL', 'MVL', 'MWVL', 'RR', 'DET', 'ADL', 'LDL', 'DIV', 'EDL', 'LAM', 'TT', 'LVL', 'EVL', 'AWVL', 'LWVL', 'LWVLI', 'EWVL', 'Ratio_DRR', 'Ratio_LD' }, orient='row') #append new dataframe to results result_df = result_df.collect().vstack(new_df, in_place=True).lazy() return result_df  When returning the dataframe outside the function, the column names are no longer in order, but the data is. e.g result.schema OrderedDict([('LAM', Float64), ('LDL', Float64), ('ADL', Float64), ('DIV', Float64), ('MDL', Float64), ('MWVL', Float64), ('LWVL', Float64), ('MVL', Float64), ('TT', Float64), ('DET', Float64), ('RR', Float64), ('EDL', Float64), ('Ratio_LD', Float64), ('Ratio_DRR', Float64), ('LVL', Float64), ('LWVLI', Float64), ('EWVL', Float64), ('EVL', Float64), ('AWVL', Float64)])  I assume this is due to my naivety about how lazyframes work, but is there a way to enfore the order without renaming the columns? Thanks.

Answer: The problem is you're using a set which is an unordered collection in Python. >>> schema = {'MDL', 'MVL', 'MWVL', 'RR', 'DET'} >>> type(schema) set >>> schema {'DET', 'MDL', 'MVL', 'MWVL', 'RR'} # "random" order  You want to use a list/tuple instead. There is a LazyFrame constructor, and you can use concat to combine frames (eager or lazy).  pl.LazyFrame()  pl.concat()   frames = [] for ...: df = pl.LazyFrame(...) frames.append(df) return pl.concat(frames)  The use of .vstack() and any in_place methods/parameters in Polars is generally not recommended.
Question: Trying to calculate in pandas the rolling sum of Values in Column A, with lookback periods given in Column B and results of rolling sums stored in Column C. Index | Column A | Column B || Column C | | -------- | -------- || -------- | 0 | 1 | 1 || 1 | 1 | 2 | 2 || 3 | 2 | 1 | 3 || 4 | 3 | 3 | 2 || 4 | 4 | 2 | 4 || 8 |  For example, for last row, the rolling sum should sum the last 4 values from Column A, since 4 is given in column B. Avoiding loops would be optimal. Although a simple task, I haven't managed to come up with a solution.

Answer: Since your rolling sums depend on all values, you will have to compute one per window. This can be done using numpy and indexing lookup: import numpy as np idx, vals = pd.factorize(df['B']) df['C'] = np.vstack([ df['A'].rolling(v, min_periods=1).sum() for v in vals ])[idx, np.arange(len(df))]  Output:  A B C 0 1 1 1.0 1 2 2 3.0 2 1 3 4.0 3 3 2 4.0 4 2 4 8.0  Variant if you want to use different aggregation functions at once (here sum and mean for the demo, although D could also be computed as C/B): import numpy as np idx, vals = pd.factorize(df['B']) df[['C', 'D']] = np.dstack([ df['A'].rolling(v, min_periods=1) .agg(['sum', 'mean']) for v in vals ])[np.arange(len(df)), :, idx]  Output:  A B C D 0 1 1 1.0 1.000000 1 2 2 3.0 1.500000 2 1 3 4.0 1.333333 3 3 2 4.0 2.000000 4 2 4 8.0 2.000000  Reproducible input: df = pd.DataFrame({'A': [1,2,1,3,2], 'B': [1,2,3,2,4]})
Question: A column is added to the dataframe with min value for every item just from columns corresponding from dictionary. How can I add a condition when calculating min value - if the values in the selected columns are greater than the values in the column 'Col7'? import pandas as pd my_dict={'Item1':['Col1','Col3','Col6'], 'Item2':['Col2','Col4','Col6','Col8'] } df=pd.DataFrame({ 'Col0':['Item1','Item2'], 'Col1':[20,25], 'Col2':[89,15], 'Col3':[36,30], 'Col4':[40,108], 'Col5':[55,2], 'Col6':[35,38], 'Col7':[30,20] }) df['min']=df.apply(lambda r:r[[col for col in my_dict.get(r['Col0'], []) if col in r]].min(),axis=1)  The result should be: df=pd.DataFrame({ 'Col0':['Item1','Item2'], 'Col1':[20,25], 'Col2':[89,15], 'Col3':[36,30], 'Col4':[40,108], 'Col5':[55,2], 'Col6':[35,38], 'Col7':[30,20], 'min':[35,38] }) 

Answer: You can get the minimum value and column name by adapting the answer given to your earlier question, passing a default value to min for the case where the condition causes there to be no matching columns: df[['min', 'name']] = df.apply( lambda r:min(((r[col], col) for col in my_dict.get(r['Col0'], []) if col in r and r[col] > r['Col7']), default=(np.nan, '')), axis=1, result_type='expand' )  Output (for your sample data):  Col0 Col1 Col2 Col3 Col4 Col5 Col6 Col7 min name 0 Item1 20 89 36 40 55 35 30 35 Col6 1 Item2 25 15 30 108 2 38 20 38 Col6  If we change the condition to r[col] > r['Col7']*1.2, the output is  Col0 Col1 Col2 Col3 Col4 Col5 Col6 Col7 min name 0 Item1 20 89 36 40 55 35 30 NaN 1 Item2 25 15 30 108 2 38 20 38.0 Col6  Note I've used NaN and '' as the default value, you can use whatever you choose in their place.
Question: Suppose we have the following dataframe:  timestamp open high low close delta atr last_index bearish bullish_turning_point 2 04-10-2024 01:54:44 18370.00 18377.75 18367.50 18376.00 32 0 1949 False True 5 04-10-2024 03:21:14 18376.50 18383.00 18375.25 18381.25 28 0 3899 False True 7 04-10-2024 04:38:54 18378.50 18386.25 18378.25 18385.50 133 0 5199 False True 9 04-10-2024 05:30:27 18384.00 18389.50 18378.75 18388.25 135 0 6499 False True 12 04-10-2024 06:06:12 18371.00 18378.00 18369.50 18378.00 130 0 8449 False True 14 04-10-2024 06:33:44 18372.25 18383.75 18372.00 18376.25 67 0 9749 False True 18 04-10-2024 07:21:14 18377.50 18387.75 18376.25 18380.00 8 0 12349 False True 22 04-10-2024 07:47:58 18388.00 18396.75 18385.25 18389.50 -30 0 14949 False True 25 04-10-2024 08:06:17 18390.75 18397.00 18387.50 18392.00 -25 0 16899 False True 28 04-10-2024 08:33:32 18384.75 18398.00 18383.25 18394.00 89 0 18849 False True 30 04-10-2024 08:54:35 18391.25 18403.00 18387.75 18399.25 84 0 20149 False True 34 04-10-2024 09:11:15 18388.75 18396.25 18385.75 18392.25 15 0 22749 False True 43 04-10-2024 10:02:22 18343.50 18350.50 18341.25 18350.50 113 0 28599 False True 46 04-10-2024 10:14:44 18352.00 18361.75 18352.00 18360.00 -42 0 30549 False True 49 04-10-2024 10:35:49 18354.00 18361.25 18347.75 18358.00 49 0 32499 False True 52 04-10-2024 10:54:18 18362.25 18372.00 18361.50 18372.00 180 0 34449 False True 56 04-10-2024 11:12:32 18369.25 18379.50 18367.00 18376.50 78 0 37049 False True 59 04-10-2024 11:27:27 18370.00 18376.50 18367.50 18373.25 54 0 38999 False True 65 04-10-2024 12:01:53 18377.75 18388.25 18377.50 18383.25 108 0 42899 False True 73 04-10-2024 12:25:04 18382.00 18386.25 18381.00 18384.75 65 0 48099 False True  How can I find the "nearest" close to "open" for each line before? E.g. For line 30 (close: 18399.25) this would be line 25 (open: 18390.75). For line 52 (close: 18372.00) this would be 14 (open: 18372.25) and so on.

Answer: If you want to include the current row's open in the closest open values, you can do this: df['nearest'] = [(abs(df.loc[:i, 'open'] - close)).idxmin() for i, close in df['close'].items()]  Output: [2, 5, 7, 9, 7, 5, 7, 22, 25, 25, 30, 30, 43, 46, 49, 14, 5, 14, 9, 28]  If you don't want to include the current row's open, it gets a little more complicated: df['nearest'] = [0] + [(abs(df.loc[:df.index[i-1], 'open'] - close)).idxmin() for i, close in enumerate(df['close'][1:], 1)]  Output: [0, 2, 5, 7, 7, 5, 7, 9, 22, 25, 25, 30, 2, 2, 46, 14, 5, 14, 9, 28]  You can substitute np.nan (or anything else) for the value for the first row as desired. The above code implements what is effectively an expanding window. If you want to implement a rolling window, you can use this code: k = 5 # length of window df['nearest'] = [0] * k + [(abs(df.loc[df.index[i-k]:df.index[i-1], 'open'] - close)).idxmin() for i, close in enumerate(df['close'][k:], k)]  Output: # k = 5 [0, 0, 0, 0, 0, 5, 7, 9, 22, 25, 25, 30, 28, 43, 46, 34, 34, 56, 59, 65] # k = 8 [0, 0, 0, 0, 0, 0, 0, 0, 22, 25, 25, 30, 12, 14, 46, 28, 28, 56, 34, 34]
Question: I have a numpy array psi of shape (3,1000) psi.__class__ Out[115]: numpy.ndarray psi.shape Out[116]: (3, 1000)  I want to partially fill psi with another array b b.__class__ Out[113]: numpy.ndarray b.shape Out[114]: (3, 500)  I can do this with a loop: for n in range(3): psi[n][:500] = b[n]  But it seems to me that there ought to be a way to do this more directly. But for instance psi[:][:500] = b  fails with error Traceback (most recent call last): File "<ipython-input-120-6b23082d9d6b>", line 1, in <module> psi[:][:500] = b ValueError: could not broadcast input array from shape (3,500) into shape (3,1000)  I've a few variations on the theme as well, with similar results. This seems pretty straightforward. Any idea how to do it?

Answer: You can use: psi[:, :500] = b  Or, if n (=3) in your loop does not match the first dimension of b: idx = np.arange(3) psi[idx, :500] = b[idx]  Comparing the two approaches (psi1 with the loop, psi2 as vectorial assignment): np.allclose(psi1, psi2) # True
Question: I've a dataframe with a non-unique MultiIndex:  A B L1 L2 7.0 7.0 -0.4 -0.1 8.0 5.0 -2.1 1.6 5.0 8.0 -1.8 -0.8 7.0 7.0 0.5 -1.2 NaN -1.1 -0.9 5.0 8.0 0.6 2.3  I want to select some rows using a tuple of values: data = df.loc[(7, 7), :]  With no surprise a warning is triggered: PerformanceWarning: indexing past lexsort depth may impact performance.  I'm trying to understand what in the current index causes this warning. I've read many answers here, some are related to old versions of pandas, other helped. From what I've read the warning is caused by two properties:  The index entries are not unique and The index entries are not sorted.  So I'm processing the dataframe index with this function designed from the answers found on this stack: def adjust_index(df): df = df.sort_index() # sort index levels = list(range(len(df.index.levels))) df_idx = df.groupby(level=levels).cumcount() # unique index df_adj = df.set_index(df_idx, append=True) # change index df_adj = df_adj.reset_index(level=-1, drop=True) # drop sorting level return df_adj  This doesn't remove the warning. Can you explain what is wrong, useless or missing? The rest of the code: import pandas as pd from numpy import nan, random as npr npr.seed(2) # Dataframe with unsorted MultiIndex def create_df(): n_rows = 6 data = npr.randn(n_rows, 2).round(1) choices = [8, 7, 5, 7, 8, nan] columns = ['A', 'B'] levels = ['L1', 'L2'] tuples = list(zip(npr.choice(choices, n_rows), npr.choice(choices, n_rows))) index = pd.MultiIndex.from_tuples(tuples, names=levels) df = pd.DataFrame(data, index=index, columns=columns) return df df = create_df() df = adjust_index(df) data = df.loc[(7, 7), :] # <-- triggers warning 

Answer: I got rid of the warning by sorting the index and putting the NaN values first: df.sort_index(inplace=True, na_position="first") data = df.loc[(7, 7), :] print(data)  Prints:  A B L1 L2 7.0 7.0 -0.4 -0.1 7.0 0.5 -1.2   I think the issue is with the NaN value you have in index. Pandas has special index.codes for each unique value in index and NaN is encoded as -1. So to have sorted index you have to have this -1 value on first position, hence na_position="first"