eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am having troubles with a data transformation I am trying to do. I have a column of data (Ex. 1,2,3,4,5,6,7,8,9)I want to create a new column that looks back n rows and concatenates the values into a new value, preferably an integer. For example, if the lookback window is 3 in my example, the new column would be Nan, Nan, 123, 234,345,456,567,678,789. So far, here is a bit of code that I tried where n is the lookback window and Streak is the dataframe with the values I am looking to combine into a new value (streakHistory): def getStreakHistory(Streak, n=20): streakHistory= "" for x in range(1, n + 1): streakHistory=str(streakHistory) + str(Streak["Streak"].shift(x)) return streakHistory df["Streak History"] = getStreakHistory(Streak)  This seems to run an error because streakHistory is a string. I have seen other options where you transpose into other cells, but I want all of the values to be combined and entered into 1 cell. Any help would be greatly appreciated. I also looked into a join, but that seems to be similar to a standard table join and not really the same as what I was looking at unless I am overlooking a particular functionality of it.

Answer: One option would be to use numpy's sliding_window_view combined with agg: from numpy.lib.stride_tricks import sliding_window_view as svw df = pd.DataFrame({'Streak': [1,2,3,4,5,6,7,8,9]}) N = 3 df['Streak History'] = (pd.DataFrame(svw(df['Streak'].astype(str), N), index=df.index[N-1:]) .agg(''.join, axis=1) )  Output:  Streak Streak History 0 1 NaN 1 2 NaN 2 3 123 3 4 234 4 5 345 5 6 456 6 7 567 7 8 678 8 9 789  Numeric variant: df['Streak History'] = pd.Series((svw(df['Streak'], N) *(10**np.arange(N-1, -1, -1))).sum(1), index=df.index[N-1:])  Output:  Streak Streak History 0 1 NaN 1 2 NaN 2 3 123.0 3 4 234.0 4 5 345.0 5 6 456.0 6 7 567.0 7 8 678.0 8 9 789.0
Question: I am a R user trying to learn python. For some analysis, I used to rename dataframe variable like this library(dplyr) variable = "c" df = data.frame(a=c(8,5,7,8), b=c(9,6,6,8), c=c(0,7,8,9)) > df a b c 1 8 9 0 2 5 6 7 3 7 6 8 4 8 8 9 out = df %>% rename(variable := !!variable) > out a b variable 1 8 9 0 2 5 6 7 3 7 6 8 4 8 8 9  I don't find way to do the same in python. Any idea to help me ? import pandas as pd df = pd.DataFrame( { "a": [8,5,7,8], "b": [9,6,6,8], "c": [0,7,8,9], } ) 

Answer: You can do the same with rename and a dictionary, the order of the "parameters" is just reversed compared to R: variable = 'c' out = df.rename(columns={variable: 'variable'})  Or, without the intermediate: out = df.rename(columns={'c': 'variable'}) # df %>% rename(variable := c)  Output:  a b variable 0 8 9 0 1 5 6 7 2 7 6 8 3 8 8 9
Question: I have the following DataFrame in Pandas:    ID snapshot_date row_hash     qwe Jan 01 2024 123   qwe Jan 03 2024 456   qwe Jan 05 2024 456   qwe Jan 07 2024 123    Note: that row_hash changed back on Jan 07 2024 I want to create 3 groups (like window function in SQL), but I can't get the desired rseult:    ID snapshot_date row_hash dense_rank I get dense_rank needed     qwe Jan 01 2024 123 1 1   qwe Jan 03 2024 456 1 2   qwe Jan 05 2024 456 2 2   qwe Jan 07 2024 123 2 3    I tried to create it with the code: snapshot_df['dense_rank'] = snapshot_df.groupby(['ID', 'row_hash'])['snapshot_date'].rank( method='dense').astype(int)  Can anybody help me?

Answer: If need counter for change by columns ID and row_hash compare by shifted values by DataFrame.shift with DataFrame.any and Series.cumsum: cols = ['ID', 'row_hash'] df['dense_rank'] = df[cols].ne(df[cols].shift()).any(axis=1).cumsum() print (df) ID snapshot_date row_hash dense_rank 0 qwe Jan 01 2024 123 1 1 qwe Jan 03 2024 456 2 2 qwe Jan 05 2024 456 2 3 qwe Jan 07 2024 123 3  How it working: print (df[cols].ne(df[cols].shift())) ID row_hash 0 True True 1 False True 2 False False 3 False True print (df[cols].ne(df[cols].shift()).any(axis=1)) 0 True 1 True 2 False 3 True dtype: bool print (df[cols].ne(df[cols].shift()).any(axis=1).cumsum()) 0 1 1 2 2 2 3 3 dtype: int32  EDIT: Solution for groups: cols = ['ID', 'row_hash'] df['dense_rank'] = df[cols].ne(df[cols].shift()).any(axis=1).groupby(df['ID']).cumsum()  Or: cols = ['ID', 'row_hash'] df['dense_rank'] = (df.assign(g = df[cols].ne(df[cols].shift()).any(axis=1)) .groupby(['ID'])['g'] .cumsum())   print (df) ID snapshot_date row_hash dense_rank 0 qwe Jan 01 2024 123 1 1 qwe Jan 03 2024 456 2 2 qwe Jan 05 2024 456 2 3 qwe Jan 07 2024 123 3 4 qwe1 Jan 01 2024 123 1 5 qwe1 Jan 03 2024 456 2 6 qwe1 Jan 05 2024 456 2 7 qwe1 Jan 07 2024 123 3
Question: I have two dfs, two df's has be to be merged by class and joining dates. Please check the below df's df1 class teacher age instructor_joining_date A mark 50 2024-01-20 07:18:29.599 A john 45 2024-05-08 05:31:21.379 df2 class count student_joining_date A 1 2024-05-17 01:05:58.072 A 50 2024-04-10 10:39:06.608 A 75 2024-04-05 09:49:07.246 Final output df class count student_joining_date teacher age A 1 2024-05-17 01:05:58.072 john 45 A 50 2024-04-10 10:39:06.608 mark 50 A 75 2024-04-05 09:49:07.246 mark 50  For df2 we have merge df1 by class and joining dates Edit: Yes if student_joining_date and instructor_joining_date is different. If student_joining_date is greater than instructor_joining_date then that teacher will be mapped here

Answer: You have to use a merge_asof, then restore the original order with reindex: df1['instructor_joining_date'] = pd.to_datetime(df1['instructor_joining_date']) df2['student_joining_date'] = pd.to_datetime(df2['student_joining_date']) out = (pd.merge_asof(df2.sort_values(by='student_joining_date').reset_index(), df1.sort_values(by='instructor_joining_date'), left_on='student_joining_date', right_on='instructor_joining_date', by='class') .set_index('index').reindex(df2.index) .drop(columns='instructor_joining_date') )  Output:  class count student_joining_date teacher age 0 A 1 2024-05-17 01:05:58.072 john 45 1 A 50 2024-04-10 10:39:06.608 mark 50 2 A 75 2024-04-05 09:49:07.246 mark 50  Intermediate before dropping the instructor_joining_date column:  class count student_joining_date teacher age instructor_joining_date 0 A 1 2024-05-17 01:05:58.072 john 45 2024-05-08 05:31:21.379 1 A 50 2024-04-10 10:39:06.608 mark 50 2024-01-20 07:18:29.599 2 A 75 2024-04-05 09:49:07.246 mark 50 2024-01-20 07:18:29.599
Question: I’m trying to plot a curve_fit for the S&P 500. I’m successful (I think) at performing a linear fit/plot. When I try to get an exponential curve_fit to work, I get this error: Optimal parameters not found: Number of calls to function has reached maxfev = 800.  import numpy as np import matplotlib.pyplot as plt import yfinance as yf from scipy.optimize import curve_fit # get data df = yf.download("SPY", interval = '1mo') df = df.reset_index() def func(x, a, b): return a * x + b # ?? return a * np.exp(-b * x) + c # ?? return a*(x**b)+c # ?? return a*np.exp(b*x) # create data arrays # convert Date to numeric for curve_fit ?? xdata = df['Date'].to_numpy().astype(np.int64)//10**9 ydata = df['Close'].to_numpy() # p0 = (?, ?, ?) use guesses? popt, pcov = curve_fit(func, xdata, ydata) print(popt) y_pred = func(xdata, *popt) plt.plot(xdata, ydata) plt.plot(xdata, y_pred, '-') plt.show()   Am I dealing with dates correctly? Should I be doing a p0 initial guess? This question/solution may provide some clues. It would be nice to have the x-axis labeled in a date format (but not important right now).

Answer: In addition to normalizing the data, it is important to actually choose a good function. In your example you had: def func(x, a, b): return a * x + b # ?? return a * np.exp(-b * x) + c # ?? return a*(x**b)+c # ?? return a*np.exp(b*x)  The correct one, when you say you want to fit an exponential, should be this, IMO: # define the exponential growth function, before you had exponential decay because of -b def ExponentialGrowth(x, a, b, c): return a * np.exp(b * x) + c # + c due to account for offset  The power function might work as well, I did not check. Anyways, here's the code: # define the exponential growth function, before you had exponential decay because of -b def ExponentialGrowth(x, a, b, c): return a * np.exp(b * x) + c # + c due to account for offset # get data x = df['Date'].to_numpy().astype(np.int64)//10**9 y = df['Close'].to_numpy() # apply z normalization xNorm = (x - x.mean()) / x.std() yNorm = (y - y.mean()) / y.std() # get the optimal parameters popt, pcov = curve_fit(ExponentialGrowth, xNorm, yNorm) # get the predicted but in the normalized range yPredNorm = ExponentialGrowth(xNorm, *popt) # reverse normalize the predicted values yPred = yPredNorm * (y.std()) + y.mean() plt.figure() plt.scatter(df['Date'], y, 1) plt.plot(df['Date'], yPred, 'r-') plt.grid() plt.legend(["Raw", "Fitted"]) plt.xlabel("Year") plt.ylabel("Close")  And the results:  If you will eventually need to get initial guesses, you can search online how to get the initial guesses for any function. For example, if I am fitting an exponential growth function and I know that the data has an offset of 100, I can set the initial guess of c to a 100... Hope this helps you.
Question: I'm trying to build a simple dashboard to display temperature and humidity on different days. I would like to have a dropdown to select the date and then have a row showing a graph of temperature throughout the day next to a graph showing humidity (two graphs in a single row). I can't get it to work even though I've tried using the embedded dbc.column within dbc.row. I believe the width parameter isn't doing anything and maybe the app is wrapping around. Here is the code with which you can reproduce what I am doing (keep in mind I am using Jupyter Notebook): import pandas as pd import dash from dash import dcc from dash import html from dash.dependencies import Input, Output import dash_bootstrap_components as dbc import plotly.express as px # Sample DataFrame data_dict = { 'date': ['2024-05-14', '2024-05-14', '2024-05-14', '2024-05-15','2024-05-15','2024-05-15'], 'time': ['12:00', '12:10', '12:20', '12:00', '12:10', '12:20'], 'temperature': [25, 26, 27, 20, 22, 28], 'humidity': [50, 55, 50, 60, 78, 77] } data = pd.DataFrame(data_dict) app = dash.Dash(__name__) app.layout = dbc.Container([ html.H1('Weather Dashboard'), dbc.Row([ dbc.Col([ dcc.Dropdown( id='date-dropdown', options=[{'label': date, 'value': date} for date in data['date'].unique()], value=data['date'].unique()[0] # Default value ) ], width=4) ]), dbc.Row([ dbc.Col([ dcc.Graph(id='temp-graph-plotly', figure={}) ], width=12), dbc.Col([ dcc.Graph(id='hum-graph-plotly', figure={}) ], width=12) ]) ]) # Define callback to update graph @app.callback( Output('temp-graph-plotly', 'figure'), Output('hum-graph-plotly', 'figure'), Input('date-dropdown', 'value') ) def update_graph(selected_date): # Filter DataFrame based on selected date filtered_data = data[data['date'] == selected_date] ### TEMPERATURE ### fig_temp_plotly = px.line(filtered_data, x='time', y='temperature').update_xaxes(tickangle=300) ### HUMIDITY ### fig_hum_plotly = px.line(filtered_data, x='time', y='humidity').update_xaxes(tickangle=300) return fig_temp_plotly, fig_hum_plotly # Run the app if __name__ == '__main__': app.run_server(debug=True, port=8010)  I thought my layout was ok but I'm getting something like the following image, where the graphs are in two rows instead of simply one next to each other: 

Answer: Dash Bootstrap Components: Layout:  Layout in Bootstrap is controlled using the grid system. The Bootstrap grid has twelve columns [...] The width of your columns can be specified in terms of how many of the twelve grid columns it should span [...]  Each of your graph column spans across the whole 12-column sized row. That is why the second dbc.Col is placed below the first one - no more space in that dbc.Row. Set the width to 6 or less to fit both columns in one row. You're also missing external_stylesheets in your application initialization, should be: app = dash.Dash( external_stylesheets=[dbc.themes.BOOTSTRAP] )
Question: I have a Python function that retrieves the first element of an arbitrary number of *args: def get_first(*args): return tuple(a[0] for a in args)  Lets say that I call this function as follows: b = (1, 2, 3) c = ("a", "b", "c", "d") x = get_first(b, c)  I expect the type Tuple[int, str]. To me it seems impossible to achieve the correct typing to accurately reveal this type. I have had no luck with the TypeVarTuple PEP 646 or with the Paramspec PEP 612.

Answer: The signature you're looking for is related to the signature of zip; in fact, you could implement your function as next(zip(*args)). Unfortunately, there is, at the moment, no good way to type such a function. It would require variadic generics which are not currently implemented. For example, you can see the type signature for zip here:  @overload def __new__(cls, *, strict: bool = ...) -> zip[Any]: ... @overload def __new__(cls, iter1: Iterable[_T1], /, *, strict: bool = ...) -> zip[tuple[_T1]]: ... @overload def __new__(cls, iter1: Iterable[_T1], iter2: Iterable[_T2], /, *, strict: bool = ...) -> zip[tuple[_T1, _T2]]: ... @overload def __new__( cls, iter1: Iterable[_T1], iter2: Iterable[_T2], iter3: Iterable[_T3], /, *, strict: bool = ... ) -> zip[tuple[_T1, _T2, _T3]]: ... @overload def __new__( cls, iter1: Iterable[_T1], iter2: Iterable[_T2], iter3: Iterable[_T3], iter4: Iterable[_T4], /, *, strict: bool = ... ) -> zip[tuple[_T1, _T2, _T3, _T4]]: ... @overload def __new__( cls, iter1: Iterable[_T1], iter2: Iterable[_T2], iter3: Iterable[_T3], iter4: Iterable[_T4], iter5: Iterable[_T5], /, *, strict: bool = ..., ) -> zip[tuple[_T1, _T2, _T3, _T4, _T5]]: ... @overload def __new__( cls, iter1: Iterable[Any], iter2: Iterable[Any], iter3: Iterable[Any], iter4: Iterable[Any], iter5: Iterable[Any], iter6: Iterable[Any], /, *iterables: Iterable[Any], strict: bool = ..., ) -> zip[tuple[Any, ...]]: ...  This typechecks precisely only up to six arguments, which is good enough in practice, but is quite repetitive and rather clumsy. Unfortunately, this is probably the best you can do for now.
Question: Let's say we have 3 variables a, b & c. There are n instances of each, and all but the first instance of c are null. We are to calculate each next c based on a given formula comprising of only present variables on the right hand side: c = [(1 + a) * (current_c) * (b)] + [(1 + b) * (current_c) * (a)] How do we go about this calculation without using native python looping? I've tried:  pl.int_range(my_index_column_value, pl.len() + 1) (my index starts form 1) pl.rolling(...) (this seems to be quite an expensive operation) pl.when(...).then(...).otherwise(...) with the above two along with .over(...) & pl.select(...).item()  to no avail. It's always the case that _the shift has already been fully made at once. I thought perhaps the most plausible way to do this would be either rolling by 1 with grouping by 2, or via pl.int_range(...) and using the current index column number as the shift value. However, these keep failing as I am unable to properly come up with the correct syntax - I'm unable to pass the index column value and have polars accept it as a number. Even casting throws the same errors. Right now I am thinking we could manage another row for shifting and passing values back to row c, but then again, I'm not sure if this would even be an efficient way to go about it... What would be the most optimal way to go about this without offloading to Rust? Code for reference: import polars as pl if __name__ == "__main__": initial_c_value = 3 df = pl.DataFrame(((2, 3, 4, 5, 8), (3, 7, 4, 9, 2)), schema=('a', 'b')) df = df.with_row_index('i', 1).with_columns(pl.lit(None).alias('c')) df = df.with_columns(pl.when(pl.col('i') == 1) .then( (((1 + pl.col('a')) * (initial_c_value) * (pl.col('b'))) + ((1 + pl.col('b')) * (initial_c_value) * (pl.col('a')))).alias('c')) .otherwise( ((1 + pl.col('a')) * (pl.col('c').shift(1)) * (pl.col('b'))) + ((1 + pl.col('b')) * (pl.col('c').shift(1)) * (pl.col('a')))).shift(1).alias('c')) print(df) 

Answer: Using numba you can make ufuncs which polars can use seamlessly. from numba import guvectorize, int64 import polars as pl @guvectorize([(int64[:], int64[:], int64, int64[:])], '(n),(n),()->(n)', nopython=True) def make_c(a,b,init_c, res): res[0]=(1+a[0]) * init_c * b[0] + (1+b[0]) * init_c * a[0] for i in range(1,a.shape[0]): res[i] = (1+a[i]) * res[i-1] * b[i] + (1+b[i]) * res[i-1] * a[i] df = pl.DataFrame(((2, 3, 4, 5, 8), (3, 7, 4, 9, 2)), schema=('a', 'b')) df.with_columns( c=make_c(pl.col('a'), pl.col('b'), 3) ) shape: (5, 3) ┌─────┬─────┬───────────┐ │ a ┆ b ┆ c │ │ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ i64 │ ╞═════╪═════╪═══════════╡ │ 2 ┆ 3 ┆ 51 │ │ 3 ┆ 7 ┆ 2652 │ │ 4 ┆ 4 ┆ 106080 │ │ 5 ┆ 9 ┆ 11032320 │ │ 8 ┆ 2 ┆ 463357440 │ └─────┴─────┴───────────┘  The way it works is that the ufunc detects that its input is a polars Expr (ie pl.col() is an Expr) and then it hands control to polars. Because of that you can NOT just do make_c('a','b',3) as then its input is just a str and it won't know what to do with that.
Question: I'm trying to make a function that automatically takes a table from a website(Wikipedia) cleans it a bit and than displays it, everything worked well with my first 2 tables but the third one is giving me some troubles. This is the code to define the function: def createTable(url, match): data= pd.read_html(url, match= match) name= data[0]["Name"] origin= data[0]["Origin"] type_= data[0]["Type"] number= data[0]["Number"] df= pd.DataFrame({"Name": name, "Origin": origin, "Type": type_, "Number": number}) df.replace("?", np.nan, inplace=True) df['Number']= df['Number'].replace(to_replace={r"\(.*\)": "", r"\[.*\]": ""}, regex=True) return df  and this is the function at work: df_avIT= pd.DataFrame() df_avIT= createTable("https://en.wikipedia.org/wiki/List_of_equipment_of_the_Italian_Army", "125 To be upgraded and remain in service until 2035") df_avUK= pd.DataFrame() df_avUK= createTable("https://en.wikipedia.org/wiki/List_of_equipment_of_the_British_Army", "Challenger 2") df_avFR= pd.DataFrame() df_avFR= createTable("https://en.wikipedia.org/wiki/List_of_equipment_of_the_French_Army", "AMX Leclerc")  As I said the first 2 give me no problem at all but when I tried on the third it returns, ValueError: If using all scalar values, you must pass an index. I know well the code isn't great I'm trying to improve it but this problem is stopping me and I can't find a valid solution, even though I scearched for similar problem to mine in various forums. (I'm sorry if my English is bad, if you didn't understand something tell me I'm gonna try to explain more).

Answer: Your script does not consistently yield Series for name/origin/type_/number, you sometimes have DataFrames, you can try to squeeze: name= data[0]["Name"].squeeze() origin= data[0]["Origin"].squeeze() type_= data[0]["Type"].squeeze() number= data[0]["Number"].squeeze()  Side note: df_avIT = pd.DataFrame() is useless, you don't need to initialize empty DataFrames since the variable will be overwritten by df_avIT = createTable(...)
Question: I have the following data: data = [{'Shape': 'Circle', 'Color': 'Green'}, {'Shape': 'Circle', 'Color': 'Green'}, {'Shape': 'Circle', 'Color': 'Green'}]  Which I create a DataFrame from: df = pd.DataFrame(data)  Giving: >>> df Shape Color 0 Circle Green 1 Circle Green 2 Circle Green  The data is always received in this form, and I cannot change it. Now I need to count the Color column for each Shape, with the Color as the index, like this:  Circle Square Green 3 0 Red 0 0  However, while I know that Shape can be either Circle or Square, they may or may not be present in the data. Likewise, Color can be either Green or Red, but also may or may not be present in the data. So at the moment my solution is to use: df2 = pd.DataFrame( [ { "Color": "Green", "Circle": len(np.where((df["Shape"] == "Circle") & (df["Color"] == "Green"))[0]), "Square": len(np.where((df["Shape"] == "Square") & (df["Color"] == "Green"))[0]), }, { "Color": "Red", "Circle": len(np.where((df["Shape"] == "Circle") & (df["Color"] == "Red"))[0]), "Square": len(np.where((df["Shape"] == "Square") & (df["Color"] == "Red"))[0]), }, ] ) df2 = df2.set_index("Color") df2.index.name = None  Which gives the desired result: >>> df2 Circle Square Green 3 0 Red 0 0  But I suspect this is inefficient. Is there a better way of doing this in Pandas directly? I tried a pivot_table, but couldn't get it to account for possible missing values in the data.

Answer: Using pivot_table and reindex: data = [{'Shape': 'Circle', 'Color': 'Green'}, {'Shape': 'Circle', 'Color': 'Green'}, {'Shape': 'Circle', 'Color': 'Green'}] df = pd.DataFrame(data) shapes = ['Circle', 'Square'] colors = ['Green', 'Red'] pivot_table = df.pivot_table(index='Color', columns='Shape', aggfunc='size', fill_value=0) pivot_table = pivot_table.reindex(index=colors, columns=shapes, fill_value=0)
Question: I'm currently working with Pydantic's URL type for URL validation in my Python project. However, it seems that Pydantic does not currently provide a built-in mechanism for this. So what is the best approach to restrict a URL to a specific list of hosts in Pydantic? I have a pydantic model called Media which has an url attribute. I want the url to be restricted to certain websites or hosts." from pydantic import BaseModel, AnyUrl class Media(BaseModel): url: AnyUrl # the url host should only be from x.com or y.com 

Answer: As you already mentioned there is no built-in support for doing so. Here an AfterValidator can do the job: from typing import Annotated, TypeAlias from pydantic import AfterValidator, AnyUrl, BaseModel valid_hosts = {"www.google.com", "www.yahoo.com"} def check_specific_hosts(url: AnyUrl) -> AnyUrl: if url.host in valid_hosts: return url raise ValueError("It's not in the list of accepted hosts") AcceptedUrl: TypeAlias = Annotated[AnyUrl, AfterValidator(check_specific_hosts)] class Media(BaseModel): url: AcceptedUrl print(Media(url="http://www.google.com")) try: print(Media(url="http://www.facebook.com")) except ValueError as e: print(e)  output: url=Url('http://www.google.com/') 1 validation error for Media url Value error, It's not in the list of accepted hosts [type=value_error, input_value='http://www.facebook.com', input_type=str] For further information visit https://errors.pydantic.dev/2.6/v/value_error