eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I've had a look and can't seem to find a solution to this issue. I'm wanting to calculate the rolling sum of the previous 30 days' worth of data at each date in the dataframe - by subgroup - for a set of data that isn't daily - it's spaced fairly irregularly. I've been attempting to use ChatGPT which is getting in a twist over it. Initially the suggestion was that I'd not converted the Date column to datetime format to allow for the rolling calculation, but now from the code below: import pandas as pd from datetime import datetime, timedelta import numpy as np # Create a dataset with irregularly spaced dates spanning two years np.random.seed(42) date_rng = pd.date_range(start='2022-01-01', end='2023-12-31', freq='10D') # Every 10 days data = {'Date': np.random.choice(date_rng, size=30), 'Group': np.random.choice(['A', 'B'], size=30), 'Value': np.random.randint(1, 30, size=30)} df = pd.DataFrame(data) # Sort DataFrame by date df.sort_values(by='Date', inplace=True) df['Date'] = pd.to_datetime(df['Date']) # Calculate cumulative sum by group within the previous 30 days from each day df['RollingSum_Last30Days'] = df.groupby('Group')['Value'].transform(lambda x: x.rolling(window='30D', min_periods=1).sum())  I'm getting an error of: ValueError: window must be an integer 0 or greater  I've found conflicting comments online as to whether the format '30D' works in rolling windows but I'm none the wiser as to a solution to this. Any help appreciated. Running in VSCode in Python 3.11.8.

Answer: The issue if that you need to specify which column to use as Date but don't have access to the Date with groupby.transform. You could use groupby.apply: # Calculate cumulative sum by group within the previous 30 days from each day df['RollingSum_Last30Days'] = (df.groupby('Group', group_keys=False) .apply(lambda x: x.rolling(window='30D', on='Date', min_periods=1)['Value'].sum()) )  Output:  Date Group Value RollingSum_Last30Days 9 2022-01-11 A 22 22.0 12 2022-01-11 A 22 44.0 6 2022-01-21 A 4 48.0 1 2022-05-21 B 14 14.0 23 2022-05-21 A 8 8.0 15 2022-07-20 B 26 26.0 4 2022-07-20 A 18 18.0 18 2022-07-30 B 10 36.0 7 2022-07-30 A 2 20.0 5 2022-08-19 A 8 10.0 10 2022-10-18 B 10 10.0 16 2022-11-17 B 12 12.0 11 2023-01-06 B 4 4.0 21 2023-02-15 B 16 16.0 26 2023-04-06 B 28 28.0 19 2023-04-26 A 4 4.0 28 2023-05-16 B 8 8.0 0 2023-05-26 B 3 11.0 8 2023-06-05 A 6 6.0 29 2023-06-25 A 21 27.0 17 2023-07-25 A 2 2.0 20 2023-08-04 B 14 14.0 22 2023-08-14 B 15 29.0 14 2023-08-14 B 18 47.0 3 2023-08-24 A 4 4.0 24 2023-09-03 B 14 47.0 25 2023-09-03 A 23 27.0 27 2023-09-03 A 25 52.0 13 2023-09-23 B 29 43.0 2 2023-12-12 A 17 17.0
Question: I want to retrieve AWS secrets using python boto3 and I came across this sample code:  aws-doc-sdk-examples/python/example_code/secretsmanager/get_secret_value.py at main · awsdocs/aws-doc-sdk-examples · GitHub Secrets Manager examples using SDK for Python (Boto3) - AWS SDK Code Examples  But it is confusing. I don't see boto3 library import in the python file. Not an expert of Python, so any help in understanding this much appreciated. I was expecting to have the AWS secrets name and boto3 library as part of the python function.

Answer: Per the documentation, each of the example folders has one or more main runner scripts. For the Secrets Manager examples, you would run either:  python scenario_get_secret.py, or python scenario_get_batch_secrets.py  Each of these 'runner' scripts imports the relevant Python code e.g. get_secret_value.py. The code is structured this way so that you can easily test the code while separating the test code from the reusable utility code (that you would potentially use).
Question: I have this list containing the following images with these the name structure "number.image"  The list stores the elements of a local folder based on a path.  [1.image, 2.image, 3.image, 4.image, 5.image, 6.image, 7.image, 8.image, 9.image, 10.image, 11.image, 12.image, 13.image]  applying the python build-in sorted() method to make sure the elements in the list are sorted in proper manner I got this result. As you see the order is not correct. 1.image 10.image 11.image 12.image 13.image 2.image 3.image 4.image 5.image 6.image 7.image 8.image 9.image 

Answer: If you want to use the inbuilt sorted function and not install a third-party library such as natsort, you can use a lambda for the key argument that interprets the stem of the file as an integer: >>> from pathlib import Path >>> filenames = [ ... '1.image', ... '10.image', ... '11.image', ... '12.image', ... '13.image', ... '2.image', ... '3.image', ... '4.image', ... '5.image', ... '6.image', ... '7.image', ... '8.image', ... '9.image', ... ] >>> sorted_filenames = sorted(filenames, key=lambda f: int(Path(f).stem)) >>> # Arguably less readable alternative using list indexing splitting on the period manually: >>> # sorted_filenames = sorted(filenames, key=lambda f: int(f.split('.')[0])) >>> print('\n'.join(sorted_filenames)) 1.image 2.image 3.image 4.image 5.image 6.image 7.image 8.image 9.image 10.image 11.image 12.image 13.image
Question: I am trying to calculate the Chi-Square result for a simply data table I have two groups, let's call them PG and KG. PG has in category 1 19 values and in category 2 11 values. KG has in category 1 0 values and in category 2 30 values. The python code would be from scipy.stats import chi2_contingency observed = [[0, 30], [19, 11]] chi2, p, dof, expected = chi2_contingency(observed) print("Chi Square value:", chi2)  This yields 24.95. But when using Stata or the online calculator https://mathcracker.com/de/chi-quadrat-test-unabhangigkeit, I get 27.80. I don't understand where the difference comes from.

Answer: There's parameter correction=, which is in default mode set to True: correction bool, optional If True, and the degrees of freedom is 1, apply Yates’ correction for continuity. The effect of the correction is to adjust each observed value by 0.5 towards the corresponding expected value.   from scipy.stats import chi2_contingency observed = [[0, 30], [19, 11]] chi2, p, dof, expected = chi2_contingency(observed, correction=False) print("Chi Square value:", chi2)  Prints: Chi Square value: 27.804878048780488
Question: I need to optimize this function in gekko and something is wrong. Black function(x2) is how teoretical it should look like.  m = GEKKO() m.options.IMODE = 6 m.time = np.linspace(0, 1, 100) x = m.Var(lb=1, ub=3) x2 = m.Var(lb=1, ub=3) J = m.Var(0) t = m.Param(value=m.time) m.Equation(J.dt() == 24*x*t + 2*x.dt()**2 -4*t) m.Equation(x2==t**3 + t + 1) Jf = m.FV() Jf.STATUS = 1 m.Connection(Jf, J, pos2 = 'end') m.Obj(Jf) m.solve() plt.plot(m.time, x.value) plt.plot(m.time, x2.value, color='black') plt.show() 

Answer: Here is a solution to the problem that aligns with the known solution:  from gekko import GEKKO import numpy as np import matplotlib.pyplot as plt m = GEKKO(remote=True) n = 101 m.time = np.linspace(0,1,n) x = m.Var(1,lb=1,ub=3) t = m.Param(value=m.time) p = np.zeros(n); p[-1]=1 final = m.Param(p) m.Equation(final*(x-3)==0) m.Minimize(final*m.integral(24*x*t + 2*x.dt()**2 -4*t)) m.options.IMODE = 6 m.options.SOLVER = 3 m.options.NODES = 2 m.solve() plt.plot(m.time, x.value,'b-',label='x gekko') tm = m.time plt.plot(tm,tm**3+t+1,'r--',label='x solution') plt.legend(); plt.xlabel('time'); plt.ylabel('x') plt.grid(); plt.show()  The variable x has an initial value 1 and bounds between 1 and 3. The parameter final is a numpy array with all zeros except for the last element set to one. This setup is used to define a constraint in the model that only applies at the final time point, ensuring x equals 3 at the final time. Don't use m.fix_final(x,3) because it also sets the derivative of x to zero at the final point. The objective function m.Minimize() contains an m.integral() expression that depends on x, t, and the derivative of x with respect to time. It is also multiplied by final so that only the final integral value is used to define the objective function.
Question: I have a dataframe df with double entries separated by a , in some columns. I want to write a function to extract only the entry before the , for columns with double entries. Example: 20.15,20.15 split to 20.15 See the dataframe import pandas as pd # initialize data of lists. data = {'Name': ['Tom', 'nick', 'krish', 'jack','Phil','Shaq','Frank','Jerome','Arpan','Sean'], 'Age': ['20.15,20.15', '21.02,21.02', '19.04,19.04','18.17,18.17','65.77,65.77','34.19,34.19','76.12,76.12','65.55,65.55','55.03,55.03','41.11,41.11'], 'Score_1':['10,10', '21,21', '19,19','18,18','65,65','34,34','76,76','65,65','55,55','41,41'], 'Score_2':['11,11', '31,31', '79,79','38,38','75,75','94,94','26,26','15,15','96,96','23,23'], 'Score_3':['101,101', '212,212', '119,119','218,218','765,765','342,342','706,706','615,615','565,565','491,491'], 'Type':[ 'A','C','D','F','B','E','H','G','J','K'], 'bonus':['3.13,3.13','5.02,5.02','4.98,4.98','6.66,6.66','0.13,0.13','4.13,4.13','5.12,5.12','4.28,4.28','6.16,6.16','5.13,5.13'], 'delta':[0.1,0.3,2.3,8.2,7.1,5.7,8.8,9.1,4.3,2.9]} # Create DataFrame df = pd.DataFrame(data) # Print the output. print(df)  Desired output (You can copy & paste) # initialize data of lists. df1 = {'Name': ['Tom', 'nick', 'krish', 'jack','Phil','Shaq','Frank','Jerome','Arpan','Sean'], 'Age': ['20.15', '21.02', '19.04','18.17','65.77','34.19','76.12','65.55','55.03','41.11'], 'Score_1':['10', '21', '19','18','65','34','76','65','55','41'], 'Score_2':['11', '31', '79','38','75','94','26','15','96','23'], 'Score_3':['101', '212', '119','218','765','342','706','615','565','491'], 'Type':[ 'A','C','D','F','B','E','H','G','J','K'], 'bonus':['3.13','5.02','4.98','6.66','0.13','4.13','5.12','4.28','6.16','5.13'], 'delta':[0.1,0.3,2.3,8.2,7.1,5.7,8.8,9.1,4.3,2.9]} # Create DataFrame df2 = pd.DataFrame(df1) # Print the output. print(df2)  I need help with a more robust function, see my attempt below def stringsplitter(data,column): # select columns with object datatype data1 = data.select_dtypes(include=['object']) cols= data1[column].str.split(',', n=1).str print(cols[0]) # applying stringsplitter to the dataframe final_df = df.apply(stringsplitter)  Thanks for your help

Answer: You can create the DataFrame and then edit the columns that have a comma. Note that this will only work if you're sure only the columns with duplicated data have commas in their values. # Create DataFrame df = pd.DataFrame(data) for col in df.columns: if df[col].dtype == "object": df[col] = df[col].astype(str).str.split(",").str[0] # Print the output. print(df)  The result will be:  Name Age Score_1 Score_2 Score_3 Type bonus delta 0 Tom 20.15 10 11 101 A 3.13 0.1 1 nick 21.02 21 31 212 C 5.02 0.3 2 krish 19.04 19 79 119 D 4.98 2.3 3 jack 18.17 18 38 218 F 6.66 8.2 4 Phil 65.77 65 75 765 B 0.13 7.1 5 Shaq 34.19 34 94 342 E 4.13 5.7 6 Frank 76.12 76 26 706 H 5.12 8.8 7 Jerome 65.55 65 15 615 G 4.28 9.1 8 Arpan 55.03 55 96 565 J 6.16 4.3 9 Sean 41.11 41 23 491 K 5.13 2.9
Question: I got a multilevel dataframe with the multivel columns, df.columns is ultiIndex([( 'county', ''), ( 'month', ''), ( 'Gender', 'Declined to self-identify'), ( 'Gender', 'Female'), ( 'Gender', 'Male'), ( 'Gender', 'Non-Binary / Gender expansive'), ( 'age', '0-4'), ( 'age', '11-13'), ( 'age', '15-18'), ( 'age', '19-24'), ( 'age', '25-34'), ( 'age', '35-44'), ( 'age', '45-54'), ( 'age', '5-10'), ( 'age', '55-64'), ( 'age', '65_vove'), ( 'age', 'Unkown'), ( 'race', 'American Indian or Alaska Native'), ( 'race', 'Asian'), ( 'race', 'Black or African American'), ( 'race', 'Native Hawaiian or other Pacific Islander'), ( 'race', 'UNKnown'), ( 'race', 'White'), ('Ethnicity', 'Hispanic or Latinx'), ('Ethnicity', 'Non Hispanic or Latinx'), ('Ethnicity', 'Unknown')], names=['variable', 'value'])  I would like to reorder the race as race_order = ['Black or African American', 'American Indian or Alaska Native','Asian', 'Native Hawaiian or other Pacific Islander','White', 'UNKnown'] 

Answer: Example Code let's make sample input import pandas as pd import numpy as np a = [('county', ''), ('month', ''), ('Gender', 'Declined to self-identify'), ('Gender', 'Female'), ('Gender', 'Male'), ('Gender', 'Non-Binary / Gender expansive'), ('age', '0-4'), ('age', '11-13'), ('age', '15-18'), ('age', '19-24'), ('age', '25-34'), ('age', '35-44'), ('age', '45-54'), ('age', '5-10'), ('age', '55-64'), ('age', '65_vove'), ('age', 'Unkown'), ('race', 'American Indian or Alaska Native'), ('race', 'Asian'), ('race', 'Black or African American'), ('race', 'Native Hawaiian or other Pacific Islander'), ('race', 'UNKnown'), ('race', 'White'), ('Ethnicity', 'Hispanic or Latinx'), ('Ethnicity', 'Non Hispanic or Latinx'), ('Ethnicity', 'Unknown')] np.random.seed(0) df = pd.DataFrame(np.random.randint(0, 10, (3, 26)), columns=pd.MultiIndex.from_tuples(a, names=['variable', 'value']))  df:   df1 = df.columns.to_frame(index=False) race_order = ['Black or African American', 'American Indian or Alaska Native','Asian', 'Native Hawaiian or other Pacific Islander','White', 'UNKnown'] df1.loc[df1['variable'].eq('race'), 'value'] = race_order out = df[pd.MultiIndex.from_frame(df1, names=['variable', 'value'])]  out:
Question: I need to find a similarity between the column in the polar data frame and the input value.I am using jaro_winkler_metric . I am getting errors while doing it. We don't want to use UDF functions as it slows the process. import polars as pl import jaro def test_polars(): name='savah' data = {"first_name": ['sarah', 'purnima'], "last_name": ['vats', 'malik']} df = pl.DataFrame(data) print(df) df = df.with_columns( [ (pl.when( jaro.jaro_winkler_metric(pl.col("first_name"), name) >= 0.8 ).then(1).otherwise(0)).alias("COMP80_FN"), ] ) print(df) if __name__ == '__main__': test_polars() C:\PythonProject\pythonProject\venv\Graph_POC\Scripts\python.exe "C:\PythonProject\pythonProject\polars data.py" shape: (2, 2) ┌────────────┬───────────┐ │ first_name ┆ last_name │ │ --- ┆ --- │ │ str ┆ str │ ╞════════════╪═══════════╡ │ sarah ┆ vats │ │ purnima ┆ malik │ └────────────┴───────────┘ Traceback (most recent call last): File "C:\PythonProject\pythonProject\polars data.py", line 22, in <module> test_polars() File "C:\PythonProject\pythonProject\polars data.py", line 12, in test_polars (pl.when( jaro.jaro_winkler_metric(pl.col("first_name"), name) >= 0.8 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\PythonProject\pythonProject\venv\Graph_POC\Lib\site-packages\jaro\__init__.py", line 43, in jaro_winkler_metric return jaro.metric_jaro_winkler(string1, string2) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\PythonProject\pythonProject\venv\Graph_POC\Lib\site-packages\jaro\jaro.py", line 235, in metric_jaro_winkler ans = string_metrics(string1, string2, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\PythonProject\pythonProject\venv\Graph_POC\Lib\site-packages\jaro\jaro.py", line 159, in string_metrics assert isinstance(s1, str) AssertionError Process finished with exit code 1 

Answer: The error lies within this snippet: jaro.jaro_winkler_metric(pl.col("first_name"), name)  You pass a polars expression to jaro_winkler_metric and I doubt the function such inputs, but expects a string instead. Instead you could try using: df = ( df .with_columns( pl.col("first_name").map_elements(lambda first_name: jaro.jaro_winkler_metric(first_name, name)).alias("jwm") ) .with_columns( pl.when(pl.col("jwm") >= 0.8).then(1).otherwise(0).alias("COMP80_FN") ) )
Question: I have a simple script to generate a document using python docx. I can generate the document fine, but when I try to change anythig related to the document sections I get the error:  ERROR - Error while processing: 'Twips' object is not callable   from docx import Document from docx.shared import Inches from docx.section import Section doc = Document() section = doc.sections[0] section.page_width(Inches(5)) section.page_height(Inches(5))  I've tried different units and ways of calling the sections array. I'm assuming the created document already has a section by default, but I've also tried to call doc.add_section() before it and the same issue happens.

Answer: Seems it's not correct usage: section.page_width(Inches(5)) section.page_height(Inches(5))  Instead try: section.page_width = Inches(5) section.page_height = Inches(5)
Question: please help me. In the function I set the optional parameter param as a boolean value, but the docs (swagger) do not display the data type for param. How to fix it? I'm using python 3.12.1 from fastapi import FastAPI from datetime import date app=FastAPI() @app.get ("/sales") def sales( date_from: date, date_to: date, param: bool | None = None, ): return date_from, date_to, param   on version python 3.9 everything is fine

Answer:  set the optional parameter "param" as a boolean value, but the docs do not display the data type  Well, it's not a bool, is it? You decided to make it a bool | None instead. (Sometimes pronounced Optional[bool], same thing.) Let's fix that:  param: bool = False,
Question: I can do either one of these separately, but not together. I think the question comes down to: after offsetting spines in a Matplotlib figure, how does one find the spine bounds in a coordinate system that can be used to plot arrowheads on the ends of the spines?  The arrows are (obviously) not aligned with the spines. The code for the arrowheads is from here https://matplotlib.org/stable/gallery/spines/centered_spines_with_arrows.html My simplified code is: import matplotlib.pyplot as plt import numpy as np spine_offset = 5 fig, ax = plt.subplots() a = np.linspace(0, 6, 100) ax.plot(a, np.sin(a) + 1) ax.spines.top.set_visible(False) ax.spines.right.set_visible(False) ax.spines.left.set_position(('outward', spine_offset)) ax.spines.bottom.set_position(('outward', spine_offset)) ax.plot(1, 0, ">k", transform=ax.transAxes, clip_on=False) ax.plot(0, 1, "^k", transform=ax.transAxes, clip_on=False) 

Answer: In set_position(('outward', spine_offset)) the offset is measured in points (1/72 of an inch, similar to how fonts are measured, e.g. a 12 point font). using a distance in "axes coordinates" Instead of 'outward', you can also use 'axes' coordinates: 0 at the bottom (left), 1 at the top (right). As usually the plot isn't square, to get a visual equal distance, you can convert the x-offset to the y-offset taking the aspect ratio of the bounding box into account. import matplotlib.pyplot as plt import numpy as np fig, ax = plt.subplots() x_offset = 0.02 y_offset = x_offset / ax.get_window_extent().height * ax.get_window_extent().width a = np.linspace(0, 2 * np.pi, 1000) ax.plot(np.sin(5 * a), np.sin(6 * a)) ax.set_facecolor('lightsalmon') # set a color to visualize the offset ax.spines.top.set_visible(False) ax.spines.right.set_visible(False) ax.spines.left.set_position(('axes', -x_offset)) ax.spines.bottom.set_position(('axes', -y_offset)) ax.plot(1, -y_offset, ">k", transform=ax.transAxes, clip_on=False) ax.plot(-x_offset, 1, "^k", transform=ax.transAxes, clip_on=False) plt.show()   Converting "points" to "axes coordinates" Alternatively, you can convert from points to pixels, and from pixels back to axes coordinates. There are 72 points in an inch, and the figure's dpi tells how many pixels there are in an inchh, import matplotlib.pyplot as plt import numpy as np fig, ax = plt.subplots() spine_offset_points = 5 spine_offset_pixels = fig.dpi * spine_offset_points / 72.0 x_offset, y_offset = ax.transAxes.inverted().transform((spine_offset_pixels, spine_offset_pixels)) \ - ax.transAxes.inverted().transform((0, 0)) a = np.linspace(0, 2 * np.pi, 1000) ax.plot(np.sin(3 * a), np.sin(8 * a)) ax.set_facecolor('violet') ax.spines.top.set_visible(False) ax.spines.right.set_visible(False) ax.spines.left.set_position(('outward', spine_offset_points)) ax.spines.bottom.set_position(('outward', spine_offset_points)) ax.plot(1, -y_offset, ">k", transform=ax.transAxes, clip_on=False) ax.plot(-x_offset, 1, "^k", transform=ax.transAxes, clip_on=False) plt.show()
Question: In polars I can get the horizontal max (maximum value of a set of columns for reach row) like this: df = pl.DataFrame( { "a": [1, 8, 3], "b": [4, 5, None], } ) df.with_columns(max = pl.max_horizontal("a", "b")) ┌─────┬──────┬─────┐ │ a ┆ b ┆ max │ │ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ i64 │ ╞═════╪══════╪═════╡ │ 1 ┆ 4 ┆ 4 │ │ 8 ┆ 5 ┆ 8 │ │ 3 ┆ null ┆ 3 │ └─────┴──────┴─────┘  This corresponds to Pandas df[["a", "b"]].max(axis=1). Now, how do I get the column names instead of the actual max value? In other words, what is the Polars version of Pandas' df[CHANGE_COLS].idxmax(axis=1)? The expected output would be: ┌─────┬──────┬─────┐ │ a ┆ b ┆ max │ │ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ str │ ╞═════╪══════╪═════╡ │ 1 ┆ 4 ┆ b │ │ 8 ┆ 5 ┆ a │ │ 3 ┆ null ┆ a │ └─────┴──────┴─────┘ 

Answer: You can concatenate the elements into a list using pl.concat_list, get the index of the largest element using pl.Expr.list.arg_max, and replace the index with the column name using pl.Expr.replace. mapping = {0: "a", 1: "b"} ( df .with_columns( pl.concat_list(["a", "b"]).list.arg_max().replace(mapping).alias("max_col") ) )  This can all be wrapped into a function to also handle the creation of the mapping dict. def max_col(cols) -> str: mapping = dict(enumerate(cols)) return pl.concat_list(cols).list.arg_max().replace(mapping) df.with_columns(max_col(["a", "b"]).alias("max_col"))  Output. shape: (3, 3) ┌─────┬──────┬─────────┐ │ a ┆ b ┆ max_col │ │ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ str │ ╞═════╪══════╪═════════╡ │ 1 ┆ 4 ┆ b │ │ 8 ┆ 5 ┆ a │ │ 3 ┆ null ┆ a │ └─────┴──────┴─────────┘
Question: I get the following warning: SyntaxWarning: invalid escape sequence '\.'  when the following shell block is present in a rule:  """ curl {params.ua} -L {params.url} \ | gzip -dc \ | sd '^.*gene_id "([A-Z0-9\.]+).*"; transcript_id "([A-Z0-9\.]+)".*;.*' '$2\t$1' \ | uniq | gzip > {output} """  The warning gives me a line number in another rule, commenting out the other rule doesn't resolve the warning (and the other rule doesn't have a '.'). Commenting out the rule with the above shell block does resolve the warning. The shell code seems to work as intended, so maybe I just need to format it differently such that Snakemake is happy and it still does what I want it to, but I can't figure this out so far.

Answer: It's possible to specify that this is a raw string:  r""" ... \. ... """