eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: This is my DataFrame: import pandas as pd import numpy as np df = pd.DataFrame( { 'a': [98, 97, 100, 101, 103, 110, 108, 109, 130, 135], 'b': [3, 3, 3, 3, 3, 3, 3, 3, 3, 3], 'c': [np.nan, np.nan, 1.0, 1.0, 1.0, 2.0, 2.0, 2.0, 3.0, 3.0], 'd': [92, 92, 92, 92, 92, 92, 92, 92, 92, 92], } )  And this is the expected output. I want to to create column x:  a b c d x 0 98 3 NaN 92 92 1 97 3 NaN 92 92 2 100 3 1.0 92 94 3 101 3 1.0 92 94 4 103 3 1.0 92 94 5 110 3 2.0 92 104 6 108 3 2.0 92 104 7 109 3 2.0 92 104 8 130 3 3.0 92 124 9 135 3 3.0 92 124  Steps: a) When c is not duplicated, df['x'] = df.a - (df.b * 2) b) If df.c == np.nan, df['x'] = df.d For example: The first new value in c is row 2. So df['x'] = 100 - (3 * 2) which is 94 and df['x'] = 94 until a new value in c appears which is row 5. For row 5, df['x'] = 110 - (3 * 2) which is 104. And the logic continues. This is what I have tried: df['x'] = df.a - (df.b * 2) df.loc[df.c.isna(), 'x'] = df.d df['x'] = df.x.cummax() 

Answer: You can use duplicated, mask, grouby.ffill and fillna: # identify duplicated "c" m = df['c'].duplicated() # compute a-(2*b) # mask the duplicated "c" # ffill per group # replace NaN with "d" df['x'] = (df['a'].sub(df['b'] * 2) .mask(m) .groupby(df['c']).ffill() .fillna(df['d']) )  Variant to work by groups of successive identical "c": g = df['c'].ne(df['c'].shift()).cumsum() m = g.duplicated() df['x'] = (df['a'].sub(df['b'] * 2) .mask(m) .groupby(m1.cumsum()).ffill() .where(df['c'].notna(), df['d']) )  Output:  a b c d x 0 98 3 NaN 92 92.0 1 97 3 NaN 92 92.0 2 100 3 1.0 92 94.0 3 101 3 1.0 92 94.0 4 103 3 1.0 92 94.0 5 110 3 2.0 92 104.0 6 108 3 2.0 92 104.0 7 109 3 2.0 92 104.0 8 130 3 3.0 92 124.0 9 135 3 3.0 92 124.0
Question: The goal is to merge three different dataframes having different timesteps (10min, 15min and 30min. The code must recognize what timestep to consider firstly and identify the next available next timestep. in This example 2019/04/02 10:40:00 does not exist in the dataframes dataset. Therefore the next timestep to consider after 2019/04/02 10:30:00 would be 2019/04/02 10:45:00. df1:     Timestamp data1     2019/04/02 10:00:00 1   2019/04/02 10:10:00 1   2019/04/02 10:20:00 1   2019/04/02 10:30:00 1     df2:     Timestamp data2     2019/04/02 10:00:00 2   2019/04/02 10:15:00 22   2019/04/02 10:30:00 222   2019/04/02 10:45:00 2222   2019/04/02 11:00:00 22222     df3:     Timestamp data3     2019/04/02 10:00:00 3   2019/04/02 10:30:00 33   2019/04/02 11:00:00 333   2019/04/02 11:30:00 3333     desired result:     Timestamp data1 data2 data3     2019/04/02 10:00:00 1 2 3   2019/04/02 10:10:00 1 NaN NaN   2019/04/02 10:15:00 NaN 22 NaN   2019/04/02 10:20:00 1 NaN NaN   2019/04/02 10:30:00 1 222 33   2019/04/02 10:45:00 NaN 2222 NaN   2019/04/02 11:00:00 NaN 22222 333   2019/04/02 11:30:00 NaN NaN 3333     I used the python concat function and the merge function but did not deliver he desired result.

Answer: # Convert Timestamp columns to datetime df1['Timestamp'] = pd.to_datetime(df1['Timestamp']) df2['Timestamp'] = pd.to_datetime(df2['Timestamp']) df3['Timestamp'] = pd.to_datetime(df3['Timestamp']) # Sort the DataFrames based on Timestamp df1 = df1.sort_values('Timestamp') df2 = df2.sort_values('Timestamp') df3 = df3.sort_values('Timestamp') # Merge using merge_asof result = pd.merge_asof(df1, df2, on='Timestamp', direction='nearest') result = pd.merge_asof(result, df3, on='Timestamp', direction='nearest')  EDIT: IF YOU WANT TO KEEP NAN VALUES # Convert Timestamp columns to datetime df1['Timestamp'] = pd.to_datetime(df1['Timestamp']) df2['Timestamp'] = pd.to_datetime(df2['Timestamp']) df3['Timestamp'] = pd.to_datetime(df3['Timestamp']) # Merge using merge with how='outer' result = pd.merge(df1, df2, on='Timestamp', how='outer') result = pd.merge(result, df3, on='Timestamp', how='outer') # Sort the result based on Timestamp result = result.sort_values('Timestamp')
Question: I have a pandas dataframe like below id name value1 value2 value3 Type ======================================================= 1 AAA 1.0 1.5 1.8 NEW 2 BBB 2.0 2.3 2.5 NEW 3 CCC 3.0 3.6 3.7 NEW  I have convert the above df into something like below so that i can join it to another df based on name (which will be unique in my case) Type AAA BBB CCC ================================================================ NEW [1.0, 1.5, 1.8] [2.0, 2.3, 2.5] [3.0, 3.6, 3.7]  Is there any way to achieve this instead of having too many looping statements. Any help is much appreciated. Thanks,

Answer: A possible solution, which first adds column value containing the list and then uses pivot: (df.assign(value=df.loc[:, 'value1':'value3'].apply(list, axis=1)) .pivot(index='Type', columns='name', values='value') .rename_axis(None, axis=1).reset_index())  Output:  Type AAA BBB CCC 0 NEW [1.0, 1.5, 1.8] [2.0, 2.3, 2.5] [3.0, 3.6, 3.7]
Question: I have a frequently occurring problem where I have two arrays of the same length: one with values and one with dynamic decay factors; and wish to calculate a vector of the decayed cumulative sum at each position. Using a Python loop to express the desired recurrence we have the following: c = np.empty_like(x) c[0] = x[0] for i in range(1, len(x)): c[i] = c[i-1] * d[i] + x[i]  The Python code is very clear and readable, but slows things down significantly. I can get around this by using Numba to JIT-compile it, or Cython to precompile it. If the discount factors were a fixed number (which is not the case), I could have used SciPy's signals library and do an lfilter (see https://stackoverflow.com/a/47971187/1017986). Is there a more "Numpythonic" way to express this without sacrificing clarity or efficiency?

Answer: Mathematically speaking, this is a first-order non-homogeneous recurrence relation with variable coefficients. Please note that coefficients (in your case, values in d[1:]) must be different from 0. Here is a way to solve your recurrence relation using NumPy functions. Note that d[0] is supposed to be equal to 1 (your algorithm is not using it anyway). In this solution, as you can see, the values in c do not depend on previous values in c. I.e., c[i] does not depend on c[i-1]. g = np.insert(x[1:], 0, 0) pd = np.cumprod(d) c = pd * (x[0] + np.cumsum(g / pd))  Let's make an example. Here I define two functions, one that computes the recurrence relation with your code, and one that uses NumPy: def rec(d, x): c = [x[0]] for i in range(1, len(x)): c.append(d[i] * c[-1] + x[i]) return np.array(c) def num(d, x): g = np.insert(x[1:], 0, 0) pd = np.cumprod(d) return pd * (x[0] + np.cumsum(g / pd))  Here are some usage examples: >>> rec(np.array([3, 4, 5.2, 6.1]), np.array([2, 3.2, 4, 5.7])) array([ 2. , 11.2 , 62.24 , 385.364]) >>> num(np.array([3, 4, 5.2, 6.1]), np.array([2, 3.2, 4, 5.7])) array([ 2. , 11.2 , 62.24 , 385.364]) >>> rec(np.array([3, 4.3, 5.2]), np.array([6.7, 7.1, 1.2])) array([ 6.7 , 35.91 , 187.932]) >>> num(np.array([3, 4.3, 5.2]), np.array([6.7, 7.1, 1.2])) array([ 6.7 , 35.91 , 187.932])  If you're interested in the mathematical details behind this answer (which are out of scope here), check this Wikipedia page. Edit This solution, while mathematically correct, might introduce numerical issues during the computation (because of the numpy.cumprod function), especially if d contains many values that are close to 0.
Question: The class scipy.interpolate.interp1d has a message that reads:  Legacy This class is considered legacy and will no longer receive updates. This could also mean it will be removed in future SciPy versions.  I'm not sure why this is marked for deprecation, but I use it heavily in my code to return an interpolating function, which numpy.interp() does not handle. What is the proper replacement for scipy.interpolate.interp1d?

Answer: According to the documentation:  interp1d is considered legacy API and is not recommended for use in new code. Consider using more specific interpolators instead.  https://docs.scipy.org/doc/scipy/tutorial/interpolate/1D.html#legacy-interface-for-1-d-interpolation-interp1d it is asking to: Consider using more specific interpolators instead  It also says  The ‘cubic’ kind of interp1d is equivalent to make_interp_spline, and the ‘linear’ kind is equivalent to numpy.interp while also allowing N-dimensional y arrays.   Another set of interpolations in interp1d is nearest, previous, and next, where they return the nearest, previous, or next point along the x-axis. Nearest and next can be thought of as a special case of a causal interpolating filter.
Question: I want to make a graph that contains bars, the value of the bars noted above them and a line on the secondary axis, but I can't configure the order of the elements the way I want where the bars would be the furthest element, followed by the line and then the texts, but so far I can only change the position of an entire axis and either the line goes over the text or is behind the bars. import matplotlib.pyplot as plt import seaborn as sns years = [2019, 2020, 2021, 2022, 2023] values1 = [1350, 1360, 1420, 1480, 1650] values2 = [57, 62, 60.5, 59.7, 62.3] fig, ax = plt.subplots(figsize=(6.2, 7.36), dpi = 150) bars = ax.bar(years, values1) heights = [] for bar in bars: bar.set_zorder(2) height = bar.get_height() heights.append(height) ax.annotate('{}'.format(round(height, 1)), xy=(bar.get_x() + bar.get_width() / 2, height), xytext=(0, 10), # 3 points vertical offset textcoords="offset points", ha='center', va='bottom', fontsize = 20).set_zorder(1) ax2 = ax.twinx() lineplot = sns.lineplot(x = years, y = values2, ax = ax2, color="gray")  the graph that the code produces: graph with line in fort of text annotation I tried changing the zorder on both axes and directly on the charts

Answer: Unfortunately respecting zorder across twinned axes is not possible. A workaround for annotations is to add them to the second axes, but using the data transform of the first. See the xycoords parameter added in the call to annotate here: import matplotlib.pyplot as plt years = [2019, 2020, 2021, 2022, 2023] values1 = [1350, 1360, 1420, 1480, 1650] values2 = [57, 62, 60.5, 59.7, 62.3] fig, ax = plt.subplots(figsize=(6.2, 7.36), dpi = 150) bars = ax.bar(years, values1) ax2 = ax.twinx() lineplot = ax2.plot(years, values2, ls="-", color="gray") for bar in bars: height = bar.get_height() ax2.annotate('{}'.format(round(height, 1)), xy=(bar.get_x() + bar.get_width() / 2, height), xytext=(0, 10), # 3 points vertical offset textcoords="offset points", ha='center', va='bottom', fontsize = 20, xycoords=ax.transData) plt.show()   Note that this can be simplified by using the bar_label method, which is a convenience wrapper for adding annotations to bars: import matplotlib.pyplot as plt years = [2019, 2020, 2021, 2022, 2023] values1 = [1350, 1360, 1420, 1480, 1650] values2 = [57, 62, 60.5, 59.7, 62.3] fig, ax = plt.subplots(figsize=(6.2, 7.36), dpi = 150) bars = ax.bar(years, values1) ax2 = ax.twinx() lineplot = ax2.plot(years, values2, ls="-", color="gray") ax2.bar_label(bars, fontsize=20, padding=10, xycoords=ax.transData) plt.show()
Question: I have a large dataframe and a list of many locations I need to set to a certain value. Currently I'm iterating over the locations to set the values one by one: import pandas as pd import numpy as np #large dataframe column_names = np.array(range(100)) np.random.shuffle(column_names) row_names = np.array(range(100)) np.random.shuffle(row_names) df = pd.DataFrame(columns=column_names, index=row_names) #index values to be set ix = np.random.randint(0, 100,1000) #column values to be set iy = np.random.randint(0, 100,1000) #setting the specified locations to 1, one by one for k in range(len(ix)): df.loc[ix[k], iy[k]]=1  This appears to be prohibitively slow. For the above example, on my machine, the last for loop takes 0.35 seconds. On the other hand, if I do df.loc[ix, iy]=1  only takes 0.035 seconds so it is ten times faster. Unfortunately, it does not give the correct result, as it sets all combinations of elements of ix and iy to 1. I was wondering whether there is a similarly fast way to set values of many locations at once, avoiding the iteration over the locations?

Answer: You can access the underlying numpy array with .values and use the position indices after conversion: cols = pd.Series(range(df.shape[1]), index=df.columns) idx = pd.Series(range(df.shape[0]), index=df.index) df.values[idx.reindex(ix), cols.reindex(iy)] = 1  Minimal example: # input df = pd.DataFrame(index=['a', 'b', 'c'], columns=['A', 'B', 'C']) ix = ['a', 'b', 'c'] iy = ['A', 'C', 'A'] # output A B C a 1 NaN NaN b NaN NaN 1 c 1 NaN NaN   previous answer df.values[ix, iy] = 1  Minimal example: df = pd.DataFrame(index=range(5), columns=range(5)) ix = [1, 2, 4] iy = [1, 3, 2] df.values[ix, iy] = 1  Output:  0 1 2 3 4 0 NaN NaN NaN NaN NaN 1 NaN 1 NaN NaN NaN 2 NaN NaN NaN 1 NaN 3 NaN NaN NaN NaN NaN 4 NaN NaN 1 NaN NaN
Question: I am currently using Beautifulsoup to parse the HTML code of a webpage. To get the text from an element, I use the ".text" attribute: soup.find('p', {'class': 'example'}).text  But the problem is that sometimes I get "\xa0" in the result: "some text «\xa0text\xa0»"  I tried using the "replace" function: soup = BeautifulSoup(driver.page_source.replace('\xa0', ' '), "lxml")  NOTE: I don't want to have to use a function for every single string I parse, I would like to have the soup already purged from those characters from the beginning.

Answer: The problem is that the HTML source probably contains &nbsp;, not the literal \xa0. Try replacing that instead, or as well. soup = BeautifulSoup( driver.page_source.replace( '&nbsp;', ' ').replace('\xa0', ' '), "lxml")
Question: My UnitTest test cases in a Django project run fine when running them directly from the Django utility (python manage.py test). However, when trying to run them from VSCode's test explorer, I get django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.. I found several similar questions that were already answered here on SO, but none of the solutions seem to work. Here's what I tried:  import my classes (e.g. models) within the SetUpTestData method and the actual test methods of the test class, rather than at the top of the file. This seemed to help a bit, in the sense that the tests now do show up in the test explorer (previously they didn't), but I still get an error when running them.  set the DJANGO_SETINGS_MODULE environment variable at the top of the file containing the test class:  import os os.environ.setdefault("DJANGO_SETTINGS_MODULE", "my_project.settings")   add a .env file to my root directory, containing DJANGO_SETINGS_MODULE = my_project.settings  add a configuration with purpose debug-test to launch.json:  "configurations": [ { "name": "Django test", "type": "python", "request": "launch", "program": "${workspaceFolder}\\manage.py", "args": [ "test" ], "purpose": ["debug-test"], "django": true, "justMyCode": true } ]   remove tests.py from my app folder, as my tests are stored in a separate folder (and apparently Django can't have both)   My folder structure looks like this: root |_ my_app |_ tests |_ __init__.py |_ test_models.py |_ my_project  For the sake of completeness, I have the following in settings.json: { "python.testing.unittestArgs": [ "-v", "-s", ".", "-p", "test_*.py" ], "python.testing.pytestEnabled": false, "python.testing.unittestEnabled": true }  Is there any way to make VSCode's test explorer play nice with Django and UnitTest?

Answer: You can try to do the following steps:  Create a __init__.py file in your tests directory (if it doesn't already exist), and add the following code:  import os import django os.environ.setdefault("DJANGO_SETTINGS_MODULE", "my_project.settings") django.setup()  These codes manually set the Django settings module and initializes Django.  In your settings.json, change the python.testing.unittestArgs to point to your tests directory:  { "python.testing.unittestArgs": [ "-v", "-s", "./my_app/tests", "-p", "test_*.py" ], "python.testing.pytestEnabled": false, "python.testing.unittestEnabled": true }  This tells VS Code to look for tests in the my_app/tests directory.
Question: # order.py class Order(Base): __tablename__ = "Order" id: Mapped[int] = mapped_column(primary_key=True) items: Mapped[List["Item"]] = relationship(back_populates="order") # item.py class Item(Base): __tablename__ = "Item" id: Mapped[int] = mapped_column(primary_key=True, autoincrement=True) order_id: Mapped[int] = mapped_column(ForeignKey("Order.id")) order: Mapped["Order"] = relationship(back_populates="items")  I'm using sqlalchemy 2.0.23 and not using plugins for mypy or flake8. Neither module should need to import the other since I'm using string relationships like "Item" in the Order class and "Order" in the relationship in the Item class. flake8 reports an F821 error on both files because of the undefined name. mypy reports a similar error on both. I can configure flake8 to ignore the F821. I'm not sure how I'd do similar in mypy. But these are important rules that shouldn't be turned off to get SQLAlchemy classes through linters. I want to keep my classes in separate files. Is there a way to correctly define them so that linters like these won't complain? Adding imports to both files quiets these linters, but results in a circular import problem so the code won't run.

Answer: mypy and flake8 are correct here, these warnings shouldn't be ignored. To resolve the circular import problem, you can use "typechecking-only import", i.e. an import statement wrapped in if TYPE_CHECKING block (TYPE_CHECKING constant is explained here) . Here's how it may look like: # order.py from typing import TYPE_CHECKING if TYPE_CHECKING: from .item import Item class Order(Base): __tablename__ = "Order" id: Mapped[int] = mapped_column(primary_key=True) items: Mapped[List["Item"]] = relationship(back_populates="order")  and # item.py from typing import TYPE_CHECKING if TYPE_CHECKING: from .order import Order class Item(Base): __tablename__ = "Item" id: Mapped[int] = mapped_column(primary_key=True, autoincrement=True) order_id: Mapped[int] = mapped_column(ForeignKey("Order.id")) order: Mapped["Order"] = relationship(back_populates="items")
Question: I need to read a .dat file in Python. The file has 3 columns in total and hundreds of rows. The second and third column contain two characters followed by a float that I would like to extract--the second column always starts with "SA" and the third column always starts with "SC". I am currently loading in the data and looping through each row to extract the values, but is there a better way to do it? Once the data is cleaned, I want to perform some calculations on the result, namely computing the average. Here is an example of two lines from the .dat file: 9:01:15 SA7.998 SC7.968 9:01:16 SA7.998 SC7.968  Here is the code I am currently using. import numpy as np import os.path from statistics import mean time=[] s_1=[] s_2=[] s1=[] s2=[] r1=[] r2=[] avgg=[] # Reading data from file with open('serial_2.dat','r') as f: dat=f.readlines() for i in dat: y=i.split() # cleaning and getting columns without spaces time.append(y[0]) s1.append(y[1]) s2.append(y[2]) #getting only numbers without strings (SA and SC) for counter in (range(0,len(s1))): S_1=s1[counter] r1.append(S_1[2:]) r1_f=np.array(r1, dtype='float32') S_2=s2[counter] r2.append(S_2[2:]) r2_f=np.array(r2, dtype='float32') avgg=r1_f+r2_f/2 print(np.mean(avgg)) 

Answer: You can use pandas to do that: #! pip install pandas import pandas as pd import numpy as np df = pd.read_csv('serial_2.dat', sep='\s+', header=None, names=['time', 's1', 's2']) df['s1'] = df['s1'].str.extract('^[\D]+(.*)').astype(float) df['s2'] = df['s2'].str.extract('^[\D]+(.*)').astype(float)  Output: >>> df time s1 s2 0 9:01:15 7.998 7.968 1 9:01:16 7.998 7.968 >>> df.dtypes time object s1 float64 s2 float64 dtype: object  If you have always 2 characters in s1 and s2 columns, you can avoid regex and strip the first two characters: df['s1'] = df['s1'].str[2:].astype(float) df['s2'] = df['s2'].str[2:].astype(float)  Then compute the global average: # With pandas >>> df[['s1', 's2']].mean().mean() 7.983 # With numpy >>> np.mean(df[['s1', 's2']]) 7.983  You can also compute the row average: df['avg'] = df[['s1', 's2']].mean(axis=1) print(df) # Output time s1 s2 avg 0 9:01:15 7.998 7.968 7.983 1 9:01:16 7.998 7.968 7.983
Question: I'm attempting to use tesserocr in my Python project, but when I try to import it, I'm getting [No module named 'tesserocr'], I run into an ImportError. The error message points to a missing symbol related to the Tesseract library. Here is the full error message:   ImportError: dlopen(/Volumes/WorkSpace/Backend/Reveratest/revera_api/venv/lib/python3.8/site-packages/tesserocr.cpython-38-darwin.so, 0x0002): symbol not found in flat namespace '__ZN9tesseract11TessBaseAPID1Ev'    Here's what I've done so far:  Installed tesseract using brew: brew install tesseract  Installed tesserocr via pip within a virtual environment: pip install tesserocr  Confirmed that tesseract command-line works correctly. I'm running this on macOS M1 and my Python version is 3.8.   How can I resolve this issue so that I can use tesserocr in my project? Are there specific paths or configurations I'm possibly missing? Any assistance or pointers as to what might be causing this error would be greatly appreciated!

Answer: I ran into a problem when trying to work with the tesserocr package, but I managed to find a fix and wanted to share it for anyone who might be experiencing the same issue. The error was resolved by first completely uninstalling the tesserocr package and then reinstalling it using the --no-binary option with pip. Here are the commands I used:  Uninstall tesserocr: pip uninstall tesserocr Reinstall tesserocr without using binary packages: pip install --no-binary :all: tesserocr  After performing these two steps, the error was gone, and tesserocr was working as expected. I hope this helps someone who faces the same problem!