eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: The pandas doc on loc states that it can be used with boolean arrays, more specifically it states the following: "Allowed inputs are: ... A boolean array (any NA values will be treated as False)." My question: How can u create a boolean array containing NA-values? I mean: A numpy bool array can't contain Nans and if we interpret this more liberal as stating "a list containing boolean values and na" then loc throws exceptions, e.g.: d_test = pd.DataFrame({"id": [1,2,3,5], "q1": [1,4,4,2], "q2": [4,np.nan,9,0]}, index=["a","b","c","d"]) t1 = [True,False,False,np.nan] d_test.loc[t1] # KeyError #same with None: t1 = [True,False,False,None]  So my question: How is this sentence to be interpreted?

Answer: I think they mean a BooleanArray (which can be created with pd.array) :  Array of boolean (True/False) data with missing values.  t1 = [True,False,False,np.nan] out = d_test.loc[pd.array(t1, dtype="boolean")]  So, since np.nan is treated as False, only the first row is being selected by the mask. Output : print(out) id q1 q2 a 1 1 4.00
Question: VS Code is missing "Python: Select Linter" option when trying to run a command (F1 or Ctrl/CMD+Shift+P). Previously, (2 days ago) this option was available. Is there a way to restore this option or is there a new alternative? I have tried creating new VS Code profile and installing just Python extension - still no mentioned option.

Answer: Rolling back Python extension to previous version (v2023.16.0) restores the option.
Question: In the What's New document for 3.12 there is a point:  sum() now uses Neumaier summation to improve accuracy and commutativity when summing floats or mixed ints and floats. (Contributed by Raymond Hettinger in gh-100425.)  But also there is a math.fsum function that is intended to be used for exact floating-point number summation. I guess it should be using some similar algorithm. I tried different cases on 3.11 and 3.12. On 3.11 sum() gives less precise results as expected. But on 3.12 they always return the same good results and I wasn't able to find a case when it differs. Which one should I use? Is there cases left when one should prefer math.fsum() over built-in sum()?

Answer: Here is the GitHub issue from your question: https://github.com/python/cpython/issues/100425 According to the detailed discussion there, sum has been about 10x faster that fsum. This new optimisation is supposed to run in parallel to the summation, making it essentially free. sum is now more accurate than before. fsum should still be even more accurate in certain cases and slower. That seems to be its point. If the accuracy of sum results is acceptable to you or you need its speed, then you can go with it.
Question: I have been using these two lines to get stock data: df = yf.download(tickers, group_by="ticker") d = {idx: gp.xs(idx, level=0, axis=1) for idx, gp in df.groupby(level=0, axis=1)}  from https://stackoverflow.com/a/66989947/21955590. While it works fine, I can't figure out how to avoid getting the warning in the title. How should this code be rewritten with df.T.groupby without the axis=1 without causing an error?

Answer: Edit: In fact, you don't need groupby: d = {ticker: df[ticker] for ticker in df.columns.levels[0]}   There is a discussion about groupby(axis=1) on github. Another solution is to stack the ticker level: d = {idx: gp.xs(idx, level=1) for idx, gp in df.stack(level=0).groupby(level=1)}  Output: >>> d {'AAPL': Open High Low Close Adj Close Volume Date 1980-12-12 0.128348 0.128906 0.128348 0.128348 0.099449 469033600.0 1980-12-15 0.122210 0.122210 0.121652 0.121652 0.094261 175884800.0 1980-12-16 0.113281 0.113281 0.112723 0.112723 0.087343 105728000.0 1980-12-17 0.115513 0.116071 0.115513 0.115513 0.089504 86441600.0 1980-12-18 0.118862 0.119420 0.118862 0.118862 0.092099 73449600.0 ... ... ... ... ... ... ... 2023-10-02 171.220001 174.300003 170.929993 173.750000 173.750000 52164500.0 2023-10-03 172.259995 173.630005 170.820007 172.399994 172.399994 49594600.0 2023-10-04 171.089996 174.210007 170.970001 173.660004 173.660004 53020300.0 2023-10-05 173.789993 175.449997 172.679993 174.910004 174.910004 48527900.0 2023-10-06 173.800003 177.990005 173.179993 177.490005 177.490005 57224100.0 [10795 rows x 6 columns], 'MSFT': Open High Low Close Adj Close Volume Date 1986-03-13 0.088542 0.101563 0.088542 0.097222 0.060396 1.031789e+09 1986-03-14 0.097222 0.102431 0.097222 0.100694 0.062553 3.081600e+08 1986-03-17 0.100694 0.103299 0.100694 0.102431 0.063632 1.331712e+08 1986-03-18 0.102431 0.103299 0.098958 0.099826 0.062014 6.776640e+07 1986-03-19 0.099826 0.100694 0.097222 0.098090 0.060936 4.789440e+07 ... ... ... ... ... ... ... 2023-10-02 316.279999 321.890015 315.179993 321.799988 321.799988 2.057000e+07 2023-10-03 320.829987 321.390015 311.209991 313.390015 313.390015 2.103350e+07 2023-10-04 314.029999 320.040009 314.000000 318.959991 318.959991 2.072010e+07 2023-10-05 319.089996 319.980011 314.899994 319.359985 319.359985 1.696560e+07 2023-10-06 316.549988 329.190002 316.299988 327.260010 327.260010 2.564550e+07 [9469 rows x 6 columns]}
Question: I have a DataFrame like this: index B 0 1 1 2 2 5 3 6 4 7 5 10  And i need to merge rows where the difference is less than or equal 2, select the line with the smaller value and set count merges The result should be like this : index B count 0 1 2 1 5 3 2 10 1  How can this be solved using pandas?

Answer: Very similar to @AndrejKesely, just shorter syntax, using named aggregations: df.groupby(df["B"].diff().gt(2).cumsum(), as_index=False).agg(B=("B", "first"), count=("B", "count"))  Output:  B count 0 1 2 1 5 3 2 10 1
Question: I have a two numpy arrays a, B like this. >> a [1 2 3] >> type(a) <class 'numpy.ndarray'> >> B [[1 2 3] [2 2 7] [3 4 6]] >> type(B) <class 'numpy.ndarray'>  I want to do the matrix multiplication like a * B * a_transpose which is (1*3)*(3*3)*(3*1) type matrix multiplication which should result in (1*1). How do I do this in numpy?

Answer: a.T is the transpose of matrix a temp = np.dot(a, B) # a * B final= np.dot(temp, a.T) #(a * B) * a_transpose  Answer for your example is 155
Question: I'm a Python newbie working on my first Turtle Graphics program. This is what I have at the moment import turtle def draw_square(): window = turtle.Screen() window.bgcolor("red") brad = turtle.Turtle() brad.forward(100) window.exitonclick() draw_square()  This code works perfectly on my friend's Windows laptop:  However, problem arises when we tried to run it on my Macbook. The issue we were facing is that Python Turtle Graphics displays nothing as below:  Why can't my program run normally on MacOS? And how can I resolve this issue?

Answer: This is a known issue that arises due to an incompatibility between MacOS and Tkinter, on which Turtle is based. In the terminal, use the following code: brew install tcl-tk env \ PATH="$(brew --prefix tcl-tk)/bin:$PATH" \ LDFLAGS="-L$(brew --prefix tcl-tk)/lib" \ CPPFLAGS="-I$(brew --prefix tcl-tk)/include" \ PKG_CONFIG_PATH="$(brew --prefix tcl-tk)/lib/pkgconfig" \ CFLAGS="-I$(brew --prefix tcl-tk)/include" \ PYTHON_CONFIGURE_OPTS="--with-tcltk-includes='-I$(brew --prefix tcl-tk)/include' --with-tcltk-libs='-L$(brew --prefix tcl-tk)/lib -ltcl8.6 -ltk8.6'" \ pyenv install 3.8.13
Question: I have a structure of files and folders like this: package1/ p1.py package2/ p2.py  Contents of package1/p1.py: def p1fun(): print("p1fun")  Contents of package1/package2/p2.py: import package1.p1 if __name__ == '__main__': package1.p1.p1fun()  Now, when I do python -m package1.package2.p2, I get the correct result = p1fun. When I do python package1/package2/p2.py, I get: Traceback (most recent call last): File "/home/marcin/projects/test/package1/package2/p2.py", line 1, in <module> import package1.p1 ModuleNotFoundError: No module named 'package1'  How the import mechanism differs in these two scenarios? How to make the "script" scenario import properly?

Answer: You just have to add the directory containing package1 to sys.path. A simple way is to set the PYTHONPATH environment variable to .: export PYTHONPATH='.' # on Linux or other Unix-like set PYTHONPATH=. # on Windows  Underlying cause: As explained by @KlausD. , python automatically adds the path of the script passed on the command line, or the current path when you start a module with python -m. A simple way to see it is to print sys.path close to the beginning of p2.py: import package1.p1 import sys print(sys.path) if __name__ == '__main__': package1.p1.p1fun()  For python -m package1.package2.p2 you can see that the first element of the list is the current folder, for python package1/package2/p2.py you will instead have: /path/to/current/package1/package2
Question: I would like to define a sort of "wrapper" Generic Type, say MyType[T], so that it has the same type interface as the wrapped type. from typing import Generic, TypeVar T = TypeVar("T") class MyType(Generic): pass # what to write here?  So, as an example, when I have a type MyType[int], the type-checker should treat it as if it was an int type. Is that possible? If so, how?

Answer: To confirm, you're looking at wanting the expression MyType[T] to mean to a static type checker "A subclass of MyType and T", such that a declaration class MyType: attr: object  will result in the following (e.g. using mypy and int.conjugate as an example): >>> reveal_type(MyType[int].conjugate) # def (self: builtins.int) -> builtins.int >>> obj: MyType[int] = MyType[int]() >>> reveal_type(obj.attr) # builtins.object   No, this isn't a supported idiom in Python typing. Your use case should be covered by a future implementation of intersection types with a proposed syntax MyType & T instead. To a static type checker, the syntax MyType[T] means and only means that MyType is a generic type (including generic structural types), and the API that MyType has is derived from any metaclasses, any base classes, and anything declared under class MyType's body.  Any type parameterisation (e.g. passing int to T in MyType[T]) only affects generic types in the class bases and statements in the class body;  There is nothing you can write in a class body which tells a static type checker that the class inherits the API from another class, as this information is already given by the metaclass and base class(es);  <type>[T] is only well-defined as a statically-inferrable type when <type> is a user-defined generic type. The static types inferred from other arbitrary attempts to make <type>[T] a valid expression is unstable and implementation-defined, regardless of the runtime implementation. from typing import * T = TypeVar("T") class M(type): def __getitem__(cls, type_: T, /) -> T: ... class A(metaclass=M): ... class B: def __class_getitem__(cls, type_: T, /) -> T: ... class C(Generic[T]): ...  >>> A_int: TypeAlias = A[int] # mypy: "A" expects no type arguments, but 1 given # pyright: Expected no type arguments for class "A" >>> B_int: TypeAlias = B[int] # mypy: "B" expects no type arguments, but 1 given # pyright: <no messages> >>> C_int: TypeAlias = C[int] >>> >>> reveal_type(A[int]) # mypy: <overloads of `int.__new__`> pyright: Type of "A[int]" is "Type[int]" >>> reveal_type(B[int]) # mypy: The type "type[B]" is not generic and not indexable pyright: Type of "B[int]" is "Type[int]" >>> reveal_type(C[int]) # mypy: Revealed type is "def () -> __main__.C[builtins.int]" pyright: Type of "C[int]" is "Type[C[int]]"
Question: Suppose I have a pandas DataFrame with the following columns and data:  user time session time_diff 0 21.0 2022-12-16 14:03:08 5 NaN 1 21.0 2022-12-16 14:03:10 5 2.0 2 21.0 2022-12-16 14:03:12 6 2.0 3 21.0 2022-12-16 14:03:13 6 1.0 4 21.0 2022-12-28 14:49:54 16 1039601.0 5 30.0 2022-12-16 14:03:16 5 1039598.0 6 30.0 2022-12-16 14:03:18 5 2.0 7 30.0 2022-12-16 14:03:20 6 2.0  I would like to select those rows where for the same user and session the time difference (time_diff column in seconds) is less than some threshold (10 seconds, for example). Which would result in the following output:  user time session time_diff 1 21.0 2022-12-16 14:03:10 5 2.0 3 21.0 2022-12-16 14:03:13 6 1.0 6 30.0 2022-12-16 14:03:18 5 2.0  I could probably iterate through each row and select records where id = id of the preceding row and session = session of the preceding row but I believe this is not the most optimal approach. df.groupby(['user', 'session']).filter(lambda x: (x.time_diff <= 10).any())  also does not produce the expected result.

Answer: Option 1  Group by ["user", "session"] (df.groupby) and check .diff for column "time". For the resulting Series check < 10 seconds using Series.lt. Finally, use the resulting Series (populated with True & False) for boolean indexing to retrieve the desired subset.  out = df[df.groupby(["user", "session"])['time'].diff() .lt(pd.Timedelta('00:00:10'))] out user time session time_diff 1 21.0 2022-12-16 14:03:10 5 2.0 3 21.0 2022-12-16 14:03:13 6 1.0 6 30.0 2022-12-16 14:03:18 5 2.0   Option 2 (Assuming your data is properly sorted on user and session.)  Apply Series.diff to column "time" and check < 10 seconds. Now, also check whether row values for user and session are both equal to (df.eq) the values in the previous row (df.shift). Use df.all row-wise to get False for all shifts to a new group. Finally, apply boolean indexing to select from the df where both conditions are True (using the bitwise operator &).  out2 = df[df.time.diff().lt(pd.Timedelta('00:00:10')) & df[['user','session']].eq(df[['user','session']].shift(1)).all(axis=1)] out2.equals(out) # True   Performance comparison Option 1 will be fastest. Both will be faster than the solution offered by @AndrejKesely. # opt1: 1.75 ms ± 176 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each) # opt2: 3.1 ms ± 133 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) # AK: 7.02 ms ± 300 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)   Data used import pandas as pd import numpy as np data = {'user': {0: 21.0, 1: 21.0, 2: 21.0, 3: 21.0, 4: 21.0, 5: 30.0, 6: 30.0, 7: 30.0}, 'time': {0: '2022-12-16 14:03:08', 1: '2022-12-16 14:03:10', 2: '2022-12-16 14:03:12', 3: '2022-12-16 14:03:13', 4: '2022-12-28 14:49:54', 5: '2022-12-16 14:03:16', 6: '2022-12-16 14:03:18', 7: '2022-12-16 14:03:20'}, 'session': {0: 5, 1: 5, 2: 6, 3: 6, 4: 16, 5: 5, 6: 5, 7: 6}, 'time_diff': {0: np.nan, 1: 2.0, 2: 2.0, 3: 1.0, 4: 1039601.0, 5: 2, 6: 2.0, 7: 2.0}} df = pd.DataFrame(data) df['time'] = pd.to_datetime(df['time'])
Question: I have a df like below:  name1 name2 value 0 A B 1300 1 A C 150 2 A D 300 3 B C 450 4 B D 200 5 C D 300  I tried to pivot the table to plot a corr-like heatmap based on the value column: table = df.pivot(columns='name1', index='name2', values='value') table  The result is: name1 A B C name2 B 1300.0 NaN NaN C 150.0 450.0 NaN D 300.0 200.0 300.0  How can I create a square matrix that contains A-D in columns and rows with value of name1 name2 is same to name2 name1 (A/B = B/A = 1300)? Desired output:  A B C D A NaN 1300.0 150.0 300.0 B 1300.0 NaN 450.0 200.0 C 150.0 450.0 NaN 300.0 D 300.0 200.0 300.0 NaN  Reproducible input: import pandas as pd products_list = [['A', 'B', 1300], ['A', 'C', 150], ['A', 'D', 300], ['B', 'C', 450], ['B', 'D', 200], ['C', 'D', 300]] df = pd.DataFrame(products_list, columns=['name1', 'name2', 'value']) 

Answer: You can combine_first the transpose after pivoting: # table = df.pivot(columns='name1', index='name2', values='value') out = table.combine_first(table.T)  Output:  A B C D A NaN 1300.0 150.0 300.0 B 1300.0 NaN 450.0 200.0 C 150.0 450.0 NaN 300.0 D 300.0 200.0 300.0 NaN  Alternatively, but less elegant, swap the columns and concat before the pivot: out = (pd.concat([df, df.rename(columns={'name1': 'name2', 'name2': 'name1'})]) .pivot(columns='name1', index='name2', values='value') )
Question: Recently, my code involving matplotlib.pyplot suddenly stopped working on all my machines (Ubuntu 22.04 LTS). I tried a simple import and got the following error: $ python Python 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import matplotlib.pyplot as plt Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/lib/python3.10/dist-packages/matplotlib/pyplot.py", line 66, in <module> from matplotlib.figure import Figure, FigureBase, figaspect File "/usr/local/lib/python3.10/dist-packages/matplotlib/figure.py", line 43, in <module> from matplotlib import _blocking_input, backend_bases, _docstring, projections File "/usr/local/lib/python3.10/dist-packages/matplotlib/projections/__init__.py", line 58, in <module> from mpl_toolkits.mplot3d import Axes3D File "/usr/lib/python3/dist-packages/mpl_toolkits/mplot3d/__init__.py", line 1, in <module> from .axes3d import Axes3D File "/usr/lib/python3/dist-packages/mpl_toolkits/mplot3d/axes3d.py", line 23, in <module> from matplotlib import _api, cbook, docstring, _preprocess_data ImportError: cannot import name 'docstring' from 'matplotlib' (/usr/local/lib/python3.10/dist-packages/matplotlib/__init__.py)  I am not sure what caused the problem, and how to diagnose or fix it. The matplotlib package is installed using pip as root, as I need it to be available to all users by default. Has anyone encountered a similar issue and know how to fix it?

Answer: As pointed out in comments by @Imsteffan, and the linked bug reports here and here, the issue happens because:  ... docstring was removed after a deprecation cycle of 2 releases and changed to be private (_docstring) However, the line that is erroring in mplot3d/axes3d.py was updated #22148, indicating that you have a version of mpl that it is picking up that is <3.6  This is the case in my OS. It turns out that the apt package python3-matplotlib is also installed as the dependency of another package (qgis). So there are two versions of matplotlib, from pip and from apt, respectively. The solution is to remove one of the two versions. In the linked bug report, the suggestion is to remove the apt version: sudo apt remove python3-matplotlib  But in my case, I can't remove the application that depends on the apt version. So I removed the pip version instead. And then the import works as expected. $ sudo pip uninstall matplotlib Found existing installation: matplotlib 3.8.0 Uninstalling matplotlib-3.8.0: Would remove: /usr/local/lib/python3.10/dist-packages/matplotlib-3.8.0.dist-info/* /usr/local/lib/python3.10/dist-packages/matplotlib/* /usr/local/lib/python3.10/dist-packages/mpl_toolkits/axes_grid1/* /usr/local/lib/python3.10/dist-packages/mpl_toolkits/axisartist/* /usr/local/lib/python3.10/dist-packages/mpl_toolkits/mplot3d/* /usr/local/lib/python3.10/dist-packages/pylab.py Proceed (Y/n)? Successfully uninstalled matplotlib-3.8.0
Question: A dataframe has two columns. One has a single integer per row. The other has a string of multiple integers, separated by ',', per row: import pandas as pd duck_ids = ["1, 4, 5, 7", "3, 11, 14, 27"] ducks_of_interest = [4,15] duck_df = pd.DataFrame( { "DucksOfInterests": ducks_of_interest, "DuckIDs": duck_ids } ) print(f"The starting dataframe:\n{duck_df}") DucksOfInterests DuckIDs 0 4 1, 4, 5, 7 1 15 3, 11, 14, 27  A new column is required that returns a True if the Duck of Interest is within the set of Duck IDs. This is attempted using a simple lambda function with the .apply method: duck_df['DoIinDIDs'] = duck_df.apply(lambda x: str(x['DuckIDs']) in [x['DucksOfInterests']], axis=1)  This was expected to return a True for the first row, as 4 is a number in "1, 4, 5, 7", and False for the second row. However, the result is False for both rows: print(f"The dataframe with the additional column:\n{duck_df}") DucksOfInterests DuckIDs DoIinDIDs 0 4 1, 4, 5, 7 False 1 15 3, 11, 14, 27 False  What is the error in the code or the approach?

Answer: You were almost there but unnecessarily used a list and swapped the names: duck_df['DoIinDIDs'] = duck_df.apply(lambda x: str(x['DucksOfInterests']) in x['DuckIDs'], axis=1)  Output:  DucksOfInterests DuckIDs DoIinDIDs 0 4 1, 4, 5, 7 True 1 15 3, 11, 14, 27 False  Note, however, that this approach might fail as you rely on the whole string and 4 would be found in 1, 14, 20. You can instead split the string: duck_df['DoIinDIDs'] = duck_df.apply(lambda x: str(x['DucksOfInterests']) in x['DuckIDs'].split(', '), axis=1)  Finally, as apply on axis=1 is slow, you can replace the whole thing by a list comprehension: duck_df['DoIinDIDs'] = [str(a) in b.split(', ') for a, b in zip(duck_df['DucksOfInterests'], duck_df['DuckIDs'])]
Question: So I have written this piece of code which is meant to read a csv file and write the data to a dictionary with keys "key1" "key2" "key3" "key4" with values given as a list comprised of data from columns of the csv file but with some modifications: import numpy as np import csv #code ... #code with open(file,'r') as fl: csv_file=csv.DictReader(fl) fdata={ 'key1':[data['THING'] for data in csv_file], 'key2':[f"{data['STR1']}, {data['STR2']}, {data['STR3']}" for data in csv_file], 'key3':[np.array([data['FL1'],data['FL2']],dtype=float) for data in csv_file], 'key4':[int(data['POPULATION']) for data in csv_file] }  The problem is that all the list come up empty except for the fist one, the one with key value "key1" and even when take out the typecast in "key4" it does not work, initially i thought it had to do with numpy, type casting or the string formatting but I don't what is wrong, online there are a lot of examples like this and using print() debugging shows that the data is being read. I don't know what to do here. Also I'm certain the keys for the csv file "THING", "STR1", "STR2", "STR3", "FL1", "FL2" are correct. Another thing is that the csv file is large but not millions of rows long,

Answer: This looks like it's to do with how your csv.DictReader object is being iterated over. When you do a list comprehension over the csv_file object for key1, you exhaust the iterator. This means that when you try to do the list comprehension for key2, there's no data left in csv_file to iterate over, and so on for the subsequent keys. Instead, try reading the CSV data into a list first, and then perform your list comprehensions on this list. Try something like this: import numpy as np import csv with open(file, 'r') as fl: csv_file = csv.DictReader(fl) all_data = list(csv_file) fdata = { 'key1': [data['THING'] for data in all_data], 'key2': [f"{data['STR1']}, {data['STR2']}, {data['STR3']}" for data in all_data], 'key3': [np.array([data['FL1'], data['FL2']], dtype=float) for data in all_data], 'key4': [int(data['POPULATION']) for data in all_data] }  By storing all the rows in all_data first, you ensure that you can iterate over the rows multiple times without exhausting the iterator.