eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: everyone. I've been learning floating-point truncation errors recently. But I found print(np.half(500.2)) and print(f"{np.half(500.2)}") yield different results. Here are the logs I got in IPython. In [11]: np.half(500.2) Out[11]: np.float16(500.2) In [12]: print(np.half(500.2)) 500.2 In [13]: print(f"{np.half(500.2)}") 500.25   I use half.hpp in c++ to compare results with numpy. It seems that 500.2 should be truncated into 500.25 instead of itself. In binary formats, 500.0 is 0b0_01000_1111010000. So the next float16 number should be 0b_01000_1111010001, which is 500.25 in deximal format.  So what makes print(np.half(500.2)) differs from print(f"{np.half(500.2)}")? Hope to see your answers.

Answer: print calls __str__, while an f-string calls __format__. __format__ with an empty format spec is usually equivalent to __str__, but not all types implement it that way, and numpy.half is one of the types that implements different behavior: In [1]: import numpy In [2]: x = numpy.half(500.2) In [3]: str(x) Out[3]: '500.2' In [4]: format(x, '') Out[4]: '500.25'
Question: I have a DataFrame with 3 column of zeroes and ones corresponding to 3 different classes. I want to get a single series of zeroes, ones, and twos depending of the class of the entry (0 for the first class, 1 for the second one and 2 for the third one): >>> results.head() HOME_WINS DRAW AWAY_WINS ID 0 0 0 1 1 0 1 0 2 0 0 1 3 1 0 0 4 0 1 0  What I want : >>> results.head() SCORE ID 0 2 1 1 2 2 3 0 4 1 

Answer: Multiply by a dictionary, sum and convert to_frame: d = {'HOME_WINS': 0, 'DRAW': 1, 'AWAY_WINS': 2} out = df.mul(d).sum(axis=1).to_frame(name='SCORE')  Or using a dot product: d = {'HOME_WINS': 0, 'DRAW': 1, 'AWAY_WINS': 2} out = df.dot(pd.Series(d)).to_frame(name='SCORE')  Or, if there is exactly one 1 per row, with from_dummies: d = {'HOME_WINS': 0, 'DRAW': 1, 'AWAY_WINS': 2} out = pd.from_dummies(df)[''].map(d).to_frame(name='SCORE')  Output:  SCORE ID 0 2 1 1 2 2 3 0 4 1
Question: Understandable and expected (tz-aware): import datetime import numpy as np import pandas as pd aware = pd.DatetimeIndex(["2024-11-21", "2024-11-21 12:00"], tz="UTC") eod = datetime.datetime.combine(aware[-1].date(), datetime.time.max, aware.tz) aware, eod, np.concat([aware, [eod]])  returns (DatetimeIndex(['2024-11-21 00:00:00+00:00', '2024-11-21 12:00:00+00:00'], dtype='datetime64[ns, UTC]', freq=None), datetime.datetime(2024, 11, 21, 23, 59, 59, 999999, tzinfo=datetime.timezone.utc), array([Timestamp('2024-11-21 00:00:00+0000', tz='UTC'), Timestamp('2024-11-21 12:00:00+0000', tz='UTC'), datetime.datetime(2024, 11, 21, 23, 59, 59, 999999, tzinfo=datetime.timezone.utc)], dtype=object))  note Timestamps (and a datetime) in the return value of np.concat. Unexpected (tz-naive): naive = pd.DatetimeIndex(["2024-11-21", "2024-11-21 12:00"]) eod = datetime.datetime.combine(naive[-1].date(), datetime.time.max, aware.tz) naive, eod, np.concat([naive, [eod]])  returns (DatetimeIndex(['2024-11-21 00:00:00', '2024-11-21 12:00:00'], dtype='datetime64[ns]', freq=None), datetime.datetime(2024, 11, 21, 23, 59, 59, 999999), array([1732147200000000000, 1732190400000000000, datetime.datetime(2024, 11, 21, 23, 59, 59, 999999)], dtype=object))  note intergers (and a datetime) in the return value of np.concat.  why do I get integers in the concatenated array for a tz-naive index? how do I avoid it? I.e., how do I append EOD to a tz-naive DatetimeIndex?  PS. Interestingly enough, at the numpy level the indexes are identical: np.testing.assert_array_equal(aware.values, naive.values) 

Answer: From Data type promotion in NumPy  When mixing two different data types, NumPy has to determine the appropriate dtype for the result of the operation. This step is referred to as promotion or finding the common dtype. In typical cases, the user does not need to worry about the details of promotion, since the promotion step usually ensures that the result will either match or exceed the precision of the input.  np.concat() accepts a casting keyword argument (casting="same_kind" default). If using casting='no' fails naive_no = np.concat([naive, [eod]], casting='no') TypeError: Cannot cast array data from dtype('<M8[ns]') to dtype('O') according to the rule 'no'  See Array-protocol type strings. In both cases the type is object naive_sk = np.concat([naive, [eod]], casting='same_kind') print(naive_sk.dtype, naive_sk)  Result object [1732147200000000000 1732190400000000000 datetime.datetime(2024, 11, 21, 23, 59, 59, 999999, tzinfo=<DstTzInfo 'America/New_York' LMT-1 day, 19:04:00 STD>)]  python 3.9 pandas 2.2.2
Question: This is supposed to be the absolute minimum Hello World using SWIG, C, and setuptools. But the following exception is raised when the module is imported: >>> import hello Traceback (most recent call last): File "<python-input-0>", line 1, in <module> import hello ImportError: dynamic module does not define module export function (PyInit_hello)  This is the directory structure: README.md pyproject.toml src src/hc src/hc/hello.c src/hc/hello.i  Here is the pyproject.toml [build-system] requires = ["setuptools>=75.6"] build-backend = "setuptools.build_meta" [project] name = "helloc" version = "0.0.1" authors = [ { name = "Foo" } ] description = "Hello world SWIG C" readme = "README.md" requires-python = ">=3.13" classifiers = [ "Development Status :: 1 - Planning", "License :: Public Domain", "Natural Language :: English", "Operating System :: OS Independent", "Programming Language :: Python", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3 :: Only", "Programming Language :: Python :: 3.13" ] [tool.setuptools] ext-modules = [ { name = "hello", sources = ["src/hc/hello.c", "src/hc/hello.i"] } ]  Here is hello.c #include <stdio.h> void say_hello() { printf("Hello, World!\n"); }  And here is the interface file: %module hello %{ void say_hello(); %} void say_hello(); 

Answer: Your extension module should be named _hello (notice the leading "_"). pyproject.toml: # ... [tool.setuptools] ext-modules = [ { name = "_hello", sources = ["src/hc/hello.c", "src/hc/hello.i"] } ]  Check:  [SO]: c program SWIG to python gives 'ImportError: dynamic module does not define init function' (@CristiFati's answer) contains the SWIG related details  [SO]: How to solve Python-C-API error "This is an issue with the package mentioned above, not pip."? (@CristiFati's answer) contains info about building with SetupTools
Question: This feels like a useful feature but haven't been able to find a setting / extension that offers this capability.

Answer: Just figured it out - you can use the "Go To" button in the top toolbar in vscode (pic below). You'll need the jupyter extension installed.
Question: The help doc for pandas.Series.nbytes shows the following example: s = pd.Series(['Ant', 'Bear', 'Cow']) s  0 Ant 1 Bear 2 Cow dtype: object s.nbytes  24 << end example >> How is that 24 bytes? I tried looking at three different encodings, none of which seems to yield that total. print(s.str.encode('utf-8').str.len().sum()) print(s.str.encode('utf-16').str.len().sum()) print(s.str.encode('ascii').str.len().sum())  10 26 10

Answer: Pandas nbytes does not refer to the bytes required to store the string data encoded in specific formats like UTF-8, UTF-16, or ASCII. It refers to the total number of bytes consumed by the underlying array of the Series data in memory. Pandas stores a NumPy array of pointers to these Python objects when using the object dtype. On a 64-bit system, each pointer/reference takes 8 bytes. 3 × 8 bytes =24 bytes. Link: nbyte source code Link: ndarray documentation
Question: Problem I have a NumPy array and need to identify repeated elements, marking the second occurrence and beyond as True, while keeping the first occurrence as False. For example, given the following array: np.random.seed(100) a = np.random.randint(0, 5, 10) # Output: [0 0 3 0 2 4 2 2 2 2]  I want to get the following output: [False True False True False False True True True True]  How can I achieve this using NumPy functions only, without using any loops or extra libraries? What did you try and what were you expecting? I was able to get it working with a loop, but I wanted to solve it using only NumPy functions. I tried implementing np.cumsum with masks, but I couldn’t make much progress. Here's the solution I came up with using one loop: np.random.seed(100) a = np.random.randint(0, 5, 10) print(a) uniques, first_indices = np.unique(a, return_index=True) all_occurrences = np.zeros_like(a, dtype=bool) for i in range(len(a)): all_occurrences[i] = np.any(a[:i] == a[i]) all_occurrences[first_indices] = False print(all_occurrences) 

Answer: To reveal that the problem is actually less complicated than it may seem at first glance, the question could be rephrased as follows: Mark all first occurrences of values with False. This leads to a bit of a simplified version of EuanG's answer¹: def find_repeated(a): mask = np.ones_like(a, dtype=bool) mask[np.unique(a, return_index=True)[-1]] = False return mask  Steps: (1) Initialize the result mask as an all-True array of appropriate shape. (2) Find the indices of the first occurrences in the given array a. (3) Only set these indices to False in the result mask. To make the code also work with n-dimensional arrays, we need to add an extra step of unraveling the result of np.unique(), as it returns the indices into the flattened given array a: def find_repeated(a): mask = np.ones_like(a, dtype=bool) mask[np.unravel_index(np.unique(a, return_index=True)[-1], a.shape)] = False return mask  In either case:  We can directly use the indices (np.unique(…, return_index=True)[-1]) for indexing the mask array. No need for catching the empty-array case here, as it is handled implicitly.  ¹) Yes, I find EuanG's answer perfectly acceptable as well. No, I did not downvote it.
Question: When running the snippet of example code below with pandas 2.2.3, I get an error saying KeyError: 'D' index = pd.MultiIndex.from_tuples( [('A', 1), ('A', 2), ('A', 3), ('B', 1), ('B', 2), ('B', 2)], names=['letter', 'number'] ) df = pd.DataFrame({'value': [10, 20, 30, 40, 50, 60]}, index=index) idx = pd.IndexSlice result = df.loc[idx[['A', 'D'], [1,2]], :]  Does pandas offer any alternatives for searching a multi-index with values that don't exist? If I run the same code using pandas 1.5.3, I get the expected value:  value letter number A 1 10 2 20 

Answer: When you run this code with pandas 1.5.3 you should in fact receive a FutureWarning:  FutureWarning: The behavior of indexing on a MultiIndex with a nested sequence of labels is deprecated and will change in a future version. series.loc[label, sequence] will raise if any members of 'sequence' or not present in the index's second level. To retain the old behavior, use series.index.isin(sequence, level=1)  (Note that it should read: "are not present".)  So, let's indeed use Index.isin to allow boolean indexing: m = (df.index.isin(['A', 'D'], level='letter') & df.index.isin([1, 2], level='number')) out = df.loc[m, :]  Output:  value letter number A 1 10 2 20  If you have many different conditions, you could consider creating a dictionary and use np.logical_and + reduce: dict_isin = { 'letter': ['A', 'D'], 'number': [1, 2] } m = np.logical_and.reduce( [df.index.isin(v, level=k) for k, v in dict_isin.items()] ) out2 = df.loc[m, :] out2.equals(out) # True
Question: I am attempting to use LangChain's ConversationSummaryBufferMemory and running into this error: pydantic.errors.PydanticUserError: `ConversationSummaryBufferMemory` is not fully defined; you should define `BaseCache`, then call `ConversationSummaryBufferMemory.model_rebuild()`.  This is what my code looks like: memory = ConversationSummaryBufferMemory( llm=llm, input_key="input", output_key="output", max_token_limit=args.get("max_tokens", DEFAULT_MAX_TOKENS), memory_key="chat_history", )  I am using langchain==0.3.7. Has anyone else encountered this?

Answer: pydantic library has been updated. pip install pydantic==2.9.2 should help
Question: I have several python packages that need to be installed on various os/environments. These packages have dependencies and some of them like Polars needs a different package depending on the OS, for example: polars-lts-cpu on MacOS (Darwin) and polars on all the other OS. I use setuptools to create a whl file, but the dependencies installed depend on the OS where the wheel file was created. Here is my code: import platform from setuptools import find_packages, setup setup( ... install_requires=["glob2>=0.7", "numpy>=1.26.4", "polars>=1.12.0" if platform.system() != "Darwin" else "polars-lts-cpu>=1.12.0"] ...)  As mentioned above, this code installs the version of Polars according to the OS where the wheel file was created, not according to where the package will be installed. How can I fix this?

Answer: Use declarative environment markers as described in PEP 496 and PEP 508:  install_requires=[ "polars>=1.12.0; platform_system!='Darwin'", "polars-lts-cpu>=1.12.0; platform_system=='Darwin'", ]
Question: My OS is Windows, newest version and all updated, I think the issue lies in the path or something to do with OneDrive. Using the code: file = "dataset.csv" with open(file, "w") as f: f.write(data)  I get the error: FileNotFoundError: [Errno 2] No such file or directory: 'dataset.csv', but this is an error I have never had before... I have enabled long paths in the Windows registry and rebooted and it still does not work. However, I have tried to run this code in different paths of my pc, and it does work if I run it in the path or with a shorter path than: C:\Users\user\OneDrive - UniversityX XXXXXXXXXX XXXXXXXX\XXXXXXXXX, where X is also part of the folder name as it is my unversity`s name. If I run it after that path it just stops working and outputs that error every time I run the code in VSCode. Update: This does not only not work in VSCode, but if I use matlab or any other program to save a file, it does not let me save the file either as it says that the file cannot be found. Does anyone know how I can fix this, please?

Answer: I found out that it was a Windows Security issue in the end... I enabled controlled folder access. To disable this follow these steps:  Settings > Update & Security (Windows 10) or Privacy & Security (Windows 11) > Windows Security > Virus & threat protection. Under Virus & threat protection settings, select Manage settings. Under Controlled folder access, select Manage Controlled folder access. Switch the Controlled folder access setting to Off.  This option apparently blocks any program from creating new files in certain directories.
Question: #Column X contains the suffix of one of V* columns. Need to put the value from V(X) in column Y. import pandas as pd import numpy as np # sample dataframes df = pd.DataFrame({ 'EMPLID': [12, 13, 14, 15, 16, 17, 18], 'V1': [2,3,4,50,6,7,8], 'V2': [3,3,3,3,3,3,3], 'V3': [7,15,8,9,10,11,12], 'X': [2,3,1,3,3,1,2] }) # Expected output: EMPLID V1 V2 V3 X Y 0 12 2 3 7 2 3 1 13 3 3 15 3 15 2 14 4 3 8 1 4 3 15 50 3 9 3 9 4 16 6 3 10 3 10 5 17 7 3 11 1 7 6 18 8 3 12 2 3  Example code I've tried (all got syntax error): df['Y'] = df[f"V + df['X']"] Any suggestion is appreciated. Thank you.

Answer: The canonical way would be to use indexing lookup, however since the column names and X values are a bit different, you first need to convert to string and prepend V: idx, cols = pd.factorize('V'+df['X'].astype(str)) df['Y'] = df.reindex(cols, axis=1).to_numpy()[np.arange(len(df)), idx]  Output:  EMPLID V1 V2 V3 X Y 0 12 2 3 7 2 3 1 13 3 3 15 3 15 2 14 4 3 8 1 4 3 15 50 3 9 3 9 4 16 6 3 10 3 10 5 17 7 3 11 1 7 6 18 8 3 12 2 3
Question: I'm tryring to setup a python wheel for my testrunner helper script to make it easyly acessible from everywhere on my Windows machine. Therefore I configure a console entry point in my setup.py. I can see the generated entry point in the entry_points.txt but if I'm trying to invoke my script I get the error message. No module named testrunner.__main__; 'testrunner' is a package and cannot be directly executed  My installer folder tree looks like this setup.py README.md LICENSE.txt testrunner/ __init__.py testrunner.py templates/ testDesc.json  The testrunner.py looks like this def runTests(): print("Hello World") #More not relevant code here if __name__ == '__main__': runTests()  The setup.py content from setuptools import find_packages, setup from pathlib import Path # read the contents of your README file this_directory = Path(__file__).parent long_description = (this_directory / "README.md").read_text() setup( name='testrunner', version='0.3.0', packages=find_packages(include=['testrunner']), description='C/C++ test runner', long_description=long_description, long_description_content_type='text/markdown', author='Me', license="Proprietary", license_files = ('LICENSE.txt',), entry_points={ 'console_scripts': ['testrunner = testrunner:main'] }, classifiers=[ 'Topic :: Software Development :: Build Tools', 'Topic :: Software Development :: Compilers', 'Private :: Do Not Upload', 'Operating System :: Microsoft :: Windows', 'Intended Audience :: Developers', 'Intended Audience :: Science/Research ', 'Intended Audience :: Education ', 'Natural Language :: English', 'License :: Other/Proprietary License', 'Programming Language :: Python :: 3', 'Programming Language :: Python :: 3.7', 'Programming Language :: Python :: 3.8', 'Programming Language :: Python :: 3.8', 'Programming Language :: Python :: 3.9', 'Programming Language :: Python :: 3.10', ], install_requires=[], package_data={'':['templates\\*.json']}, )  And finally the __init__.py from .testrunner import main, runTests  I build the wheel with the command: py -m pip wheel --no-deps -w dist . After installation with pip and checking the content in the site-packages directory executing py -m testrunner -h results in No module named testrunner.__main__; 'testrunner' is a package and cannot be directly executed

Answer:  No module named testrunner.__main__; 'testrunner' is a package and cannot be directly executed  You don't have __main__.py so you cannot do python -m testrunner. To fix the problem: echo "from .testrunner import runTests" >testrunner/__main__.py echo "runTests()" >>testrunner/__main__.py  The second problem:  from .testrunner import main, runTests  You don't have main in testrunner.py. Code if __name__ == '__main__': doesn't create any main. You only have runTests to import. So change the import to from .testrunner import runTests  and change your setup.py:  entry_points={ 'console_scripts': ['testrunner = testrunner:runTests'] },
Question: I need to replace the last two values in the value column of a pl.DataFrame with zeros, whereby I need to group_by the symbol column. import polars as pl df = pl.DataFrame( {"symbol": [*["A"] * 4, *["B"] * 4], "value": range(8)} ) shape: (8, 2) ┌────────┬───────┐ │ symbol ┆ value │ │ --- ┆ --- │ │ str ┆ i64 │ ╞════════╪═══════╡ │ A ┆ 0 │ │ A ┆ 1 │ │ A ┆ 2 │ │ A ┆ 3 │ │ B ┆ 4 │ │ B ┆ 5 │ │ B ┆ 6 │ │ B ┆ 7 │ └────────┴───────┘  Here is my expected outcome: shape: (8, 2) ┌────────┬───────┐ │ symbol ┆ value │ │ --- ┆ --- │ │ str ┆ i64 │ ╞════════╪═══════╡ │ A ┆ 0 │ │ A ┆ 1 │ │ A ┆ 0 │<-- replaced │ A ┆ 0 │<-- replaced │ B ┆ 4 │ │ B ┆ 5 │ │ B ┆ 0 │<-- replaced │ B ┆ 0 │<-- replaced └────────┴───────┘ 

Answer: You can use  pl.Expr.head() with pl.len() to get data without last two rows. pl.Expr.append() and pl.repeat() to pad it with zeroes.  df.with_columns( pl.col.value.head(pl.len() - 2).append(pl.repeat(0, 2)) .over("symbol") )  shape: (8, 2) ┌────────┬───────┐ │ symbol ┆ value │ │ --- ┆ --- │ │ str ┆ i64 │ ╞════════╪═══════╡ │ A ┆ 0 │ │ A ┆ 1 │ │ A ┆ 0 │ │ A ┆ 0 │ │ B ┆ 4 │ │ B ┆ 5 │ │ B ┆ 0 │ │ B ┆ 0 │ └────────┴───────┘  Alternatively, you can use  pl.when() to create conditional column. pl.int_range() with pl.len() to affect only first n - 2 rows.  df.with_columns( pl.when(pl.int_range(pl.len()) < pl.len() - 2).then(pl.col.value) .otherwise(0) .over("symbol") )  shape: (8, 2) ┌────────┬───────┐ │ symbol ┆ value │ │ --- ┆ --- │ │ str ┆ i64 │ ╞════════╪═══════╡ │ A ┆ 0 │ │ A ┆ 1 │ │ A ┆ 0 │ │ A ┆ 0 │ │ B ┆ 4 │ │ B ┆ 5 │ │ B ┆ 0 │ │ B ┆ 0 │ └────────┴───────┘
Question: I want to draw 3D Positions in a webcam Image using OpenCV's projectPoints function. During testing I noticed, that I always have a certain offset from the real object. This is most obvious when trying to project the origin (0,0,0) to the image center.  The image shape is (2988, 5312, 3), the big red dot is the image center at (1494, 2656) and the small red dot with the lines attached is the projection of the origin (0,0,0) with NO translation and NO rotation leading to (1476, 2732). The main question: Why is the projected point not in the middle of the image? I determined the projected point like so: origin_2d, jacobian = cv2.projectPoints( np.array([(0.0, 0.0, 0.0)]), np.array([(0.0, 0.0, 0.0)]), np.array([(0.0, 0.0, 0.0)]), mtx, dist, )  So the origin (0,0,0) should be projected to 2D with no translation and no rotation and thus appear in the image center, right? I obtained the camera mtx and dist by executing a camera calibration routine as described here OpenCV docs:Camera Calibration using the chessboard you see in the image. Is there something wrong with my calibration? Or am I misunderstanding something in the projection process? Thanks and best regards, Felix

Answer:  The image shape is (2988, 5312, 3), the big red dot is the image center at (1494, 2656)  You calibrated that camera, right? Then that's not the optical center. The optical center is in the camera matrix, along with the focal length.  (1476, 2732)  That looks more like it, although I'd have expected these numbers to also have digits after the decimal point.
Question: Just curious why the Pandas Series rolling window method doesn't preserve the data-type of the original series: import numpy as np import pandas as pd x = pd.Series(np.ones(6), dtype='float32') x.dtype, x.rolling(window=3).mean().dtype  Output: (dtype('float32'), dtype('float64')) 

Answer: x.rolling(window=3) gives you a pandas.core.window.rolling.Rolling object. help(pandas.core.window.rolling.Rolling.mean) includes the note:  Returns ------- Series or DataFrame Return type is the same as the original object with ``np.float64`` dtype.  that's the little why. The big why it would do such a thing, I don't know. Perhaps it's a way to keep from losing precision since you can always choose to convert to float32 again.