eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: TL;DR How does one reliably include files from LICENSES/ (REUSE-style) in source archive and wheels for a Python package with a src/ layout? How does one exclude specific files? Details I have a project structure that looks like . ├── pyproject.toml ├── LICENSES │ ├── MAIN.txt │ ├── SECUNDARY.txt ├── MANIFEST.in ├── random_package │ ├── __init__.py │ ├── foo1.cpp │ ├── foo2.cpp │ ├── submodule1 │ │ ├── __init__.py │ │ ├── bar1.cpp │ ├── submodule2 │ │ ├── __init__.py │ │ ├── bar2.cpp  The pyproject.toml looks like [build-system] requires = ["setuptools>=61.0"] build-backend = "setuptools.build_meta" [project] name = "random_package" version = "0.1.0" license = {file = "LICENSES/MAIN.txt"} [metadata] # EDIT: metadata was the issue license-files = ["LICENSES/*.txt"] # this line should be in [tool.setuptools] [tool.setuptools] package-dir = {"" = "."} include-package-data = true # tried both true and false [tool.setuptools.packages.find] where = ["."] include = ["random_package*"]  How do I include all cpp files except submodule1/bar1.cpp into the installation? I have tried the following entries in the toml (one at a time): [tool.setuptools.exclude-package-data] "*" = ["bar1.cpp"] "random_package.submodule1" = ["bar1.cpp"]  I even set include-package-data to false and entered cpp files manually (except bar1.cpp) and even that did not work for both source and wheels. Nothing works reliably: for any and all combinations of these options, I always get bar1.cpp in either the zip/tar.gz archive or the wheel when I do python -m build. As for the license files, I get LICENSE/MAIN.txt in the source build, but not the others and no licenses are present in the wheels. Partial solution I have something that works for source dist using a MANIFEST.in with an include for the LICENSES/*.txt files and a manual include for the .cpp files instead of the data options in pyproject.toml but even this does not work for the wheel: I don't get the licenses in random_package-0.1.0.dist-info. Am I wrong in expecting the license files in the wheel? With the old setup.py scheme, back when I was using a single License.txt file, I did get the license file in there... And is there no way to do that with the toml alone?

Answer: It turns out that I was mistaken about the location of license-files (I first saw it in the "metadata" section on the doc); it must actually be in [tool.setuptools]. The other data include issue was maybe a cache issue, it seems to work in the following pyproject.toml: [build-system] requires = ["setuptools>=61.0"] build-backend = "setuptools.build_meta" [project] name = "random_package" license = {file = "LICENSES/MAIN.txt"} version = "0.1.0" [tool.setuptools] package-dir = {"" = "."} include-package-data = false license-files = ["LICENSES/*.txt"] [tool.setuptools.packages.find] where = ["."] include = ["random_package*"] [tool.setuptools.package-data] random_package = ["*.cpp"] [tool.setuptools.exclude-package-data] "*" = ["bar1.cpp"]  With this, no MANIFEST.in file is required.
Question: I've been using cpython forever, but I'm new to pypy. In cpython, this is how I use virtual environments and pip. python3 -m venv venv source venv/bin/activate python3 -m pip install <package>  I recently started using pypy for a project, and noticed that the following works. pypy3 -m venv venv source venv/bin/activate pypy3 -m pip install <package>  Questions:  Are there any differences between cpython venv/pip and pypy venv/pip? Can I create a venv using cpython, and use it with pypy, or vice-versa? Similarly, can I install packages using cpython's pip, and use them from pypy interpreter, or vice-versa? Is what I'm doing "correct", or are there any downsides/issues I'll face in future if I go down this road.  Reasons why I prefer the python3 -m ... invocations:  venv is present in std. lib, so I don't have to globally install virtualenv. Less ambiguous than using pip and pip3.  References:  What is the difference between venv, pyvenv, pyenv, virtualenv, virtualenvwrapper, pipenv, etc? Should I use pip or pip3?   EDIT: Tried to share venv's between cpython and venv doesn't work (seems obvious in hindsight). It's still possible to create two separate venv's like python3 -m venv cpython_venv and pypy3 -m venv pypy_venv and switch between them as needed. python will be bound to cpython or pypy based on which virtual env is active, and pypi packages need to be installed in every venv where it's needed.

Answer:  Are there any differences between cpython venv/pip and pypy venv/pip?  Yes, PyPy make some changes in the venv Python code, so they may have some differences. Example for 3.7:  CPython stdilb: https://github.com/python/cpython/blob/v3.7.13/Lib/venv/__init__.py PyPy stdlib: https://github.com/mozillazg/pypy/blob/release-pypy3.7-v7.3.9/lib-python/3/venv/__init__.py   Can I create a venv using cpython, and use it with pypy, or vice-versa?  I wouldn't recommend that, since they presumably have good reasons for patching the stdlib venv code.  Similarly, can I install packages using cpython's pip, and use them from pypy interpreter, or vice-versa?  I wouldn't recommend that, for several reasons. In the case of binary distributions with compatibility tags, the installer may select a wheel file which is specific to the Python interpreter that pip was running in. This package could be totally broken for a different Python runtime. Use python3 -m pip debug --verbose or pypy3 -m pip debug --verbose to list the supported compatibility tags of each runtime. Even for pure-python packages with no compiled extensions you're not safe - it's also the job of the installer to generate bytecode (.pyc files) at installation time. If you install with a different interpreter, you'll get incompatible bytecode. Python packages can and do specify conditional dependencies using environment markers in the packaging metadata. It's possible for dependency trees to be different between CPython and PyPy based on the platform_python_implementation environment marker.  Is what I'm doing "correct", or are there any downsides/issues I'll face in future if I go down this road.  Your usage shown in the question is correct.
Question: I am trying to open an xlsx file with Pandas and i recieve this error: Output exceeds the size limit. Open the full output data in a text editor --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) Cell In[20], line 1 ----> 1 df = pd.read_excel("C:/Users/*****/OneDrive - *******/Bureau/file_name.xlsx", sheet_name="") File c:\Users\*******\OneDrive - ********\Bureau\file_name\venv\lib\site-packages\pandas\util\_decorators.py:211, in deprecate_kwarg.<locals>._deprecate_kwarg.<locals>.wrapper(*args, **kwargs) 209 else: 210 kwargs[new_arg_name] = new_arg_value --> 211 return func(*args, **kwargs) File c:\Users\******\OneDrive - *******\Bureau\file_name\venv\lib\site-packages\pandas\util\_decorators.py:331, in deprecate_nonkeyword_arguments.<locals>.decorate.<locals>.wrapper(*args, **kwargs) 325 if len(args) > num_allow_args: 326 warnings.warn( 327 msg.format(arguments=_format_argument_list(allow_args)), 328 FutureWarning, 329 stacklevel=find_stack_level(), 330 ) --> 331 return func(*args, **kwargs) File c:\Users\******\OneDrive - ******\Bureau\file_name\venv\lib\site-packages\pandas\io\excel\_base.py:482, in read_excel(io, sheet_name, header, names, index_col, usecols, squeeze, dtype, engine, converters, true_values, false_values, skiprows, nrows, na_values, keep_default_na, na_filter, verbose, parse_dates, date_parser, thousands, decimal, comment, skipfooter, convert_float, mangle_dupe_cols, storage_options) 480 if not isinstance(io, ExcelFile): 481 should_close = True --> 482 io = ExcelFile(io, storage_options=storage_options, engine=engine) 483 elif engine and engine != io.engine: 484 raise ValueError( ... --> 109 sheet.defined_names[name] = defn 111 elif reserved == "Print_Titles": 112 titles = PrintTitles.from_string(defn.value) AttributeError: 'ReadOnlyWorksheet' object has no attribute 'defined_names' "  I was using python 3.10. versions and tried it with 3.11.2 also and version of pandas is 1.5.3 also i tried the solution in this topic too;  'ReadOnlyWorksheet' object has no attribute 'defined_names'  i downgraded openpyxl and upgraded it again, tried several times but nope. still getting the same error. I tried different file xlsm, worked fine but not with xlsx. so i am blocked atm!

Answer: Restart Kernel. Had the same problem, downgraded openpyxl version to the 3.1.0, still wasn't working. Did a restart and runned it again. Worked!
Question: I previously got this rewrite of history commit messages to work: #!/bin/bash # Create a temporary file to store the commit messages temp_file=$(mktemp) main_head_hash=$(git rev-parse main) suffix="⚠️ rebased since!" # Use git log to retrieve the commit messages and store them in the temporary file git log --pretty=format:%s $main_head_hash.. | grep '📌 build version' | grep -v "$suffix" > $temp_file # Create a file to store the replacements echo > replacements.txt # Iterate over the commit messages in the temporary file while read commit_message; do # Print the replacement message to the replacements.txt file echo "$commit_message==>$commit_message $suffix" >> replacements.txt done < $temp_file # # ⚠️⚠️ Rewriting history ⚠️⚠️ git filter-repo --replace-message replacements.txt --force # # Remove the temporary files rm $temp_file rm replacements.txt  but I have no began to notice that it causes my git repo to lose the remotes setup. Af first I thought it was some other git tool I was using, but now I have this script being the suspect 🕵️ How do I avoid losing the remotes?

Answer: Removing the remote is by design. Quote from the author:  filter-repo defaults to rewriting the whole repository and as such it creates history incompatible with the original. There are lots of things that can go wrong with pushing a rewritten history back up to the original location, something that other tools did not discuss or warn about in detail and which will trip up many people. Removing the origin remote helps avoid these errors; it is encouragement for folks to stop and read the docs to make sure they are prepared and understand the ramifications (and take appropriate preventative steps) before they do something that may bite them. (They can still push to the remote by just setting it back up, can just re-fetch, etc., so it's easy to circumvent, but I want them to at least notice something is different).  Note it's easy to add the remote back: git remote add <repo-url-here>  Or, you can also use the --refs or --partial options to avoid removing it during the filter. Note these options do more than just "not removing the remote", so you should decide whether they are acceptable to you:  --partial Do a partial history rewrite, resulting in the mixture of old and new history. This implies a default of update-no-add for --replace-refs, disables rewriting refs/remotes/origin/* to refs/heads/*, disables removing of the origin remote, disables removing unexported refs, disables expiring the reflog, and disables the automatic post-filter gc. Also, this modifies --tag-rename and --refname-callback options such that instead of replacing old refs with new refnames, it will instead create new refs and keep the old ones around. Use with caution. --refs <refs+> Limit history rewriting to the specified refs. Implies --partial. In addition to the normal caveats of --partial (mixing old and new history, no automatic remapping of refs/remotes/origin/* to refs/heads/*, etc.), this also may cause problems for pruning of degenerate empty merge commits when negative revisions are specified.
Question: There is a string: input_str = 'The substring of "python" from index @ to index @ inclusive is "tho"' and a list of indices: idx_list = [2, 4] I want to replace the character @ in str_input with each element of the idx_list to have the following output: output_str = 'The substring of "python" from index 2 to index 4 inclusive is "tho"' So I have coded it as follows: def replace_char(input_str, idx_list): output_str = "" idx = 0 for i in range(0, len(input_str)): if input_str[i] == '@': output_str += str(idx_list[idx]) idx += 1 else: output_str += input_str[i] return output_str  I wonder if there is any shorter and faster way than the concatenation that I have used?

Answer: One concise approach uses re.sub with a callback function: input_str = 'The substring of "python" from index @ to index @ inclusive is "tho"' idx_list = [2, 4] output_str = re.sub(r'\bindex @', lambda m: str(idx_list.pop(0)), input_str) print(output_str) # The substring of "python" from 2 to 4 inclusive is "tho"  The idea here is that every time a match of index @ is found, we replace with the first entry in the list of indices. We also then pop that first index, so that it doesn't get used again.
Question: I have the following dataset df id medication_date 1 2000-01-01 1 2000-01-04 1 2000-01-06 2 2000-04-01 2 2000-04-02 2 2000-04-03  I would like to first reshape the data set into days after the first observation per patient: id day1 day2 day3 day4 1 yes no no yes 2 yes yes yes no  in order to ultimately create a plot with the above table: columns the dates and in black if yes, and white if not. any help really appreciated it

Answer: Transform the sparse Series ('yes' medication) to dense Series by adding missing days ('no' medication) then reset the Series index (2000-01-01 -> 0, 2000-04-01 -> 0). Finally, reshape your dataframe. def f(sr): # Create missing dates dti = pd.date_range(sr.min(), sr.max(), freq='D') # Fill the Series with 'yes' or 'no' return (pd.Series('yes', index=sr.tolist()) .reindex(dti, fill_value='no') .reset_index(drop=True)) df['medication_date'] = pd.to_datetime(df['medication_date']) out = (df.groupby('id')['medication_date'].apply(f).unstack(fill_value='no') .rename(columns=lambda x: f'day{x+1}').reset_index())  Output: >>> out id day1 day2 day3 day4 day5 day6 0 1 yes no no yes no yes 1 2 yes yes yes no no no  Update import matplotlib.pyplot as plt from matplotlib.colors import LinearSegmentedColormap colors = ["white", "black"] cmap = LinearSegmentedColormap.from_list('Custom', colors, len(colors)) plt.matshow(out.set_index('id').eq('yes').astype(int), cmap=cmap) plt.show()
Question: I am trying to organise DataFrame columns based on the specific rules, but I don't know the way. For example, I have a DataFrame related to chemistry as shown below. Each row shows the number of chemical bonds in a chemical compound.  OH HO CaO OCa OO NaMg MgNa 0 2 3 2 0 1 1 1 1 0 2 3 4 5 2 0 2 1 2 3 0 0 0 0  In chemistry, OH (Oxygen-Hydrogen) bond is equal to HO (Hydrogen-Oxygen) bond and CaO (Calcium-Oxygen) bond is equal to OCa (Oxygen-Calcium) bond in the meaning. Thus, I'd like to organise the DataFrame as shown below.  OH CaO OO NaMg 0 5 2 1 2 1 2 7 9 2 2 3 3 0 0  I’m struggling because:  there are a variety of chemical bonds in my real DataFrame, so it is impossible to organise the information one by one (The number of columns is more than 3,000 and I don't know which kinds of chemical bonds exist and are duplicates.) the number of letters depends on each element symbol and some symbols include lowercase (e.g. Hydrogen: H (one letter and only uppercase), Calcium: Ca (Two letters and uppercase & lowercase)  I looked for the same question online and wrote codes by myself, but I was not able to find the way. I would like to know the codes which solve my problem.

Answer: You can use str.findall to extract individual element and use frozenset and sort individual elements to reorganize the pairs. Using frozenset is not a good solution because for OO, the second will be lost. Now you can group by this sets and apply sum: # Modified from https://www.johndcook.com/blog/2016/02/04/regular-expression-to-match-a-chemical-element/ pat = r'(A[cglmrstu]|B[aehikr]?|C[adeflmnorsu]?|D[bsy]|E[rsu]|F[elmr]?|G[ade]|H[efgos]?|I[nr]?|Kr?|L[airuv]|M[dgnot]|N[abdeiop]?|Os?|P[abdmortu]?|R[abefghnu]|S[bcegimnr]?|T[abcehilm]|U(?:u[opst])?|V|W|Xe|Yb?|Z[nr])' grp = df.columns.str.findall(pat).map(lambda x: tuple(sorted(x)))) out = df.groupby(grp, axis=1).sum().rename(columns=''.join)  Output: >>> out CaO HO MgNa OO 0 2 5 2 1 1 7 2 2 5 2 3 3 0 0
Question: I am completely perplexed. We came across a bug, which we easily fixed, but we are perplexed as to why the value the bug was generating created the output it did. Specifically: Why does ~True equal -2 in python? ~True >> -2  Shouldn't the bitwise operator ~ only return binary? (Python v3.8)

Answer: True is a specialization of int. In python, integers are signed and unbounded. If one were to invert a fixed-size integer like the 16 bit 0x0001, you'd get 0xfffe which is -2 signed. But python needs a different definition of that operation because it is not size bounded. In Unary arithmetic and bitwise operations python defines unary inversion as  The unary ~ (invert) operator yields the bitwise inversion of its integer argument. The bitwise inversion of x is defined as -(x+1). It only applies to integral numbers or to custom objects that override the invert() special method.  This has the same effect as fixed-size bit inversion without messing around with infinity. Sure enough >>> -(True+1) -2
Question: I am trying to sort a list based on dictionary values. The only problem is that if a list contains something that doesn't exist in the dictionary it wont sort it. here is what I have d = { "hello0" : 0, "hello1" : 1, "hello2" : 2, "hello3" : 3 } l1 = ["hello1","hello3","hello0","hello2"] #list 1 l2 = ["hello4","hello1","hello3","hello0","hello2"] #list 2 try: sorted_l1 = sorted(l1,key=lambda x : d[x]) sorted_l2 = sorted(l2,key=lambda x : d[x]) #print(sorted_l1) print(sorted_l2) except Exception as e: print(f"Keyerror {e}")  List 1 will get sorted just fine because everything in that list exists in the dictionary but list 2 will not get sorted and will get a keyerror. How do I sort list 2 so that the missing key gets added to the end of the sorted list 2 ? Do I use {}.get ? or is there some another way ?

Answer: max_value_dict = max(d.values()) sorted_l2 = sorted(l2,key=lambda x : d.get(x, max_value_dict + 1))  Whenever x is not a key in d, the value associated to x by d.get in the sort is strictly greater than every value in d. Thus x is placed at the end.
Question: I have a dataframe in the following format with ~80K rows. df = pd.DataFrame({'Year': [1900, 1902, 1903], 'Name': ['Tom', 'Dick', 'Harry']}) Year Name 0 1900 Tom 1 1902 Dick 2 1903 Harry  I need to call a function with each combination of the name column as parameters. Currently I am doing this with the following code (substituting print for function call): for i, n1 in enumerate(df.itertuples()): for n2 in df[i:].itertuples(): print(n1.Name, n2.Name)  Is there a way to speed this up that I am missing? PS: I need to keep track of the indices for each name pair. So if I run itertools.combinations on the index, then I still have to make costly df.loc calls.

Answer: Another solution to keep track of index/years would be to use a cross join: import pandas as pd df = pd.DataFrame({'Year': [1900, 1902, 1903], 'Name': ['Tom', 'Dick', 'Harry']}) df = df.reset_index() print(df.join(df, how='cross', lsuffix='_1', rsuffix='_2'))  Output:  index_1 Year_1 Name_1 index_2 Year_2 Name_2 0 0 1900 Tom 0 1900 Tom 1 0 1900 Tom 1 1902 Dick 2 0 1900 Tom 2 1903 Harry 3 1 1902 Dick 0 1900 Tom 4 1 1902 Dick 1 1902 Dick 5 1 1902 Dick 2 1903 Harry 6 2 1903 Harry 0 1900 Tom 7 2 1903 Harry 1 1902 Dick 8 2 1903 Harry 2 1903 Harry
Question: I have a parquet file (~1.5 GB) which I want to process with polars. The resulting dataframe has 250k rows and 10 columns. One column has large chunks of texts in it. I have just started using polars, because I heard many good things about it. One of which is that it is significantly faster than pandas. Here is my issue / question: The preprocessing of the dataframe is rather slow, so I started comparing to pandas. Am I doing something wrong or is polars for this particular use case just slower? If so: is there a way to speed this up? Here is my code in polars import polars as pl df = (pl.scan_parquet("folder/myfile.parquet") .filter((pl.col("type")=="Urteil") | (pl.col("type")=="Beschluss")) .collect() ) df.head()  The entire code takes roughly 1 minute whereas just the filtering part takes around 13 seconds. My code in pandas: import pandas as pd df = (pd.read_parquet("folder/myfile.parquet") .query("type == 'Urteil' | type == 'Beschluss'") ) df.head()  The entire code also takes roughly 1 minute whereas just the querying part takes <1 second. The dataframe has the following types for the 10 columns:  i64 str struct[7] str (for all remaining)  As mentioned: a column "content" stores large texts (1 to 20 pages of text) which I need to preprocess and the store differently I guess.  EDIT: removed the size part of the original post as the comparison was not like for like and does not appear to be related to my question. 

Answer: Edit 30 January 2025: The answer below isn't true anymore and Polars switched to a faster implementation of the string type.  As mentioned: a column "content" stores large texts (1 to 20 pages of text) which I need to preprocess and the store differently I guess.  This is where polars must do much more work than pandas. Polars uses arrow memory format for string data. When you filter your DataFrame all the columns are recreated for where the mask evaluates to true. That means that all the text bytes in the string columns need to be moved around. Whereas for pandas they can just move the pointers to the python objects around, e.g. a few bytes. This only hurts if you have really large values as strings. E.g. when you are storing whole webpages for instance. You can speed this up by converting to categoricals.
Question: I do not seem to find an equivalent for Polars library. But basically, what I want to do is fill missing dates between two dates for a big dataframe. It has to be Polars because of the size of the data (> 100 mill). Below is the code I use for Pandas, but how can I do the same thing for Polars? import janitor import pandas as pd from datetime import datetime, timedelta def missing_date_filler(d): df = d.copy() time_back = 1 # Look back in days td = pd.to_datetime(datetime.now().strftime("%Y-%m-%d")) helper = timedelta(days=time_back) max_date = (td - helper).strftime("%Y-%m-%d") # Takes todays date minus 1 day df_date = dict(Date = pd.date_range(df.Date.min(), max_date, freq='1D')) # Adds the full date range between the earliest date up until yesterday df = df.complete(['Col_A', 'Col_B'], df_date).sort_values("Date") # Filling the missing dates return df 

Answer: It sounds like you're looking for .upsample() Note that you can use the group_by parameter to perform the operation on a per-group basis. import polars as pl from datetime import datetime df = pl.DataFrame({ "date": [datetime(2023, 1, 2), datetime(2023, 1, 5)], "value": [1, 2] })  shape: (2, 2) ┌─────────────────────┬───────┐ │ date | value │ │ --- | --- │ │ datetime[μs] | i64 │ ╞═════════════════════╪═══════╡ │ 2023-01-02 00:00:00 | 1 │ │ 2023-01-05 00:00:00 | 2 │ └─────────────────────┴───────┘  >>> df.upsample(time_column="date", every="1d") shape: (4, 2) ┌─────────────────────┬───────┐ │ date | value │ │ --- | --- │ │ datetime[μs] | i64 │ ╞═════════════════════╪═══════╡ │ 2023-01-02 00:00:00 | 1 │ │ 2023-01-03 00:00:00 | null │ │ 2023-01-04 00:00:00 | null │ │ 2023-01-05 00:00:00 | 2 │ └─────────────────────┴───────┘