eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: How would I make a column with the delta (in days) of two date columns. I thought I could just subtract the date objects, but I'm obviously missing something (pl.from_records([{'start': '2021-01-01', 'end': '2022-01-01'}]) .with_columns(pl.col(['start', 'end']).str.to_date('%Y-%m-%d')) .with_columns(delta = pl.col('end') - pl.col('start')) ) 

Answer: You can try using the Expr.sub() function instead of the - operator:  (pl.from_records([{'start': '2021-01-01', 'end': '2022-01-01'}]) .with_columns(pl.col(['start', 'end']).str.to_date('%Y-%m-%d')) .with_columns(delta = pl.col('end').sub(pl.col('start'))))
Question: I have some problems in my project: I use SQLalchemy 2.0.2 in modules for working with the database however i try to use Apache Airflow 2.6.1 which has sqlalchemy 1.4.48 dependencies. After I run the code, the interpreter either does not work correctly with the database functions (if sqlalchemy 1.4.48 is installed), or it throws an exception(if version is 2.0) like: TypeError: Invalid argument(s) 'encoding' sent to create_engine(), using configuration SQLiteDialect_pysqlite/QueuePool/Engine  Step by step:  I install sqlalchemy(2.0) into a clean virtual environment on my computer with Windows OS. Install apache-airflow with next command  pip install "apache-airflow[postgres]==2.6.1" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.6.1/constraints-3.9.txt"   Then i tried to start the code or use pip install --upgrade sqlalchemy==2.0 and then start the code.  Please tell me what I'm doing wrong? The airflow documentation has a good description of this situation, but I have not been able to figure out how to solve this problem. I tried to uninstall SQLalchemy and apache-airflow and then tried to install with constrain. Tried to change some settings in airflow.cfg

Answer: Have you tried using the PythonVirtualEnvOperator ? It will allow you to install the library at runtime so you don't need to make changes on the server just for one job. To run a function called my_callable, simply use the following: my_task = PythonVirtualenvOperator( task_id="my_task ", requirements="sqlalchemy==2.0", python_callable=my_callable, )  Since the task runs in a virtual environment, it shouldn't be limited by Airflow dependencies.
Question: I make heavy use of mypy static type checking. I have a large lib where I know I have many imports which are being done just for typehints that could be protected with an if TYPE_CHECKING to speed things up. But searching for them all is proving difficult. Is there a way to identify these "unused" imports automatically so I can fix them?

Answer: You can use the flake8-type-checking library for this. Installed using pip install flake8-type-checking
Question: I have a dataset that I need to fill in the blanks of the ID. ID Name Adam 101 Adam Adam 101 Adam 102 Ben 102 Ben Cathy Cathy 103 Cathy  What I need: ID Name 101 Adam 101 Adam 101 Adam 101 Adam 102 Ben 102 Ben 103 Cathy 103 Cathy 103 Cathy  I tried using df['Name'].ffill() but does not work when trying on multiple names. Any other suggestions?

Answer: Try with first df['ID'] = df.groupby('Name')['ID'].transform('first')
Question: I need RegEx to find string based on it's first letters Hello. I have string ABDADBADBADBABDABDA  I want program to find string with mask А*А*А*A.....  where "*" is group of any symbols except "A", but every "*" is the same group I've tried /((A[^A]+)+A)/g  but it matches the whole line Example Input: AxxAxxAbxAx  Output: AxxAxxA  Is there any intended RegEx solution or I have to use some code too?

Answer: Yon need to use backreferences for this. For supplied example A(.*?)A(?:\1A)* should do the trick. Here:  A matches A, (.*?) matches everything till the next A and puts it into group #1, (?:\1A)* matches content of group #1 followed by A any number of times.  Demo here.
Question: I would like to match a b if between a and b is only whitespace with at most one newline. Python example: import re r = "a\s*b" # ? # should match: print(re.match(r, "ab")) print(re.match(r, "a b")) print(re.match(r, "a \n b")) # shouldn't match: print(re.match(r, "a\n\nb")) print(re.match(r, "a \n\n b")) 

Answer: You need to exclude a newline from \s and then optional match a newline with any zero or more whitespace chars other than a newline: a[^\S\n]*(?:\n[^\S\n]*)?b  See the regex demo. Details:  a - an a letter [^\S\n]* - zero or more whitespace chars other than newline (?:\n[^\S\n]*)? - one or zero occurrences of  \n - a newline char [^\S\n]* - zero or more whitespace chars other than newline   b - a b letter.
Question: I am using numpy.histogram and I am getting this error: import numpy as np np.histogram(np.arange(4), bins=np.arange(5), normed=True) TypeError: histogram() got an unexpected keyword argument 'normed'  I was expecting: (array([0.2,0.25,0.25]),array([0,1,2,3,4]))  I am using numpy 1.24.3

Answer: The normed parameter in the numpy.histogram function was deprecated in NumPy version 1.21.0 and removed in version 1.24.0. Example import numpy as np result = np.histogram(a=np.arange(4), bins=np.arange(5), density=True) print(result) # (array([0.25, 0.25, 0.25, 0.25]), array([0, 1, 2, 3, 4]))
Question: When upgrading from pandas version 1 to 2.0.0, I suddenly get a ValueError in a script that worked fine before upgrading pandas to version 2: ValueError: Cannot convert from timedelta64[ns] to timedelta64[D]. Supported resolutions are 's', 'ms', 'us', 'ns'  This is a minimally reproducible example: import pandas as pd df = pd.DataFrame({'designation_date': ['2021-01-01', '2021-01-02']}) df['recency'] = pd.to_datetime('today') - pd.to_datetime(df['designation_date']) df['recency'] = df['recency'].astype('timedelta64[D]')  What do I need to replace df['recency'].astype('timedelta64[D]') with so that the code works with pandas v2? Using astype('timedelta64[D]') is used quite a bit in answers across SO, e.g. here.

Answer: Use the .dt.days accessor instead of astype('timedelta64[D]): df['recency'] = df['recency'].dt.days  The change in behaviour from v1 to v2 is documented here in the Pandas changelog.
Question: I am facing a problem where I have to change a line in a built-in file in a particular library (installed using pip). I have located the file in  app\build\pip\debug\common\<library folder>  But every time I run the Gradle (for installing or creating APK), the entire folder is recreated, and hence, the file is again the same as previous. Is there any way to make the change permanent?

Answer: As mentioned in the comment by David K Hess, monkey patching may be the easiest solution. If monkey patching isn't suitable for your issue, then assuming the library is pure-Python, you can download it from PyPI, edit your local copy, and then install from that:  For example, you could download a .whl file, edit the file inside it, and then add an install line pointing to the .whl.  Or you could download an sdist (.tar.gz file), extract it to a directory, edit the file inside it, and then add an install line pointing to the directory.   In both cases, the install line should probably come first in the requirements list, before anything else which may depend on the library.
Question: I'm trying to understand how to make this work: def someMethod() -> dict[any, any]: if not os.path.exists('some path'): return {} config = {'a': 1, 'b': 2} return config  I don't think that's correct. Seeing this error - Declared return type, "dict[Unknown, Unknown]", is partially unknownPylance The idea is to return empty dict if a path doesn't exist (or on some condition) or correct dict with key-value pairs. Any ideas?

Answer: Lowercase any is a Python built-in function and not a type. Instead, you have to import capital Any from the typing module. from typing import Any import os def someMethod() -> dict[Any, Any]: if not os.path.exists('some path'): return {} config = {'a': 1, 'b': 2} return config
Question: I have a function which I want to yield bytes from (generator behaviour) and also write to a file (non-generator behaviour) depending on whether the save boolean is set. Is that possible? def encode_file(source, save=False, destination=None): # encode the contents of an input file 3 bytes at a time print('hello') with open(source, 'rb') as infile: # save bytes to destination file if save: print(f'saving to file {destination}') with open(destination, 'wb') as outfile: while (bytes_to_encode := infile.read(3)): l = len(bytes_to_encode) if l < 3: bytes_to_encode += (b'\x00' * (3 - l)) outfile.write(bytes_to_encode) return # yield bytes to caller else: while (bytes_to_encode := infile.read(3)): l = len(bytes_to_encode) if l < 3: bytes_to_encode += (b'\x00' * (3 - l)) # pad bits if short yield encode(bytes_to_encode) return  In the above implementation, the function always behaves as a generator. When I call encode_file('file.bin', save=True, destination='output.base64')  it does not print "hello" instead, it returns a generator object. This does not make sense to me. Shouldn't "hello" be printed and then shouldn't control be directed to the if save: portion of the code thus avoiding the part of the function that yields completely?

Answer: A function can’t be a generator and also not be one, but of course you can decide whether to return a generator object or not by defining a helper function. To avoid duplicating the (read) with between the two (and reduce redundancy in general), make one branch a client of the other: def encode_file(source, save=False, destination=None): # encode the contents of an input file 3 bytes at a time print('hello') # save bytes to destination file if save: print(f'saving to file {destination}') with open(destination, 'wb') as outfile: for bytes_to_encode in encode_file(source): outfile.write(bytes_to_encode) # yield bytes to caller else: def g(): with open(source, 'rb') as infile: while (bytes_to_encode := infile.read(3)): l = len(bytes_to_encode) if l < 3: bytes_to_encode += (b'\x00' * (3 - l)) # pad bits if short yield encode(bytes_to_encode) return g()  (Thanks to interjay for pointing out the need for the with in g.)
Question: I have the following df:     index jan feb marc april     One 1 7 0 0   two 0 8 7 0   three 0 0 0 1     I'd like to get the number of zeros after the last non-zero value per row. So the output should look like     index num     One 2   two 1   three 0    

Answer: Similar logic to that of @BrJ but more straightforward in my opinion. Using a reversed cummin to set to False all True preceding a False, then sum: out = (df.loc[:,::-1].eq(0) .cummin(axis=1).sum(axis=1) .to_frame('num') )  Output:  num One 2 two 1 three 0  Intermediates: # boolean mask (0s are True) jan feb marc april One False False True True two True False False True three True True True False # after reversed cummin (and reversed again for clarity) # all True that preceded a False are now False jan feb marc april One False False True True two False False False True three False False False False
Question: When selecting cell from Dataframe, it returns me Series value and append it to list as series. How to convert cell into single int value? df = pd.DataFrame({"name": ['John', 'George', 'Ray'], "score": [123, 321, 112]}) x = df.loc[df['name']=='John', 'score'].reset_index(drop=True) x.astype(int) x list=[] list.append(x) list  Return me [0 123 Name: score, dtype: int64], but need just [123]

Answer: This happens because in general df["name"] == "John" could be true for several rows. A simple way to to work around this is to temporarily turn the "name" column into the DataFrame's index with set_index: import pandas as pd df = pd.DataFrame({"name": ['John', 'George', 'Ray'], "score": [123, 321, 112]}) x = df.set_index("name").loc["John", "score"] list_ = [] list_.append(x) print(list_)  Additional notes:  If you have "John" more than once in the "name" column, you get the same problem with this solution as in your original code.  Depending on your data model, it might make sense to use the names as index permanently by moving set_index to the line where you define the DataFrame.  It's bad practice to shadow built-in names like list because you might need them later.
Question: In Python, a set can be created from a list using either the set constructor or by unpacking the list into curly brackets. For example: my_list = [1, 2, 3] my_set = set(my_list)  or my_list = [1, 2, 3] my_set = {*my_list}  Are there any specific reasons or use cases where one approach is preferred over the other? What are the advantages or disadvantages of each method in terms of performance, readability, or any other relevant factors?

Answer: There is a subtle difference. set(my_list) produces whatever the callable bound to set returns. set is a built-in name for the set type, but it's possible to shadow the name with a global or local variable. {*my_list}, on the other hand, always creates a new set instance. It's not possible to change what the brace syntax means without modifying the Python implementation itself (and then, you are no longer implementing Python, but a very Python-like language). In CPython, using {*mylist} also avoids a function call, it uses BUILD_SET and SET_UPDATE opcodes rather than calling whatever set is bound to.
Question: Im trying to use an XGBClassifier with a validation set and a metric taken from sklearn.metrics as eval_metric, as suggested by the XGBoost documentation. The MWE looks like this: import numpy as np from xgboost import XGBClassifier from sklearn.metrics import accuracy_score x_train, y_train = np.random.rand(10,3), np.where(np.random.rand(10,)>0.5, 1, 0) x_valid, y_valid = np.random.rand(5,3), np.where(np.random.rand(5,)>0.5, 1, 0) model = XGBClassifier( n_estimators=100, eval_metric=accuracy_score ) model.fit( X=x_train, y=y_train, eval_set=[(x_train, y_train), (x_valid, y_valid)] )  This code raises the following error message: --------------------------------------------------------------------------- ValueError Traceback (most recent call last) <ipython-input-5-b63cd5cfabda> in <cell line: 1>() ----> 1 model.fit( 2 X=x_train, y=y_train, 3 eval_set=[(x_train, y_train), (x_valid, y_valid)] 4 ) 9 frames /usr/local/lib/python3.10/dist-packages/sklearn/metrics/_classification.py in _check_targets(y_true, y_pred) 93 94 if len(y_type) > 1: ---> 95 raise ValueError( 96 "Classification metrics can't handle a mix of {0} and {1} targets".format( 97 type_true, type_pred ValueError: Classification metrics can't handle a mix of binary and continuous targets  The same code works commenting out the eval_set line, or using instead eval_metric="error", for example. What am I doing wrong and how is it solved? Edit: I'd like to use in the future different metrics like sklearn.metrics.balanced_accuracy_score or sklearn.metrics.recall_score.

Answer: The reason is that xgboost will feed probability outputs to the evaluation function (your accuracy here), but sklearn's accuracy score is expecting hard decisions (1s or 0s) not probabilities. It is unaware of your decision threshold, so it cannot map them to hard decisions. You can use model = xgb.XGBClassifier( n_estimators=100, eval_metric='error' )  or model = xgb.XGBClassifier( n_estimators=100, eval_metric='error@0.6' )  for a threshold of 0.6 instead of 0.5. See https://xgboost.readthedocs.io/en/stable/parameter.html For recall, since it's not in the xgboost builtin options, you need to manually threshold your predictions: import numpy as np from xgboost import XGBClassifier import xgboost as xgb from sklearn.metrics import accuracy_score, recall_score x_train, y_train = np.random.rand(10,3), np.where(np.random.rand(10,)>0.5, 1, 0) x_valid, y_valid = np.random.rand(5,3), np.where(np.random.rand(5,)>0.5, 1, 0) def thresholded_recall_score(y_true, y_preds, thresh=0.5): return recall_score(y_true, y_preds > thresh) model = xgb.XGBClassifier( n_estimators=100, eval_metric=thresholded_recall_score ) model.fit( X=x_train, y=y_train, eval_set=[(x_train, y_train), (x_valid, y_valid)] )
Question: I have a Pandas DataFrame that looks like this: df = pd.DataFrame({'col1': [1, 2, 3], 'col2': [4, 5, 6], 'col3': [7, 8, 9]}) df col1 col2 col3 0 1 4 7 1 2 5 8 2 3 6 9  I would like to create a Pandas DataFrame like this: df_new col1 col2 col3 0 1 4 7 1 1 5 8 2 1 6 9 3 2 4 7 4 2 5 8 5 2 6 9 6 3 4 7 7 3 5 8 8 3 6 9  Is there built-in or combination of built-in Pandas methods that can achieve this? Even if there are duplicates in df, I would like the output to be the same format. In other words: df col1 col2 col3 0 1 4 7 1 2 5 8 2 2 6 8 df_new col1 col2 col3 0 1 4 7 1 1 5 8 2 1 6 8 3 2 4 7 4 2 5 8 5 2 6 8 6 2 4 7 7 2 5 8 8 2 6 8 

Answer: I would also have gone for a cross merge as suggested by @Henry in comments: out = df[['col1']].merge(df[['col2', 'col3']], how='cross').reset_index(drop=True)  Output:  col1 col2 col3 0 1 4 7 1 1 5 8 2 1 6 9 3 2 4 7 4 2 5 8 5 2 6 9 6 3 4 7 7 3 5 8 8 3 6 9  Comparison of the different approaches:  Note that @sammywemmy's approach behaves differently when rows are duplicated, which leads to a non comparable timing.
Question: I have a middleware implemented for FastAPI. For responses that includes some content, it works perfectly. But if a response has no body, it is causing LocalProtocolError("Too much data for declared Content-Length") exception. To isolate the problem, I've reduced the middleware class to this: from starlette.middleware.base import BaseHTTPMiddleware from fastapi import FastAPI, Request class LanguageManagerMiddleware(BaseHTTPMiddleware): def __init__(self, app: FastAPI): super().__init__(app) async def dispatch(self, request: Request, call_next) -> None: return await call_next(request)  It basically does nothing. When I add the middleware, I have an exception:  raise LocalProtocolError("Too much data for declared Content-Length") h11._util.LocalProtocolError: Too much data for declared Content-Length  When I disable the middleware, I have no problem. Here is the line that creates the response which triggers the exception: return Response(status_code=HTTP_204_NO_CONTENT)  To further debug the problem, I've activated a breakpoint in the h11/_writers.py ContentLengthWriter class, where the actual exception occurs. I've tried to decode the byte stream with utf-8 and cp437, but had no luch.  class ContentLengthWriter(BodyWriter): def __init__(self, length: int) -> None: self._length = length def send_data(self, data: bytes, write: Writer) -> None: self._length -= len(data) if self._length < 0: raise LocalProtocolError("Too much data for declared Content-Length") write(data)  I'm stopping the code at this line: self._length -= len(data) If the middleware is disabled, data looks like this: b'' If the middleware is enabled, data looks like this: b'\x1f\x8b\x08\x00\xf6God\x02\xff' What would be modifying the content of the response?

Answer: I've solved it. I've just changed the order of which middlewares are added. When I moved my middleware after the GZIP middleware, problem disappeared. Thanks to MatsLindh for pointing out that it is a gzip header.
Question: I am trying to make a real number complex using numpy. I am using numpy version 1.24.3 Here is the code: import numpy as np c=np.complex(1)  However, I get this error: AttributeError: module 'numpy' has no attribute 'complex'. 

Answer: np.complex was a deprecated alias for the builtin complex. Instead of np.complex you can use: complex(1) #output (1+0j) #or np.complex128(1) #output (1+0j) #or np.complex_(1) #output (1+0j) #or np.cdouble(1) #output (1+0j)  Link to doc: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
Question: I have the following string, which I extract from a pdf: This is Fig. 13: John holding his present and the flowers Source: official photographer a beautiful Table: a table of some kind and fully complete Table: John holding his present and Source: official photographer sentence  The text includes figs and tables, most of which have a caption on top and a source on bottom, but some don't. Fundamentally, the text I want to be left with should be: This is a beautiful and fully complete sentence  I have tried the following: s = re.sub(r'(Fig|Table)[\s\S]+?Source:.*\n', '', mystring,flags=re.MULTILINE)  But unfortunately it returns: This is a beautiful sentence  With my limited knowledge of regex I cannot figure out how to put such a condition: It should stop at the first \n after Source, only if there is no new fig|table in between, in which case it should have stopped at the first \n from start. Any idea? Thank you.

Answer: What you need to match is a Fig or Table followed by either  Characters up to and including a line starting with Source, with no Fig or Table in between the original one and Source; or Characters up to the end of line  You can achieve #1 above by using a tempered greedy token, which ensures that each character processed until Source is found does not precede Fig or Table. This regex will do what you want: (?:Fig|Table)(?:(?:(?!Fig|Table)[\s\S])+?Source[^\n]*\n|[^\n]*\n)  This matches:  (?:Fig|Table) : a word Fig or Table; and then either (?:(?!Fig|Table)[\s\S])+? : a minimal number of characters, none of which precede either of the words Fig or Table Source[^\n]*\n : The word Source followed by some number of characters until newline; or [^\n]*\n some number of characters until newline  Regex demo on regex101 In python: s = re.sub(r'(?:Fig|Table)(?:(?:(?!Fig|Table)[\s\S])+?Source[^\n]*\n|[^\n]*\n)', '', mystring) print(s)  Output: This is a beautiful and fully complete sentence  Note this does leave newlines (if present in the original string) at the start and end of the string, they can be removed with strip.
Question: I want to subset a df by returning the first and last consecutive value from a pandas col. Drop_duplciates won't work because it doesn't account for consecutive groupings. I'm using .shift() (below) but this only returns the last consecutive value, where I want the first and last. import pandas as pd df = pd.DataFrame({"Item":['A', 'A', 'A', 'B', 'B', 'B', 'B', 'A', 'A'], "Val1":[-20, -21, -20, -20, -20, -21, -20, -23, -22], "Val2":[150, 151, 150, 148, 149, 150, 151, 150, 148] }) df1 = df[df['Item'].ne(df['Item'].shift())] print(df1)  intended output:  Item Val1 Val2 0 A -20 150 2 A -20 150 3 B -20 148 6 B -20 151 7 A -23 150 8 A -22 148 

Answer: You need to compare against both the forward and backward shifted values so that you can find the start and finish of each group: df1 = df[(df['Item'].ne(df['Item'].shift())) | (df['Item'].ne(df['Item'].shift(-1)))]  Output:  Item Val1 Val2 0 A -20 150 2 A -20 150 3 B -20 148 6 B -20 151 7 A -23 150 8 A -22 148
Question: I'm having trouble calling typing.get_type_hints() for classes that have forward references as strings. My code works with not defined inside of a function. I've reproduced a minimal example below in Python 3.10: import typing class B: pass class A: some_b: "B" print(typing.get_type_hints(A)) # prints {'some_b': <class '__main__.B'>}  import typing def func(): class B: pass class A: some_b: "B" print(typing.get_type_hints(A)) func() # NameError: name 'B' is not defined  Is this expected behavior? Is there any way to get around this, and make sure that forward references with strings get evaluated in the correct scope?

Answer: typing.get_type_hints allows you to explicitly pass the local namespace to use for resolving references via the localns parameter. from typing import get_type_hints def func(): class A: some_b: "B" class B: pass print(get_type_hints(A, localns=locals())) func()  Output: {'some_b': <class '__main__.func.<locals>.B'>} See the docs for locals.  Side note: By utilizing postponed evaluation of annotations (PEP 563) you can omit the quotation marks: from __future__ import annotations from typing import get_type_hints def func(): class A: some_b: B class B: pass print(get_type_hints(A, localns=locals()))