eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: The breakpoints data is the following: breakpoints = pl.DataFrame( { "features": ["feature_0", "feature_0", "feature_1"], "breakpoints": [0.1, 0.5, 1], "n_possible_bins": [3, 3, 2], } ) print(breakpoints) out: shape: (3, 3) ┌───────────┬─────────────┬─────────────────┐ │ features ┆ breakpoints ┆ n_possible_bins │ │ --- ┆ --- ┆ --- │ │ str ┆ f64 ┆ i64 │ ╞═══════════╪═════════════╪═════════════════╡ │ feature_0 ┆ 0.1 ┆ 3 │ │ feature_0 ┆ 0.5 ┆ 3 │ │ feature_1 ┆ 1.0 ┆ 2 │ └───────────┴─────────────┴─────────────────┘  The df has two continous variables that we wish to encode according to the breakpoints DataFrame: df = pl.DataFrame( {"feature_0": [0.05, 0.2, 0.6, 0.8], "feature_1": [0.5, 1.5, 1.0, 1.1]} ) print(df) out: shape: (4, 2) ┌───────────┬───────────┐ │ feature_0 ┆ feature_1 │ │ --- ┆ --- │ │ f64 ┆ f64 │ ╞═══════════╪═══════════╡ │ 0.05 ┆ 0.5 │ │ 0.2 ┆ 1.5 │ │ 0.6 ┆ 1.0 │ │ 0.8 ┆ 1.1 │ └───────────┴───────────┘  After the encoding we should have the resulting DataFrame encoded_df: encoded_df = pl.DataFrame({"feature_0": [0, 1, 2, 2], "feature_1": [0, 1, 0, 1]}) print(encoded_df) out: shape: (4, 2) ┌───────────┬───────────┐ │ feature_0 ┆ feature_1 │ │ --- ┆ --- │ │ i64 ┆ i64 │ ╞═══════════╪═══════════╡ │ 0 ┆ 0 │ │ 1 ┆ 1 │ │ 2 ┆ 0 │ │ 2 ┆ 1 │ └───────────┴───────────┘   We can assume that the unique list of features in encoded_df are also available in breakpoints Labels should be an array: np.array([str(i) for i in range(n_possible_bins)]), assuming n_possible_bins is a positive integer. n_possible_bins may be different across features. All the encoding follows left_closed=False where the bins are defined as (breakpoint, next breakpoint]  I know that Polars.Expr.cut() takes in breaks parameter as Sequence[float], but how do I pass in these breakpoints and labels from the breakpoints DataFrame effectively?

Answer: Given that breakpoints will most likely be a very small DataFrame, I think the simplest and most efficient solution is something like: import polars as pl breakpoints = pl.DataFrame( { "features": ["feature_0", "feature_0", "feature_1"], "breakpoints": [0.1, 0.5, 1], "n_possible_feature_brakes": [3, 3, 2], } ) df = pl.DataFrame( {"feature_0": [0.05, 0.2, 0.6, 0.8], "feature_1": [0.5, 1.5, 1.0, 1.1]} ) # Aggregate the breakpoints by feature feature_breaks = breakpoints.group_by("features").agg( pl.col("breakpoints").sort().alias("breaks") ) # For each feature, call `pl.cut` with the respective `breaks` result = df.select( pl.col(feat).cut(breaks, labels=[str(x) for x in range(len(breaks) + 1)]) for feat, breaks in feature_breaks.iter_rows() )  Output: >>> feature_breaks shape: (2, 2) ┌───────────┬────────────┐ │ features ┆ breaks │ │ --- ┆ --- │ │ str ┆ list[f64] │ ╞═══════════╪════════════╡ │ feature_0 ┆ [0.1, 0.5] │ │ feature_1 ┆ [1.0] │ └───────────┴────────────┘ >>> result shape: (4, 2) ┌───────────┬───────────┐ │ feature_0 ┆ feature_1 │ │ --- ┆ --- │ │ cat ┆ cat │ ╞═══════════╪═══════════╡ │ 0 ┆ 0 │ │ 1 ┆ 1 │ │ 2 ┆ 0 │ │ 2 ┆ 1 │ └───────────┴───────────┘
Question: In order to improve the speed of my code in Python I use a njit library from numba. For the number 94906267, which I use in my calculations, I get a wrong answer. At the same time, if I do not use njit, my code gives a correct answer. Here are two examples. from numba import njit @njit def main_with_njit(): a = 94906267 print(f'main_with_njit: a**2 = {a ** 2}') # -> a**2 = 9007199515875288 # the type is int64 main_with_njit() def main_without_njit(): a = 94906267 print(f'main_without_njit: a**2 = {a ** 2}') # -> a**2 = 9007199515875289 # the type is <class 'int'> main_without_njit()  How can I use njit without getting a wrong answer?  I use Python 3.11.3 with PyCharm 2023.1.2 (Community Edition) on Windows, 64 bit, 0.61.0 version of numba. I can reproduce the problem originally described by @Roshan N using the math library. import math print(math.pow(94906267, 2)) # -> 9007199515875288.0 

Answer: First things first. This error is only possible by first converting the operands to 64-bit floating point number. The reason for this is that 64-bit floating point numbers do not have the precision to represent 9007199515875289, and the result is instead rounded to 9007199515875288. You have not done anything to suggest that you want the operands to be floating point numbers. This suggests that the behaviour is a bug somewhere down the numba / llvm chain. Indeed, if you look at the LLVM generated code you can see that the code transforms a ** 2 in to a * a. However, for some reason it converts the result to a double and then back again before returning it. As pointed out by @ken, this behaviour is not seen on all versions of python. Both 3.10 and 3.12 work for me. It is only 3.11 that produced the erroneous result. You can view the LLVM generated code by doing: import numba @numba.njit def f(a: int) -> int: return a ** 2 f(94906267) # NB. code is not generated unless you call the function args = (numba.int64,) code = f.inspect_llvm()[args]  Python 3.11 LLVM Code Snippet  %.185.i = mul nsw i64 %arg.a, %arg.a %.224.i = sitofp i64 %.185.i to double %.229.i = fptosi double %.224.i to i64 store i64 %.229.i, i64* %retptr, align 8  You can see that it first does the multiplication, but then for some reason converts the result to a 64-bit floating point number (sitofp i64 %.185.i to double, ie. sitofp = signed integer to floating point) and then back again (fptosi double %.224.i to i64), before returning it. Python 3.12 LLVM Code Snippet  %.191.i = mul nsw i64 %arg.a, %arg.a store i64 %.191.i, i64* %retptr, align 8
Question: Can somebody explain why in the following code Two matches? >>> class One: ... pass ... >>> class Two: ... pass ... >>> a = One() >>> >>> match a.__class__: ... case Two: ... print("is two") ... is two >>> 

Answer: It's important to remember that match-case is not designed to be used as switch-case. It's created for Structural Pattern Matching, and the syntax is suited for that purpose. Here you can check various possible patterns that may be given for case statements. One that matches what you used is Capture Pattern, which always succeeds. In order to match members of a class, you should use Class Patterns: class One: def __init__(self, arga): self.arga = arga class Two: def __init__(self, argb): self.argb = argb a = One(arga = 1) b = Two(argb = 2) match a: case One(): print("is one") case Two(): print("is two") match b: case One(): print("is one") case Two(): print("is two")  Note: One() in case statement is not resolved as simply creating instance of One and matching against it. If that were the case, we would get a TypeError as One.__init__ requires an argument.
Question: There is a python library that only wants to be installed in a virtual environment, but how to import the library into scripts running in my standalone application that does not run in a virtual environment? I'm writing a a Delphi/Lazarus application using the component Python4Delphi and Python4Lazarus to run python scripts.

Answer: There are two ways you can try to fix this:  In your Delphi/Lazarus application, you could configure Python4Delphi to use the Python interpreter from your virtual environment instead of the system Python. This way, it will automatically have access to all packages installed in that virtual environment.  PythonEngine1.DllPath := 'PATH';   You can add the venv's package directory into your standalone application like:  import sys import os venv_path = r"C:\path\to\your\venv\Lib\site-packages" if venv_path not in sys.path: sys.path.append(venv_path) # Now you can import your library import your_library
Question: I believe I am witnessing a memory leak in the following Python code that calls OpenCV functions. Can you reproduce this? Why does this happen? How can I work around it or fix it? My environment:  macOS (10.12 and 10.13) Python 3.8.10 OpenCV 3.4.18 NumPy 1.24.4  I've now also tested it on another machine in Python 3.9.1 with OpenCV 4.10.0 and NumPy 2.02 and experience the same behavior. I believe it's linked to cv2.imshow. With each loop iteration (i.e. every time the image updates) the memory usage of Python increases by an amount relative to the size of the image. Adding small = cv2.resize(img,(640,360)) and changing the imshow line to cv2.imshow('Image',small) makes the RAM increase by a much smaller amount. By inserting a continue statement above the imshow call, and thereby skipping imshow altogether, makes the RAM stay more or less unchanged throughout the loop. So, I guess this isolates the problem to cv2.imshow. import cv2, numpy as np flag = False while True: img = np.zeros((2160,3840,3),np.uint8) if flag: img = cv2.circle(img, (1920,1080),128,(255,255,255),-1) flag = not flag cv2.imshow('Image',img) k = cv2.waitKey() cv2.destroyAllWindows() if k == 27: break 

Answer: Thanks to user Christoph Rackwitz, the cause of the problem was identified as cv2.destroyAllWindows(). It seems to be present only on Mac systems and a bug report was filed, but apparently, there aren't currently any plans on fixing it.
Question: I have 2 DataFrame with different columns and want to merge to csv without comma for the one having single column. How can we remove comma for the one having single column? import pandas as pd # 1st DataFrame with single column pd_title = pd.DataFrame(['Category: A', '']) # 2nd DataFrame with double columns data = [ ["Date", "Value"], ['2025-01-01', 50], ['2025-01-02', 40], ['2025-01-03', 45] ] result = pd_title._append(data).reset_index(drop=True) result.to_csv('/content/test.csv', index=False, header=False)  The result from code :  The result what I mean : 

Answer: There is no direct way to do this in pandas since you're using rows of data as header. You could however convert to CSV string and post-process it: import re with open('/content/test.csv', 'w') as f: f.write(re.sub(',*\n,+\n', '\n\n', result.to_csv(index=False, header=False)))  A better option would be to first create the output file with the header, then export a dataframe with normal data/header and append it to the file with the mode='a' of to_csv: filename = '/content/test.csv' with open(filename, 'w') as f: f.write('Category: A\n\n') df = pd.DataFrame(data[1:], columns=data[0]) # Date Value # 0 2025-01-01 50 # 1 2025-01-02 40 # 2 2025-01-03 45 df.to_csv(filename, index=False, mode='a')  Output: Category: A Date,Value 2025-01-01,50 2025-01-02,40 2025-01-03,45
Question: How to extend this df = df.select( pl.col("x1").map_batches(custom_function).alias("new_x1") )  to something like df = df.select( pl.col("x1","x2").map_batches(custom_function).alias("new_x1", "new_x2") )  Or the way to go is doing it one by one df = df.select( pl.col("x1").map_batches(custom_function).alias("new_x1") pl.col("x2").map_batches(custom_function).alias("new_x2") ) 

Answer: The syntax df.select( pl.col("x1", "x2").some_method_chain() )  is equivalent to df.select( pl.col("x1").some_method_chain(), pl.col("x2").some_method_chain(), )  Especially, your example is almost correct, but fails on the last call to pl.Expr.alias in the method chain [...].alias("new_x1", "new_x2"). You basically try to set the name of each expression to "new_x1", "new_x2". This issue can be fixed using pl.Expr.name.prefix. df.select( pl.col("x1", "x2").map_batches(custom_function).name.prefix("new_") )
Question: I have a very large dataset (about 10^7 rows and 1000 columns) and need to make cuts into one of the columns, for trining/validation separation, with the bins changing based on another column. I am pretty new to python and am using this function: SEGMENT is either A, B or C, and DATE is what I am cutting (yes, it is a numerical column, I know it looks terrible but it was not my choice), with different bins for different values of SEGMENT. cuts = { "A": {"cut":[0,20240101,20240801,20241201,20250000], "class":["out", "training", "validation", "out"]}, "B": {"cut":[0,20230701,20240701,20241201,20250000], "class":["out", "training", "validation", "out"]}, "C": {"cut":[0,20230701,20240701,20250101,20250201], "class":["out", "training", "validation", "out"]} } def divisions(row): rules = cuts[row["SEGMENT"]] return pd.cut([row["DATE"]], bins=rules["cut"], labels=rules["class"], right=False, ordered=False)[0] df["CLASS"] = df.apply(divisions, axis=1)  This seems to work but has been insanely slow even on samples of less than 0.1% of the actual dataset. How can I improve this? All I need is this CLASS column, to check if the training and validation datasets show similar behaviors. I am not yet doing the actual modeling.

Answer: You could use a groupby.apply to handle all rows of a SEGMENT simultaneously: df['CLASS'] = (df.groupby('SEGMENT', group_keys=False)['DATE'] .apply(lambda x: pd.cut(x, bins=cuts[x.name]['cut'], labels=cuts[x.name]['class'], ordered=False)) )  Example output:  SEGMENT DATE CLASS 0 A 20240102 training 1 A 20241215 out 2 A 20231201 out 3 B 20240102 training 4 B 20241215 out 5 C 20231201 training 6 C 20240102 training 7 C 20241215 validation  Testing on 10K rows, this is ~1000x faster that the original approach: # groupby 3.68 ms ± 76.3 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) # original 3.34 s ± 10.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  Testing on 1M rows, this takes about half a second: 529 ms ± 277 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
Question: I am looking to convert a column with dates in a list [D, M, Y] to a datetime column. The below works but there must be a better way? new_df = pd.DataFrame({'date_parts': [[29, 'August', 2024], [28, 'August', 2024], [27, 'August', 2024]]}) display(new_df) ## Make new columns with dates new_df = pd.concat([new_df, new_df['date_parts'].apply(pd.Series)], axis=1).rename(columns={0:'D', 1:'M', 2:'Y'}) month_map = { 'January':1, 'February':2, 'March':3, 'April':4, 'May':5, 'June':6, 'July':7, 'August':8, 'September':9, 'October':10, 'November':11, 'December':12 } ## make datetime column new_df['release_date'] = pd.to_datetime(dict(year=new_df.Y, month=new_df.M.apply(lambda x: month_map[x]), day=new_df.D), format='%d-%B-%Y') new_df.drop(columns=['D', 'M', 'Y'])  ## Input date_parts 0 [29, August, 2024] 1 [28, August, 2024] 2 [27, August, 2024] ## Output date_parts release_date 0 [29, August, 2024] 2024-08-29 1 [28, August, 2024] 2024-08-28 2 [27, August, 2024] 2024-08-27 

Answer: Just combine the parts into a single string, and pass to to_datetime: new_df['release_date'] = pd.to_datetime(new_df['date_parts'] .apply(lambda x: '-'.join(map(str, x))), format='%d-%B-%Y')  Output:  date_parts release_date 0 [29, August, 2024] 2024-08-29 1 [28, August, 2024] 2024-08-28 2 [27, August, 2024] 2024-08-27  You could also convert the list of list to DataFrame with day/month/year columns: new_df['release_date'] = pd.to_datetime( pd.DataFrame( new_df['date_parts'].to_list(), index=new_df.index, columns=['day', 'month', 'year'], ).replace({'month': month_map}) )
Question: I was just writing a program where I wanted to insert a newline after a specific pattern. The idea was to match the pattern and replace with the overall match (i.e. capture group \0) and \n. s = "abc" insert_newline_pattern = re.compile(r"b") re.sub(insert_newline_pattern, r"\0\n", s)  However the output is a\x00\nc, reading \0 as a null character. I know that I can "simply" rewrite this as: s = "abc" insert_newline_pattern = re.compile(r"(b)") re.sub(insert_newline_pattern, r"\1\n", s)  which outputs the desired ab\nc with the idea of wrapping the overall match into group \1 and substituting this. See also a Python regex101 demo. Is there a way to access the overall match in any way, similar to this PCRE regex101 demo in Python?

Answer: You can use the form \g<0> in Python for the zeroeth group (or overall match from the pattern) which would be the same as $0 in PCRE (alternatively, in PCRE, you can use $& or \0 in replacement strings). s="abc" insert_newline_pattern=re.compile(r"b") re.sub(insert_newline_pattern,r"\g<0>\n",s)  Result: 'ab\nc'  This form is to avoid the potential ambiguity of \10 used in PCRE. Is that the tenth backreference or the first followed by a literal '0'? It is documented under the docs for re.sub.  Note: If you are referring to a match group, such as in a lambda in the replacement or as the result of re.search, you can also use .group(0) for the same function: s="abc123efg456hij" re.sub(r"[a-z](?!$)",lambda m: rf"{m.group(0)}\t",s) # Python 3.9+ you can use m[0] instead of m.group(0)  Result: a\tb\tc\t123e\tf\tg\t456h\ti\tj  Here is an example of using re.Match Object from re.search (or other re method that produces a match object): >>> s='abc123' >>> m=re.search(r'\d', s) >>> m[0] # what matched? $0 in PCRE '1' >>> m.span() # Where? (3, 4) >>> m.re # With what regex? re.compile('\\d')  If you want to see what re.sub would use as a string result, you can use match.expand: >>> m.expand(r"\g<0>\n") '1\n'
Question: Can someone explain what is meant by  Both loc and iloc [in Pandas] are row-first, column-second. This is the opposite of what we do in native Python, which is column-first, row-second.  Because I thought when accessing arrays or lists of lists, the first index always represents the row: matrix = [ [1,2,3], # row 1, index 0 [4,5,6], # row 2, index 1 [7,8,9] # row 3, index 2 ] print(matrix[1][2]) # Output = 6 

Answer: I would say that statement is incorrect or, at least, very misleading and likely to cause confusion. Both iloc and loc are row-first & column-second, but this is exactly the same as how indexing works in native Python and your example. First index refers to the row, and the second index refers to the column. Your example in pandas using iloc/loc also outputs 6: import pandas as pd data = [ [1, 2, 3], # row 0 [4, 5, 6], # row 1 [7, 8, 9] # row 2 ] df = pd.DataFrame(data) print(df.iloc[1, 2]) # Output: 6  There has already been some discussion about this exact statement in this Kaggle discussion, but to me is still not clear to what the author was referring to. As per Siraz Naorem understanding, the statement might be referring to the creation of DataFrames from column-oriented data, e.g. dictionaries, where each list or array represents a column, not a row. If we replicate again your example but create the DataFrame from a dictionary like this: df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6], 'C': [7, 8, 9]}) print(df) # Output: # A B C # 0 1 4 7 # 1 2 5 8 # 2 3 6 9  Now, when we access index [1,2], we do not get 6: print(df.iloc[1, 2]) # Output: 8 print(df.iloc[2, 1]) # Output: 6  In this case, the row and column indices might seem reversed and may lead to the mistaken idea that indexing is different: iloc[1,2] give us now 8, and we have to use iloc[2,1] to get the value 6. However, iloc/loc indexing itself has not changed, is still row-first & column-second, and what is different is the structure of the DataFrame, since pandas internally has treated each list in the dictionary as a column.
Question: How to parse xls data to this struct, both row and column have merged cells, simply use df.index.to_series().ffill() cannot handle. { "time": "time", "category": "A", "variety": "A1", "specification": "S1", "unit": "U1", "average": 1.25, "region": "RegionA", "market": "MarketA", "price": 1.1, }  

Answer: I figured out this solution: def test_xls_parse(): file_path = 'test.xls' df = pd.read_excel(file_path, engine='xlrd') time_label = df.iloc[0, 0] categories = df.iloc[1, 2:] varieties = df.iloc[2, 2:] specifications = df.iloc[3, 2:] units = df.iloc[4, 2:] averages = df.iloc[5, 2:] regions = df.iloc[6:, 0].ffill() markets = df.iloc[6:, 1] prices = df.iloc[6:, 2:] result = [] for i in range(len(categories)): for j in range(len(regions)): obj = { "date": time_label, "category": categories.iloc[i], "variety": varieties.iloc[i], "specification": specifications.iloc[i], "unit": units.iloc[i], "average": None if averages.iloc[i] == '-' else float(averages.iloc[i]), "region": regions.iloc[j], "market": markets.iloc[j], "price": None if prices.iloc[j, i] == '-' else float(prices.iloc[j, i]) } result.append(obj) return pd.DataFrame(result)
Question: I encountered some confusing behavior with polars type-casting (silently truncating floats to ints without raising an error, even when explicitly specifying strict=True), so I headed over to the documentation page on casting and now I'm even more confused. The text at the top of the page says:  The function cast includes a parameter strict that determines how Polars behaves when it encounters a value that cannot be converted from the source data type to the target data type. The default behaviour is strict=True, which means that Polars will thrown an error to notify the user of the failed conversion while also providing details on the values that couldn't be cast.  However, the code example immediately below (section title "Basic example") shows a df with a floats column taking values including 5.8 being truncated to int 5 during casting with the code pl.col("floats").cast(pl.Int32).alias("floats_as_integers"), i.e. without strict=False. What am I misunderstanding here? The text seems to indicate that this truncation, with strict=True as default, should "throw an error," but the code example in the documentation (and my own polars code) throws no error and silently truncates values.

Answer: It is accepted in Python (and more generally) that casting a float to an int will truncate the float and not raise an exception. E.g. in Python: >>> int(5.8) 5  Similarly, in Polars, casting a float to an int can be converted from the source data type to the target data type. For anyone else looking, this answer provides further detail / examples.
Question: Until now I used to convert all values in a pandas dataframe with combined numerical and string entries to numeric values if possible in one easy step, using .map and .to_numeric with "errors = 'ignore'". It worked perfectly, but after updating to the latest version of Pandas (2.2.3) I get a FutureWarning. import pandas as pd A = pd.DataFrame({ 'x' : ['1','2','3'], 'y' : ['not_a_number','5',9999], }) # example data B = A.map(pd.to_numeric, errors = 'ignore') # FutureWarning: errors='ignore' is deprecated and will raise in a future version. Use to_numeric without passing errors and catch exceptions explicitly instead B = A.map(pd.to_numeric, errors = 'ignore')  How could I code this future proof in an elegant, vectorised way? I could not think of any solution that is not very cumbersome (looping over each individual entry of the dataframe).

Answer: When you use errors='ignore', to_numeric returns the original Series. As mentioned in the documentation:  errors {‘ignore’, ‘raise’, ‘coerce’}, default ‘raise’  If ‘raise’, then invalid parsing will raise an exception. If ‘coerce’, then invalid parsing will be set as NaN. If ‘ignore’, then invalid parsing will return the input.  Changed in version 2.2. “ignore” is deprecated. Catch exceptions explicitly instead.  Catch the error explicitly if you want to keep the previous behavior: def to_numeric(s): try: return pd.to_numeric(s, errors='raise') except ValueError: return s A.apply(to_numeric)  NB. use apply rather than map for a vectorial operation. Relevant issues: #54467, #59221.