eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I want to find a way to "lasso around" a bunch of contiguous/touching values in a sparse table, and output a set of new tables. If any values are "touching", they should be part of a subarray together. For example: if I have the following sparse table/array: [[0 0 0 1 1 0 0 0 1 1 1 1 0 0 0 0 0 0 0] [0 0 0 1 1 0 0 0 1 1 1 1 1 0 0 1 1 0 0] [0 0 0 0 0 0 0 0 1 1 1 0 0 0 1 1 1 1 0] [0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0]]  The algorithm should "find" subtables/subarrays. It would identify them like this: [[0 0 0 1 1 0 0 0 2 2 2 2 0 0 0 0 0 0 0] [0 0 0 1 1 0 0 0 2 2 2 2 2 0 0 3 3 0 0] [0 0 0 0 0 0 0 0 2 2 2 0 0 0 3 3 3 3 0] [0 0 0 0 0 0 0 2 0 0 0 0 0 3 0 0 0 0 0]]  But the final output should be a series subarrays/subtables like this: [[1 1] [1 1]]  [[0 1 1 1 1 0] [0 1 1 1 1 1] [0 1 1 1 0 0] [1 0 0 0 0 0]]  [[0 0 1 1 0] [0 1 1 1 1] [1 0 0 0 0]]  How can I do this in python? I've tried looking at sk-image and a few things seem to be similar to what I'm trying to do, but nothing I have seen seems to fit quite right. EDIT: it looks like scipy.ndimage.label is extremely close to what I want to do, but it will break the corner-case values into their own separate arrays. So it's not quite right. EDIT: ah ha, the structure argument is what I am after. If I get time I will update my question with an answer.

Answer: A possible solution, which based on the following ideas:  First, measure.label assigns a unique label to each connected component in the array based on an 8-connectivity criterion (connectivity=2).  Second, measure.regionprops retrieves properties of these labeled regions, such as their bounding boxes.  Then, the code iterates through each detected region, extracts the minimum and maximum row and column indices from the region's bounding box, and slices the original array a to obtain the corresponding subarray.    labels = measure.label(a, connectivity=2) regions = measure.regionprops(labels) list_suba = [] for region in regions: min_row, min_col, max_row, max_col = region.bbox subarray = a[min_row:max_row, min_col:max_col] list_suba.append(subarray) list_suba   Or, more concisely: labels = measure.label(a, connectivity=2) regions = measure.regionprops(labels) [a[region.bbox[0]:region.bbox[2], region.bbox[1]:region.bbox[3]] for region in regions]   Output: [array([[1, 1], [1, 1]]), array([[0, 1, 1, 1, 1, 0], [0, 1, 1, 1, 1, 1], [0, 1, 1, 1, 0, 0], [1, 0, 0, 0, 0, 0]]), array([[0, 0, 1, 1, 0], [0, 1, 1, 1, 1], [1, 0, 0, 0, 0]])]
Question: It should match:  "abc 12-34 def" precisely "12-34" "Phone number: 123/45", precisely "123/45" "sequence: 12//-34", precisely "12//-34" "My code is 1-2-3-4", precisely "1-2-3-4"  It should not match:  "too many: 1234-567-89"  "too many; 1234-567"   Here is what I have tried: pattern = r'\d([\/-]\d){1,5}'  but didn't succeed

Answer: In your pattern, this part \d([\/-]\d expects a match for either / or - You might use a single digit, and repeat 1 - 5 times a digit with zero or more occurrences of - or / in between. On the left you can place a negative lookbehind and on the right a negative lookahead to assert not - / or a digit to mark the boundaries. (?<![/\d-])\d(?:[/-]*\d){1,5}(?![/\d-])  See a regex demo
Question: In a polars dataframe I have a column that contains the names of other columns (column "id_column_name"). I want to use those names in a when-then expression with pl.col() to create a new column ("id") which gets its values out of these other columns ("id_column1", "id_column2"). Every row can gets its value from another column in the df. # initial df df = pl.DataFrame({ 'id_column1': [123, 456], 'id_column2': ['abc', 'def'], 'id_column_name': ['id_column1', 'id_column2'] }) # required output df df_out = pl.DataFrame({ 'id_column1': [123, 456], 'id_column2': ['abc', 'def'], 'id_column_name': ['id_column1', 'id_column2'], 'id': ['123', 'def'] }) # one of the trings I tried df = df.with_columns( pl.when(pl.col('id_column_name').is_not_null()) .then(pl.col(pl.col('id_column_name'))) .otherwise(None) .cast(pl.String) .alias('id') )   This leads to the error: Expected str or DataType, got 'Expr'.  Using str() or .str to create the expression into a regular string lead to other errors:  Expected str or DataType, got 'ExprStringNameSpace'.  This cant be that hard, can it?

Answer:  pl.Series.unique() to get all possible values of id_column_name column. pl.when() to create conditional results. pl.coalesce() to fill the final result with first non-empty value.  df.with_columns( id = pl.coalesce( pl.when(pl.col.id_column_name == col).then(pl.col(col)) for col in df.schema if col not in ("id_column_name") # or this if amount of columns is large # for col in df["id_column_name"].unique() ) )  shape: (2, 4) ┌────────────┬────────────┬────────────────┬─────┐ │ id_column1 ┆ id_column2 ┆ id_column_name ┆ id │ │ --- ┆ --- ┆ --- ┆ --- │ │ i64 ┆ str ┆ str ┆ str │ ╞════════════╪════════════╪════════════════╪═════╡ │ 123 ┆ abc ┆ id_column1 ┆ 123 │ │ 456 ┆ def ┆ id_column2 ┆ def │ └────────────┴────────────┴────────────────┴─────┘
Question: I'm writing a function that needs to determine whether an object passed to it is an imputer (can check with isinstance(obj, _BaseImputer)), a scaler, or something else. While all imputers have a common base class that identifies them as imputers, scalers do not. I found that all scalers in sklearn.preprocessing._data inherit (OneToOneFeatureMixin, TransformerMixin, BaseEstimator), so I could check if they are instances of all of them. However that could generate false positives (not sure which other object may inherit the same base classes). It doesn't feel very clean or pythonic either. I was also thinking of checking whether the object has the .inverse_transform() method. However, not only scalers have that, a SimpleImputer (and maybe other objects) have also. How can I easily check if my object is a scaler?

Answer: Unfortunately, the cleanest way to do this is to check each scaler type individually, any other check will potentially let through non-scaler objects as well. Nevertheless, I'll offer some "hack-jobs" too. The most failsafe solution is to import your scalers and then check if your object is any of these scalers or not. from sklearn.preprocessing import MinMaxScaler, RobustScaler # ... other scalers your code base uses SCALER_TYPES = [MinMaxScaler, RobustScaler] # Extend list if needed if any([isinstance(YourObject, scaler_type) for scaler_type in SCALER_TYPES]): # Do something pass else: # Do something else pass  Now, if you want something that catches them all without listing all the scalers you use in your code, you could rely on private properties of the scaler objects. These are private for a good reason though, and are subject to change without notice even between patch versions, so nothing at all guarantees that your code will work if you update sklearn to a new version. You could rely on the string representation (__repr__) of the object to check if it contains Scaler. This is how you can do it: if 'Scaler' in str(YourObject): # Do something pass else: # Do something else pass  or if 'Scaler' in YourObject.__repr__(): # Do something pass else: # Do something else pass  This will let through anything that has Scaler in its string representation though, so you are definitely better off with being explicit and defining your list of scalers.
Question: I have two dataframes say df1 and df2, for example import pandas as pd col_1= ["A", ["B","C"], ["A","C","D"], "D"] col_id = [1,2,3,4] col_2 = [1,2,2,3,3,4,4] d1 = {'ID': [1,2,3,4], 'Labels': col_1} d2 = {'ID': col_2, } d_2_get = {'ID': col_2, "Labels": ["A", "B", "C", "A", "C", "D", np.nan] } df1 = pd.DataFrame(data=d1) df2 = pd.DataFrame(data=d2) df_2_get = pd.DataFrame(data=d_2_get)  df1 looking like  ID col2 0 1 A 1 2 [B, C] 2 3 [A, C, D] 3 4 D  and df2 looking like  ID 0 1 1 2 2 2 3 3 4 3 5 4 6 4  I want to add a column Labels to df2, taken from df1, in such a way that:  for index i, start with the first value in df1 if the new row in df2["ID"] has a repeated entry, get the next value in df1, if it exists. If not, set NaN.  Given df1 and df2, the output should look like df_2_get below  ID Labels 0 1 A 1 2 B 2 2 C 3 3 A 4 3 C 5 4 D 6 4 NaN  My current clumsy attempt is below, from collections import Counter def list_flattener(list_of_lists): return [item for row in list_of_lists for item in row] def my_dataframe_filler(df1, df2): list_2_fill = [] repeats = dict(Counter(df2["ID"])) for k in repeats.keys(): available_labels_list = df1[df1["ID"]==k]["Labels"].tolist() available_labels_list+=[[np.nan]*10] available_labels_list = list_flattener(available_labels_list) list_2_fill+=available_labels_list[:repeats[k]] return list_2_fill  and then use as df2["Labels"] = my_dataframe_filler(df1, df2)  but I would like to learn how a pandas black belt would handle the problem, thanks

Answer: IIUC, you could explode and perform a merge after deduplication with groupby.cumcount: out = (df2 .assign(n=df2.groupby('ID').cumcount()) .merge(df1.explode('Labels').assign(n=lambda x: x.groupby('ID').cumcount()), on=['ID', 'n'], how='left' ) #.drop(columns='n') )  Output:  ID n Labels 0 1 0 A 1 2 0 B 2 2 1 C 3 3 0 A 4 3 1 C 5 4 0 D 6 4 1 NaN  Alternatively, a pure python approach using iterators, map and next: # for each list, build an iterator d = dict(zip(df1['ID'], map(iter, df1['Labels']))) # take the appropriate list and get the next available item # default to None if exhausted df2['Labels'] = df2['ID'].map(lambda x: next(d[x], None))  Output:  ID Labels 0 1 A 1 2 B 2 2 C 3 3 A 4 3 C 5 4 D 6 4 None
Question: How can we pass aggregation functions as argument to a custom aggregation function in Polars? You should be able to pass a single function for all columns or a dictionary if you have different aggregations by column. import polars as pl # Sample DataFrame df = pl.DataFrame({ "category": ["A", "A", "B", "B", "B"], "value": [1, 2, 3, 4, 5] }) def agg_with_sum(df: pl.DataFrame | pl.LazyFrame) -> pl.DataFrame | pl.LazyFrame: return df.group_by("category").agg(pl.col("*").sum()) # Custom function to perform aggregation def agg_with_expr(df: pl.DataFrame | pl.LazyFrame, agg_expr: pl.Expr | dict[str, pl.Expr]) -> pl.DataFrame | pl.LazyFrame: if isinstance(agg_expr, dict): return df.group_by("category").agg([pl.col(col).aggexpr() for col, aggexpr in agg_expr.items()]) return df.group_by("category").agg(pl.col("*").agg_expr()) # Trying to pass a Polars expression for sum aggregation print(agg_with_sum(df)) # ┌──────────┬───────┐ # │ category ┆ value │ # │ --- ┆ --- │ # │ str ┆ i64 │ # ╞══════════╪═══════╡ # │ A ┆ 3 │ # │ B ┆ 12 │ # └──────────┴───────┘ # Trying to pass a custom Polars expression print(agg_with_expr(df, pl.sum)) # AttributeError: 'Expr' object has no attribute 'agg_expr' print(agg_with_expr(df, {'value': pl.sum})) # AttributeError: 'Expr' object has no attribute 'aggexpr' 

Answer: You can pass it as anonymous function with expression as parameter (I simplified your example just to illustrate the point): def agg_with_expr(df, agg_expr): return df.group_by("category").agg(agg_expr(pl.col("*"))) agg_with_expr(df, lambda x: x.sum())  shape: (2, 2) ┌──────────┬───────┐ │ category ┆ value │ │ --- ┆ --- │ │ str ┆ i64 │ ╞══════════╪═══════╡ │ B ┆ 12 │ │ A ┆ 3 │ └──────────┴───────┘  update. as @orlp mentioned in comments, in this particular case you could do it without anonymous function, with plain usage of pl.Expr.sum(), which is much more neat. agg_with_expr(df, pl.Expr.sum)  shape: (2, 2) ┌──────────┬───────┐ │ category ┆ value │ │ --- ┆ --- │ │ str ┆ i64 │ ╞══════════╪═══════╡ │ A ┆ 3 │ │ B ┆ 12 │ └──────────┴───────┘
Question: I have this python function and the bytecode it translates to: Text code: x = "-".join(str(z) for z in range(5)) assert x == "0-1-2-3-4" print("Assert test case for generator_expression_in_join") Disassembled code: 0 0 RESUME 0 2 2 LOAD_CONST 0 ('-') 4 LOAD_ATTR 1 (NULL|self + join) 24 LOAD_CONST 1 (<code object <genexpr> at 0x1026abe30, file "<dis>", line 2>) 26 MAKE_FUNCTION 0 28 PUSH_NULL 30 LOAD_NAME 1 (range) 32 LOAD_CONST 2 (5) 34 CALL 1 42 GET_ITER 44 CALL 0 52 CALL 1 60 STORE_NAME 2 (x) 3 62 LOAD_NAME 2 (x) 64 LOAD_CONST 3 ('0-1-2-3-4') 66 COMPARE_OP 40 (==) 70 POP_JUMP_IF_TRUE 2 (to 76) 72 LOAD_ASSERTION_ERROR 74 RAISE_VARARGS 1 4 >> 76 PUSH_NULL 78 LOAD_NAME 3 (print) 80 LOAD_CONST 4 ('Assert test case for generator_expression_in_join') 82 CALL 1 90 POP_TOP 92 RETURN_CONST 5 (None) Disassembly of <code object <genexpr> at 0x1026abe30, file "<dis>", line 2>: 2 0 RETURN_GENERATOR 2 POP_TOP 4 RESUME 0 6 LOAD_FAST 0 (.0) >> 8 FOR_ITER 15 (to 42) 12 STORE_FAST 1 (z) 14 LOAD_GLOBAL 1 (NULL + str) 24 LOAD_FAST 1 (z) 26 CALL 1 34 YIELD_VALUE 1 36 RESUME 1 38 POP_TOP 40 JUMP_BACKWARD 17 (to 8) >> 42 END_FOR 44 RETURN_CONST 0 (None) >> 46 CALL_INTRINSIC_1 3 (INTRINSIC_STOPITERATION_ERROR) 48 RERAISE 1  I have trouble understanding what the instruction with label 44 stands for. I understand that I have a range iterator on the top of my stack after instruction 42 (iter(range(5))), but I don't know why one would call an iterator. I'm trying to implement a python virtual machine, and am struggling to implement the CALL opcode correctly. I don't see any help in the provided spec. What is the logic behind doing CALL 0 on an iterator since an iterator isn't even callable?

Answer:  argc is the total of the positional and named arguments, excluding self when a NULL is not present.  Here’s one call:  28 PUSH_NULL 30 LOAD_NAME 1 (range) 32 LOAD_CONST 2 (5) 34 CALL 1  And here’s another:  24 LOAD_CONST 1 (<code object <genexpr> at 0x1026abe30, file "<dis>", line 2>) 26 MAKE_FUNCTION 0 ⋮ 42 GET_ITER 44 CALL 0  That is, you’re calling the generator expression with the iterator as self. Note that this instruction is different again in Python 3.13.
Question: I haven't been able to find anything online about this. Pylance seems to be marking the mean_squared_error function from sklearn.metrics as deprecated, although only the squared parameter is deprecated. I am running Python through micromamba and have the latest version of both sklearn (1.5.2) and Pylance (v2024.10.1).  I have uninstalled and reinstalled scikit-learn in my micromamba environment as well as updating micromamba itself.

Answer: Pyright/Pylance made no mistake, and neither did you. This is a problem with the type stubs for sklearn.metrics, in which mean_squared_error is defined as: @deprecated() def mean_squared_error( y_true: MatrixLike | ArrayLike, y_pred: MatrixLike | ArrayLike, *, sample_weight: None | ArrayLike = None, multioutput: ArrayLike | Literal["raw_values", "uniform_average", "uniform_average"] = "uniform_average", squared: bool = True, ) -> ndarray | Float: ...  The decorator was added recently in this PR. I created an issue and submitted a fix. Since these stubs come with Pylance, you will have to wait for the next version of Pylance with which the fix is included. In the meantime, you can either ignore the warning or enable the python.analysis.disableTaggedHints setting:
Question: Here is a function to approximate log10(x+1) for (x+1) < ~1.2: a = 1.097 b = 0.085 c = 2.31 ans = 1 / (a - b*x + c/x)  It should look like that:  It works by adjusting harmonic mean to match log10, but the problem is in values of a, b, c. The question is how to get just right a, b and c and how to make better approximation. I made this code that can give a pretty good approximation for a, b, c, but my code wasn't able to make it any better. import numpy as np a = 1 b = 0.01 c = 2 def mlg(t): x = t if t == 0: x = 0.00000001 x2 = x*x o = a - (b * x) + (c / x) return 1/o def mlg0(t): x = t if t == 0: x = 0.00000001 x2 = x*x o = a - (b * x) + (c / x) return o for i in range(9000): n1 = np.random.uniform(0,1.19,1000) for i in range(1000): n = n1[i] o = np.log10(n+1) u = mlg(n) - o e = u ** 2 de_da = 0 - 2 * (u) / (mlg0(n) ** 2) de_db = de_da * n de_dc = de_da / n a -= de_da * 0.00001 b -= de_db * 0.00001 c -= de_dc * 0.00001 print(a,b,c)  How could the code be changed to generate better values? I've used a method alike back propagation in NN, but it could not give me values any better. Here is how the error is calculated: 

Answer: Here are two approaches. Method 1: series expansion in x (better for negative and positive x) Method 2: fit the curve that passes through 3 points (here, x=0, ½ and 1) Method 1. If you expand them by Taylor series as powers of x then  Equating coefficients of x, x^2 and x^3 gives  In code: import math import numpy as np import matplotlib.pyplot as plt c = math.log( 10 ) a = c / 2 b = c / 12 print( "a, b, c = ", a, b, c ) x = np.linspace( -0.2, 0.2, 50 ) y = np.log10( 1 + x ) fit = x / ( a * x - b * x ** 2 + c ) plt.plot( x, y , 'b-', label="Original" ) plt.plot( x, fit, 'ro', label="Fit" ) plt.legend() plt.show()  Output: a, b, c = 1.151292546497023 0.19188209108283716 2.302585092994046   Method 2. Fit to three points. Here we require  If we require this to fit at x=0, ½ and 1 we get (including L’Hopital’s rule for the limit at x=0)  This time I have used your interval x in [0,1] to plot the fit import math import numpy as np import matplotlib.pyplot as plt c = math.log( 10 ) a = c * ( 2/math.log(1.5) - 1/math.log(2) - 3 ) b = 2 * c * ( 1/math.log(1.5) - 1/math.log(2) - 1 ) print( "a, b, c = ", a, b, c ) x = np.linspace( 0.0, 1.0, 50 ) y = np.log10( 1 + x ) fit = x / ( a * x - b * x ** 2 + c ) plt.plot( x, y , 'b-', label="Original" ) plt.plot( x, fit, 'ro', label="Fit" ) plt.legend() plt.show()  Output: a, b, c = 1.1280638006656465 0.1087207987723298 2.302585092994046
Question: I ran the following benchmark code on google collab CPU with high ram enabled. Please point out any errors in the way I am benchmarking, (if any) as well as why there is a such a high performance boost with tinygrad. # Set the size of the matrices size = 10000 # Generate a random 10000x10000 matrix with NumPy np_array = np.random.rand(size, size) # Generate a random 10000x10000 matrix with PyTorch torch_tensor = torch.rand(size, size) # Generate a random 10000x10000 matrix with TinyGrad tg_tensor = Tensor.rand(size, size) # Benchmark NumPy start_np = time.time() np_result = np_array @ np_array # Matrix multiplication np_time = time.time() - start_np print(f"NumPy Time: {np_time:.6f} seconds") # Benchmark PyTorch start_torch = time.time() torch_result = torch_tensor @ torch_tensor # Matrix multiplication torch_time = time.time() - start_torch print(f"PyTorch Time: {torch_time:.6f} seconds") # Benchmark TinyGrad start_tg = time.time() tg_result = tg_tensor @ tg_tensor # Matrix multiplication tg_time = time.time() - start_tg print(f"TinyGrad Time: {tg_time:.6f} seconds")   NumPy Time: 11.977072 seconds PyTorch Time: 7.905509 seconds TinyGrad Time: 0.000607 seconds  These were the results. After running the code many times, the results were very similar

Answer: Tinygrad performs operations in a "lazy" way, so the matrix multiplication hasn't been performed yet. Change your matrix multiplication line to: tg_result = (tg_tensor @ tg_tensor).realize()  or tg_result = (tg_tensor @ tg_tensor).numpy()
Question: My dataframe loads from a csv file that looks like this RepID Account Rank 123 Abcd 1 345 Zyxw 2 567 Hijk 3 ... ... 837 Kjsj 8  and I have another csv that has only one column RepID 345 488  I load the first csv in a dataframe df and the other csv in dataframe dE I want to have a new datafrmae dX that is all records from df that RepID does not exist in dE and dY all the records that RepID exist in dE how to do that?

Answer: A possible solution, which uses boolean indexing and isin: df[df['RepID'].isin(dE['RepID'])] # dY df[~df['RepID'].isin(dE['RepID'])] # dX  Output: # dY RepID Account Rank 1 345 Zyxw 2 # dX RepID Account Rank 0 123 Abcd 1 2 567 Hijk 3 3 837 Kjsj 8
Question: I'm struggling to split some text in a piece of code that I'm writing. This software is scanning through about 3.5 million lines of text of which there are varying formats throughout. I'm kind of working my way through everything still, but the line below appears to be fairly standard within the file: EXAMPLE_FILE_TEXT ID="20211111.111111 11111"  I want to split it as follows: EXAMPLE_FILE_TEXT, ID, 20211111.111111 11111  As much as possible, I'd prefer to avoid hard coding any certain text to look for as I'm still parsing through the file & trying to determine all the different variables. I've tried running the following code: conditioned_line = re.sub(r'(\w+=)(\w+)', r'\1"\2"', input_line) output = shlex.split(conditioned_line)  When I run this code, I'm getting this output: ['EXAMPLE_FILE_TEXT', 'ID=20211111.111111 11111']  I've managed to successfully split each and every element of this, but I have not managed to split them all together successfully. I suspect this is manageable via a regular expression, or with a regular expression and a shlex split, but I could really use some suggestions if anyone has some ideas. As requested, here's another example of some text that's in the file I'm scanning: EXAMPLE_TEXT TAG="AB-123-ABCD_$B" ABCDE_ABCD="ABCD_A" ABCDEF_ABCDE="ABCDEF_ABCDEF_$A" ABCDEFGH=""  This should separate to the following: EXAMPLE_TEXT, TAG, AB-123-ABCD_$B, ABCDE_ABCD, ABCD_A, ABCDEF_ABCDE, ABCDEF_ABCDEF_$A, ABCDEFGH 

Answer: I suggest a tokenizing approach with regex: create a regex with alternations, starting with the most specific ones, and ending with somewhat generic ones. In your case, you may try import re x = 'EXAMPLE_FILE_TEXT ID="20211111.111111 11111"' res = re.findall(r'"([^"]*)"|(\d+(?:\.\d+)*)|(\w+)', x) print( ["".join(r) for r in res] ) # => ['EXAMPLE_FILE_TEXT', 'ID', '20211111.111111 11111']  See the Python demo. The regex matches  "([^"]*)" - a string between two double quotes: " matches a ", then ([^"]*) captures zero or more chars other than " and then " matches a " char (NOTE: to match string between quotes with escaped quote support use "([^"\\]*(?:\\.[^"\\]*)*)", add a similar pattern for single quotes if needed) | - or (\d+(?:\.\d+)*) - Group 2: one or more digits and then zero or more sequences of . and one or more digits | - or (\w+) - Group 3: one or more word chars.
Question: I want to align the numbers to the top left corners of each cell. I was able to find the set_text_props, but it isn't doing the alignment as I expect. import matplotlib.pyplot as plt fig = plt.figure() ax = fig.add_subplot() ax.axis('off') table = ax.table( cellText=[[1, 2, 3, 4, 5, 6, 7], [8, 9, 10, 11, 12, 13, 14]], loc='center', cellLoc='left', ) table.set_fontsize(10) table.scale(1, 4) for (row, col), cell in table.get_celld().items(): cell.set_text_props(va='bottom') 

Answer: (This is a hack, not a solution) Add a newline to your text and adjust the cell.PAD. colab  import matplotlib.pyplot as plt fig = plt.figure(figsize=(4,3)) ax = fig.add_subplot() ax.axis('off') cellText=[[1, 2, 3, 4, 5, 6, 7], [8, 9, 10, 11, 12, 13, 14]] for i in range(len(cellText)): for j in range(len(cellText[i])): cellText[i][j] = f"{cellText[i][j]}\n" table = ax.table( cellText=cellText, loc='center', cellLoc='left', ) table.set_fontsize(10) table.scale(1, 4) for (row, col), cell in table.get_celld().items(): cell.set_text_props(va='bottom') cell.PAD=0.05
Question: I have a csv file with very old dates, and pd.to_datetime fails. It works in polars. Is this an inherent limitation in pandas, a bug or something else? import pandas as pd dates = ["12/31/1672","12/31/1677","10/19/2024"] df = pd.DataFrame(dates, columns=['Date']) df['Date'] = pd.to_datetime(df['Date'], format='%m/%d/%Y', errors='coerce') df Date 0 NaT 1 1677-12-31 2 2024-10-19  in polars import polars as pl df = pl.DataFrame({ 'Date': dates}) df = df.with_columns(pl.col('Date').str.strptime(pl.Date, format="%m/%d/%Y")) df shape: (3, 1) ┌────────────┐ │ Date │ │ --- │ │ date │ ╞════════════╡ │ 1672-12-31 │ │ 1677-12-31 │ │ 2024-10-19 │ └────────────┘ 

Answer: pandas has timestamp limitations; the docs suggests use of period for such cases (of course it depends if the period data type covers your use case): df.assign(new_dates=pd.PeriodIndex(df.Date, freq='D')) Date new_dates 0 12/31/1672 1672-12-31 1 12/31/1677 1677-12-31 2 10/19/2024 2024-10-19