eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Is there a smart, fast way to create an n x n numpy array filled with 0's and 1's with bias towards 0's? I did np.random.randint(2, (size,size)) but the bias is not accounted for here. I could do for loop but i want a faster cleaner way to populate the matrix. Thanks!

Answer: I would use numpy.random.choice with custom probabilities: n = 10 p = 0.9 # probability of 0s out = np.random.choice([0, 1], size=(n, n), p=[p, 1-p])  Another option, generating numbers in the 0-1 range, comparing to the probability of 0s and converting the booleans to integer: out = (np.random.random((n, n))>p).astype(int)  Example output: [[0 0 0 0 0 0 0 0 1 0] [0 0 0 0 0 0 0 0 0 0] [0 0 0 0 0 0 0 0 0 0] [0 1 0 0 0 0 0 0 0 1] [0 1 0 0 0 0 0 0 0 0] [0 0 0 0 0 0 0 0 0 0] [0 0 0 0 0 0 0 0 0 0] [0 1 0 0 0 0 0 0 0 0] [0 0 1 0 0 0 0 0 0 0] [0 0 0 0 0 0 0 0 0 1]]
Question: Question I'm trying to make all groups for a given data frame have the same size. In Starting point below, I show an example of a data frame that I whish to transform. In Goal I try to demonstrate what I'm trying to achieve. I want to group by the column group, make all groups have a size of 4, and fill 'missing' values with null - I hope it's clear. I have tried several approaches but have not been able to figure this one out. Starting point dfa = pl.DataFrame(data={'group': ['a', 'a', 'a', 'b', 'b', 'c'], 'value': ['a1', 'a2', 'a3', 'b1', 'b2', 'c1']}) ┌───────┬───────┐ │ group ┆ value │ │ --- ┆ --- │ │ str ┆ str │ ╞═══════╪═══════╡ │ a ┆ a1 │ │ a ┆ a2 │ │ a ┆ a3 │ │ b ┆ b1 │ │ b ┆ b2 │ │ c ┆ c1 │ └───────┴───────┘  Goal >>> make_groups_uniform(dfa, group_by='group', group_size=4) ┌───────┬───────┐ │ group ┆ value │ │ --- ┆ --- │ │ str ┆ str │ ╞═══════╪═══════╡ │ a ┆ a1 │ │ a ┆ a2 │ │ a ┆ a3 │ │ a ┆ null │ │ b ┆ b1 │ │ b ┆ b2 │ │ b ┆ null │ │ b ┆ null │ │ c ┆ c1 │ │ c ┆ null │ │ c ┆ null │ │ c ┆ null │ └───────┴───────┘  Package version polars: 1.1.0

Answer: The advantage of the approach below is that we don’t transform original DataFrame (except maybe sorting if you want to rearrange the groups), we only create additional rows and append them back to the original DataFrame.  I've adjusted my answer a bit, based on assumption that you want size of the group to be max of the size of all groups, but it works as well for fixed group_size.  group_by() allows per-group calculation. len() to determine size of the group. repeat_by() creates lists based on previously calculated group size and max() group size. filter() to filter out empty lists for case when we don't need to add extra rows. explode() lists into column. concat() back to existing DataFrame. (optional) sort() if you need groups to be together.  # you can use fixed group_size instead of pl.col.len.max() as well pl.concat([ dfa, ( dfa.group_by("group").len() .select(pl.col.group.repeat_by(pl.col.len.max() - pl.col.len)) .filter(pl.col.group.list.len() != 0) .explode("group") ) ], how="diagonal").sort("group")  shape: (9, 2) ┌───────┬───────┐ │ group ┆ value │ │ --- ┆ --- │ │ str ┆ str │ ╞═══════╪═══════╡ │ a ┆ a1 │ │ a ┆ a2 │ │ a ┆ a3 │ │ b ┆ b1 │ │ b ┆ b2 │ │ b ┆ null │ │ c ┆ c1 │ │ c ┆ null │ │ c ┆ null │ └───────┴───────┘  If you need fixed group size, then repeat() is probably more performant, but the idea is the same - only generate additional rows and append them back to original DataFrame. group_size = 3 # you can make it dynamic as well though # group_size = dfa.group_by("group").len().max()["len"].item() pl.concat([ dfa, ( dfa.group_by("group") .agg(value = pl.repeat(None, group_size - pl.len().cast(int))) .filter(pl.col.value.list.len() != 0) .explode("value") ) ]).sort("group")  shape: (9, 2) ┌───────┬───────┐ │ group ┆ value │ │ --- ┆ --- │ │ str ┆ str │ ╞═══════╪═══════╡ │ a ┆ a1 │ │ a ┆ a2 │ │ a ┆ a3 │ │ b ┆ b1 │ │ b ┆ b2 │ │ b ┆ null │ │ c ┆ c1 │ │ c ┆ null │ │ c ┆ null │ └───────┴───────┘
Question: Consider having two pl.DataFrames with identical schema. One of the columns has dtype=pl.Enum. import polars as pl enum_col1 = pl.Enum(["type1"]) enum_col2 = pl.Enum(["type2"]) df1 = pl.DataFrame( {"enum_col": "type1", "value": 10}, schema={"enum_col": enum_col1, "value": pl.Int64}, ) df2 = pl.DataFrame( {"enum_col": "type2", "value": 200}, schema={"enum_col": enum_col2, "value": pl.Int64}, ) print(df1) print(df2) shape: (1, 2) ┌──────────┬───────┐ │ enum_col ┆ value │ │ --- ┆ --- │ │ enum ┆ i64 │ ╞══════════╪═══════╡ │ type1 ┆ 10 │ └──────────┴───────┘ shape: (1, 2) ┌──────────┬───────┐ │ enum_col ┆ value │ │ --- ┆ --- │ │ enum ┆ i64 │ ╞══════════╪═══════╡ │ type2 ┆ 200 │ └──────────┴───────┘  If I try to do a simple pl.concat([df1, df2]), I get the following error: polars.exceptions.SchemaError: type Enum(Some(local), Physical) is incompatible with expected type Enum(Some(local), Physical) You can get around this issue by "enlarging" the enums like this: pl.concat( [ df1.with_columns(pl.col("enum_col").cast(pl.Enum(["type1", "type2"]))), df2.with_columns(pl.col("enum_col").cast(pl.Enum(["type1", "type2"]))), ] ) shape: (2, 2) ┌──────────┬───────┐ │ enum_col ┆ value │ │ --- ┆ --- │ │ enum ┆ i64 │ ╞══════════╪═══════╡ │ type1 ┆ 10 │ │ type2 ┆ 200 │ └──────────┴───────┘  I guess, there is a more pythonic way to do this?

Answer: you can cast enum_col to combined enum type: enum_col = enum_col1 | enum_col2 pl.concat( df.with_columns(pl.col.enum_col.cast(enum_col)) for df in [df1, df2] )  shape: (2, 2) ┌──────────┬───────┐ │ enum_col ┆ value │ │ --- ┆ --- │ │ enum ┆ i64 │ ╞══════════╪═══════╡ │ type1 ┆ 10 │ │ type2 ┆ 200 │ └──────────┴───────┘  You can also create new enum_col dynamically, for example: from functools import reduce enum_col = reduce(lambda x,y: x | y, [df.schema["enum_col"] for df in [df1, df2]])  Enum(categories=['type1', 'type2'])
Question: import pandas as pd concordance_region = pd.DataFrame( { "country 1": pd.Series([1, 0], index=["region a", "region b"]), "country 2": pd.Series([0, 1], index=["region a", "region b"]), "country 3": pd.Series([0, 1], index=["region a", "region b"]), } ) display(concordance_region) country_index = concordance_region.columns region_index = concordance_region.index sector_index = ['sector a','sector b'] country_sector = pd.MultiIndex.from_product([country_index, sector_index], names=["country", "sector"]) region_sector = pd.MultiIndex.from_product([region_index, sector_index], names=["region", "sector"])  concordance_region_expanded = pd.DataFrame([[1,0,0,0,0,0],[0,1,0,0,0,0],[0,0,1,0,1,0],[0,0,0,1,0,1]], index=region_sector, columns=country_sector) display(concordance_region_expanded)  I want to achieve the above expansion without hard-coding the number. An option is that: concordance_region_extended = pd.DataFrame(index=region_sector, columns=country_sector) for region in region_index: for sector_1 in sector_index: for country in country_index: for sector_2 in sector_index: if sector_1 == sector_2 and concordance_region.loc[region, country] == 1: concordance_region_expanded.loc[(region, sector_1),(country, sector_2)] = 1 concordance_region_expanded = concordance_region_expanded.fillna(value=0).infer_objects(copy=False) concordance_region_expanded  But I think the above code is neither efficient nor elegant. Any way to solve the above problem?

Answer: Code use np.kron and identity matrix (identity matrix can be created with np.eye.) import pandas as pd import numpy as np # taken from questioner's code sector_index = ['sector a', 'sector b'] country_sector = pd.MultiIndex.from_product( [country_index, sector_index], names=["country", "sector"]) region_sector = pd.MultiIndex.from_product( [region_index, sector_index], names=["region", "sector"]) # start answer n = len(sector_index) out = pd.DataFrame( np.kron(concordance_region.values, np.eye(n)), index=region_sector, columns=country_sector, dtype='int' )  out
Question: I'm using form prefix to render the same django form twice in the same template and avoid identical fields id's. When you do so, the separator between the prefix and the field name is '-', I would like it to be '_' instead. Is it possible ? Thanks

Answer: You could "monkey patch" [wiki] the BaseForm code, for example in some AppConfig: # app_name/config.py from django.apps import AppConfig class MyAppConfig(AppConfig): def ready(self): from django.forms.forms import BaseForm def add_prefix(self, field_name): return f'{self.prefix}_{field_name}' if self.prefix else field_name BaseForm.add_prefix = add_prefix But I would advise not to do this. This will normally generate the correct prefixes with an underscore. But some Django apps or some logic in the Django project itself might not use this method, and make the assumption it works with an hyphen instead. While probably most of such packages could indeed get fixed, it would take a lot of work.
Question: I'm having problems understanding why the inheritance is not working in the following example: import vlc class CustomMediaPlayer(vlc.MediaPlayer): def __init__(self, *args): super().__init__(*args) def custom_method(self): print("EUREKA") custom_mp = CustomMediaPlayer() print(custom_mp) custom_mp.custom_method()  This outputs: <vlc.MediaPlayer object at 0x7743d37db8f0> AttributeError: 'MediaPlayer' object has no attribute 'custom_method'  instead of a CustomMediaPlayer object, with the custom_method. Why is this happening? Is it because vlc.MediaPlayer is a _Ctype class?

Answer: This can happen if the base class overrides the __new__ method, which controls what happens when someone attempts to instantiate the class. The typical behaviour is that a new instance of the given class argument is created; but the __new__ function is free to return an existing object, or create an object according to its own internal logic. In their comment, Rogue links to the source code for the module, which does indeed override __new__. If there is a way to subclass vlc.MediaPlayer successfully, it should be listed in the module's documentation; otherwise the way to address this problem will depend on your specific requirements.
Question: I would like to understand why this works fine: >>> test_string = 'long brown fox jump over a lazy python' >>> 'formatted "{test_string[0]}"'.format(test_string=test_string) 'formatted "l"'  Yet this fails: >>> 'formatted "{test_string[-1]}"'.format(test_string=test_string) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: string indices must be integers >>> 'formatted "{test_string[11:14]}"'.format(test_string=test_string) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: string indices must be integers  I know this could be used: 'formatted "{test_string}"'.format(test_string=test_string[11:14])  ...but that is not possible in my situation. I am dealing with a sandbox-like environment where a list of variables is passed to str.format() as dictionary of kwargs. These variables are outside of my control. I know the names and types of variables in advance and can only pass formatter string. The formatter string is my only input. It all works fine when I need to combine a few strings or manipulate numbers and their precision. But it all falls apart when I need to extract a substring.

Answer: Why it doesn't work This is explained in the spec of str.format():  The arg_name can be followed by any number of index or attribute expressions. An expression of the form '.name' selects the named attribute using getattr(), while an expression of the form '[index]' does an index lookup using __getitem__().  That is, you can index the string using bracket notation, and the index you put inside the brackets will be the argument of the __getitem__() method of the string. This is indexing, not slicing. The bottom line is that str.format() simply doesn't support slicing of the replacement field (= the part between {}), as this functionality isn't part of spec. Regarding negative indices, the grammar specifies: element_index ::= digit+ | index_string  This means that the index can either be a sequence of digits (digit+) or a string. Since any negative index such as -1 is not a sequence of digits, it will be parsed as index_string. However, str.__getitem__() only supports arguments of type integer. Hence the error TypeError: string indices must be integers, not 'str'. Solutions to the problem Use f-strings >>> test_string = 'long brown fox jump over a lazy python' >>> f"formatted {test_string[0]}" 'formatted l' >>> f"formatted {test_string[0:2]}" 'formatted lo' >>> f"formatted {test_string[-1]}" 'formatted n'  Use str.format() but slice the argument of str.format() directly, rather than the replacement field >>> test_string = 'long brown fox jump over a lazy python' >>> 'formatted {replacement}'.format(replacement=test_string[0:2]) 'formatted lo' >>> 'formatted {replacement}'.format(replacement=test_string[-1]) 'formatted n'
Question: I want to do exactly the same thing as this post, but in python; aka given a list of natural integers, generate a random adjacency matrix whose degrees would match the list. I had great hope as the solution proposed uses a function from igraph, sample_degseq. However it seems like this function does not exist in the python version of igraph, at least as far as looked into it. I could program such a function myself but I'm not exactly smart enough to make it fast enough, and I would like this to be done in an efficient way.

Answer: The equivalent of R/igraph's sample_degseq() in python-igraph is Graph.Degree_Sequence(). Note that not all methods sample uniformly, and not all methods produce the same kind of graph (simple graph vs multigraph). "configuration_simple" and "edge_switching_simple" sample simple graphs uniformly. The former is exactly uniform (but very slow for anything but small degrees) and the latter almost exactly uniform. I recommend "edge_switching_simple". It basically generates a first graph using Graph.Realize_Degree_Sequence(), then it rewires it using rewire(), using 10 times as many steps as the number of edges.
Question: I would like to fill a value in a column based on another columns' name, in the Polars library from python (I obtained the following DF by exploding my variables' column names): Input: df = pl.from_repr(""" ┌────────┬─────────┬────────┬─────┬──────────┐ │ Name ┆ Average ┆ Median ┆ Q1 ┆ Variable │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ i64 ┆ i64 ┆ i64 ┆ str │ ╞════════╪═════════╪════════╪═════╪══════════╡ │ Apple ┆ 2 ┆ 3 ┆ 4 ┆ Average │ │ Apple ┆ 2 ┆ 3 ┆ 4 ┆ Median │ │ Apple ┆ 2 ┆ 3 ┆ 4 ┆ Q1 │ │ Banana ┆ 1 ┆ 5 ┆ 10 ┆ Average │ │ Banana ┆ 1 ┆ 5 ┆ 10 ┆ Median │ │ Banana ┆ 1 ┆ 5 ┆ 10 ┆ Q1 │ └────────┴─────────┴────────┴─────┴──────────┘ """)  Expected output: shape: (6, 6) ┌────────┬─────────┬────────┬─────┬──────────┬───────┐ │ Name ┆ Average ┆ Median ┆ Q1 ┆ Variable ┆ value │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ i64 ┆ i64 ┆ i64 ┆ str ┆ i64 │ ╞════════╪═════════╪════════╪═════╪══════════╪═══════╡ │ Apple ┆ 2 ┆ 3 ┆ 4 ┆ Average ┆ 2 │ │ Apple ┆ 2 ┆ 3 ┆ 4 ┆ Median ┆ 3 │ │ Apple ┆ 2 ┆ 3 ┆ 4 ┆ Q1 ┆ 4 │ │ Banana ┆ 1 ┆ 5 ┆ 10 ┆ Average ┆ 1 │ │ Banana ┆ 1 ┆ 5 ┆ 10 ┆ Median ┆ 5 │ │ Banana ┆ 1 ┆ 5 ┆ 10 ┆ Q1 ┆ 10 │ └────────┴─────────┴────────┴─────┴──────────┴───────┘  I have tried: df = df.with_columns(value = pl.col(f"{pl.col.variable}"))  But that does not work because polars perceives the argument as a function (?). Does anyone know how to do this? Note: I have also tried to transpose the dataframe, which, not only was that computationally expensive, also did not work! Because it would transpose the DF into a 5-rows-long DF. What I need is a (Name * Number of Variables)-rows-long DF. That is, for example, I have 3 different names (say, Apple, Banana, and Dragonfruit), and I have 3 variables (Average, Median, Q1), then my DF should be 9-rows-long!

Answer: You can use  when/then() to check whether the value of the column Variable is the same as the column name. coalesce() to choose first non-empty result.  df.with_columns( value = pl.coalesce( pl.when(pl.col.Variable == col).then(pl.col(col)) for col in df["Variable"].unique() ) ) ┌────────┬─────────┬────────┬─────┬──────────┬───────┐ │ Name ┆ Average ┆ Median ┆ Q1 ┆ Variable ┆ value │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ i64 ┆ i64 ┆ i64 ┆ str ┆ i64 │ ╞════════╪═════════╪════════╪═════╪══════════╪═══════╡ │ Apple ┆ 2 ┆ 3 ┆ 4 ┆ Average ┆ 2 │ │ Apple ┆ 2 ┆ 3 ┆ 4 ┆ Median ┆ 3 │ │ Apple ┆ 2 ┆ 3 ┆ 4 ┆ Q1 ┆ 4 │ │ Banana ┆ 3 ┆ 5 ┆ 10 ┆ Average ┆ 3 │ │ Banana ┆ 3 ┆ 5 ┆ 10 ┆ Median ┆ 5 │ │ Banana ┆ 3 ┆ 5 ┆ 10 ┆ Q1 ┆ 10 │ └────────┴─────────┴────────┴─────┴──────────┴───────┘
Question: I have a simple pl.DataFrame with a number of columns that only contain boolean values. import polars as pl df = pl.DataFrame( {"s1": [True, True, False], "s2": [False, True, True], "s3": [False, False, False]} ) shape: (3, 3) ┌───────┬───────┬───────┐ │ s1 ┆ s2 ┆ s3 │ │ --- ┆ --- ┆ --- │ │ bool ┆ bool ┆ bool │ ╞═══════╪═══════╪═══════╡ │ true ┆ false ┆ false │ │ true ┆ true ┆ false │ │ false ┆ true ┆ false │ └───────┴───────┴───────┘  I need to add another column that contains lists of varying length. A list in any individual row should contain the column name where the values of the columns S1, s2, and s3 have a True value. Here's what I am actually looking for: shape: (3, 4) ┌───────┬───────┬───────┬──────────────┐ │ s1 ┆ s2 ┆ s3 │ list │ │ --- ┆ --- ┆ --- │ --- │ │ bool ┆ bool ┆ bool │ list[str] │ ╞═══════╪═══════╪═══════╡══════════════╡ │ true ┆ false ┆ false │ ["s1"] │ │ true ┆ true ┆ false │ ["s1", "s2"] │ │ false ┆ true ┆ false │ ["s2"] │ └───────┴───────┴───────┴──────────────┘ 

Answer: List API You could build a list of when/then expressions and then remove the nulls. df.with_columns( pl.concat_list( pl.when(col).then(pl.lit(col)) for col in df.columns ) .list.drop_nulls() .alias("list") )  shape: (3, 4) ┌───────┬───────┬───────┬──────────────┐ │ s1 ┆ s2 ┆ s3 ┆ list │ │ --- ┆ --- ┆ --- ┆ --- │ │ bool ┆ bool ┆ bool ┆ list[str] │ ╞═══════╪═══════╪═══════╪══════════════╡ │ true ┆ false ┆ false ┆ ["s1"] │ │ true ┆ true ┆ false ┆ ["s1", "s2"] │ │ false ┆ true ┆ false ┆ ["s2"] │ └───────┴───────┴───────┴──────────────┘  Unpivot If "raw performance" is of concern, it can be done at the frame level. You can reshape with .unpivot() and .group_by to create the lists. (df.with_row_index() .unpivot(index="index") .filter(pl.col.value) .group_by("index", maintain_order=True) .agg(pl.col.variable.alias("list")) )  shape: (3, 2) ┌───────┬──────────────┐ │ index ┆ list │ │ --- ┆ --- │ │ u32 ┆ list[str] │ ╞═══════╪══════════════╡ │ 0 ┆ ["s1"] │ │ 1 ┆ ["s1", "s2"] │ │ 2 ┆ ["s2"] │ └───────┴──────────────┘  As we've maintained the order, we can horizontally .concat() to combine them. pl.concat( [ df, df.with_row_index() .unpivot(index="index") .filter(pl.col.value) .group_by("index", maintain_order=True) .agg(pl.col.variable.alias("list")) .drop("index") # optional ], how = "horizontal" )  shape: (3, 4) ┌───────┬───────┬───────┬──────────────┐ │ s1 ┆ s2 ┆ s3 ┆ list │ │ --- ┆ --- ┆ --- ┆ --- │ │ bool ┆ bool ┆ bool ┆ list[str] │ ╞═══════╪═══════╪═══════╪══════════════╡ │ true ┆ false ┆ false ┆ ["s1"] │ │ true ┆ true ┆ false ┆ ["s1", "s2"] │ │ false ┆ true ┆ false ┆ ["s2"] │ └───────┴───────┴───────┴──────────────┘  Timing As a basic comparison. bigger_df = df.sample(2_000_000, with_replacement=True)     Name Time     concat_list 1.4s   unpivot + concat 0.2s
Question: i recently ran into the problem, that all my python scripts which utilize the selenium module are broken apparently due to a google chrome update. It seems like selenium/google chrome always asks to select a user profile no matter what options are given inside the python script e.g. "user-data-dir" has no effect at all. This occurs even in headless mode. The original script was able to download a pdf in the background without the "user-data-dir" option. Has anyone run into the same issue and found a solution to it? Below an example code which worked fine before. from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.chrome.options import Options options = Options() options.add_argument('--headless') #options.add_argument(r"--user-data-dir=C:\Users\xxx\AppData\Local\Google\Chrome\User Data\Profile 1") # this option does not make any difference options.add_argument('log-level=3') options.add_experimental_option("prefs", { "download.default_directory": directory, "download.prompt_for_download": False, "download.directory_upgrade": True, "plugins.always_open_pdf_externally": True, }) driver = webdriver.Chrome(options=options) driver.get("http://www.google.com") #original code used a different website  Using selenium in python without having to select a user profile.

Answer: This is not an actual answer to selenium, but for now my solution is to abandon selenium and instead use playwright which can be used just the same way i need.
Question: I just started to learn pandas, and the call of df.iloc[[1][0]] (df is the pd.DataFrame data type with a shape of (60935, 54)) appeared in a code. Understanding df.iloc[[1 ][0]] should be a line of df, but how should we understand [[1][0]]? Why do the parameters in iloc[] allow the acceptance of two adjacent lists? How to deal with the inside of iloc[]? This is obviously not an index of rows and columns. In addition, I found that when the second number is not 0 or -1, there will be an index out-of-bounds error. Why is this? mydict = [{'a': 1, 'b': 2, 'c': 3, 'd': 4}, {'a': 100, 'b': 200, 'c': 300, 'd': 400}, {'a': 1000, 'b': 2000, 'c': 3000, 'd': 4000}] df = pd.DataFrame(mydict) print(df.iloc[[0][-1]].shape) output(4,) print(df.iloc[[0][0]].shape) output(4,) print(df.iloc[[0]].shape) output(1, 4) print(df.iloc[[0][1]].shape) outputIndexError: list index out of range print(type(df.iloc[[0]])) output<class 'pandas.core.frame.DataFrame'> print(type(df.iloc[[0][0]])) output<class 'pandas.core.series.Series'   

Answer: You misunderstood the syntax [0][-1] an so on. [0] is a list of length 1 containing the number 0 [1] is still al list of length 1 containing the number 1 [0][-1] means: "take the last element of the list [0]", which is equivalent to .iloc[0] [1][0] means: "take the first element of the list [1]", which is equivalent to .iloc[1] [0][1] means: "take the second element of the list [0], which does not exist because the list [0] does not have a second element. The error you get is not coming from .iloc, it is coming from the wrong list indexing [0][1].
Question: I would like to create a typed DataFrame from a Pydantic BaseModel class, let's call it MyModel that has Optional fields. As I create multiple instances of MyModel, some will have Optional fields with None values, and if I initialize a DataFrame with such rows, they will may have inconsistent column dtypes. I'd like thus to cast Optional[TypeX] to TypeX, e.g.: import pydantic import pandas as pd import numpy as np from typing import Optional class MyModel(pydantic.BaseModel): thisfield: int thatfield: Optional[str] ... col_types = {kk: ff.annotation for kk, ff in MyModel.model_fields.items()} pd.DataFrame(np.empty(0, dtype=[tuple(tt) for tt in col_types.items()]))  This fails with TypeError: Cannot interpret 'typing.Optional[str]' as a data type. I need a function or method of Optional[X] -> X. Any suggestions other than using repr with regex?

Answer: As long as Optional[X] is equivalent to Union[X, None]: from typing import Union, get_args, get_origin def get_optional_arg(typ: type) -> type | None: # make sure typ is really Optional[...], otherwise return None if get_origin(typ) is Union: args = get_args(typ) if len(args) == 2 and args[1] is type(None): return args[0] col_types = { k: get_optional_arg(f.annotation) or f.annotation for k, f in MyModel.model_fields.items() }