eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I was thinking about using polars in place of numpy in a parsing problem where I turn a structured text file into a character table and operate on different columns. However, it seems that polars is about 5 times slower than numpy in most operations I'm performing. I was wondering why that's the case and whether I'm doing something wrong given that polars is supposed to be faster. Example: import requests import numpy as np import polars as pl # Download the text file text = requests.get("https://files.rcsb.org/download/3w32.pdb").text # Turn it into a 2D array of characters char_tab_np = np.array(file.splitlines()).view(dtype=(str,1)).reshape(-1, 80) # Create a polars DataFrame from the numpy array char_tab_pl = pl.DataFrame(char_tab_np) # Sort by first column with numpy char_tab_np[np.argsort(char_tab_np[:,0])] # Sort by first column with polars char_tab_pl.sort(by="column_0")  Using %%timeit in Jupyter, the numpy sorting takes about 320 microseconds, whereas the polars sort takes about 1.3 milliseconds, i.e. about five times slower. I also tried char_tab_pl.lazy().sort(by="column_0").collect(), but it had no effect on the duration. Another example (Take all rows where the first column is equal to 'A'): # with numpy %%timeit char_tab_np[char_tab_np[:, 0] == "A"]  # with polars %%timeit char_tab_pl.filter(pl.col("column_0") == "A")  Again, numpy takes 226 microseconds, whereas polars takes 673 microseconds, about three times slower. Update Based on the comments I tried two other things: 1. Making the file 1000 times larger to see whether polars performs better on larger data. Results: numpy was still about 2 times faster (1.3 ms vs. 2.1 ms). In addition, creating the character array took numpy about 2 seconds, whereas polars needed about 2 minutes to create the dataframe, i.e. 60 times slower. To re-produce, just add text *= 1000 before creating the numpy array in the code above. 2. Casting to integer. For the original (smaller) file, casting to int sped up the process for both numpy and polars. The filtering in numpy was still about 5 times faster than polars (30 microseconds vs. 120), wheres the sorting time became more similar (150 microseconds for numpy vs. 200 for polars). However, for the large file, polars was marginally faster than numpy, but the huge instantiation time makes it only worth if the dataframe is to be queried thousands of times.

Answer: Polars does extra work in filtering string data that is not worth it in this case. Polars uses arrow large-utf8 buffers for their string data. This makes filtering more expensive than filtering python strings/chars (e.g. pointers or u8 bytes). Sometimes it is worth it, sometimes not. If you have homogeneous data, numpy is a better fit than polars. If you have heterogenous data, polars will likely be faster. Especially if you consider your whole query instead of these micro benchmarks.
Question: I am trying to automatically convert a Pydantic model to a DB schema. To do that, I am recursively looping through a Pydantic model's fields to determine the type of field. As an example, I have this simple model: from typing import List from pydantic import BaseModel class TestModel(BaseModel): tags: List[str]  I am recursing through the model using the __fields__ property as described here: https://docs.pydantic.dev/usage/models/#model-properties If I do type(TestModel).__fields__['tags'] I see: ModelField(name='tags', type=List[str], required=True)  I want to programatically check if the ModelField type has a List origin. I have tried the following, and none of them work:  type(TestModel).__fields__['tags'].type_ is List[str] type(TestModel).__fields__['tags'].type_ == List[str] typing.get_origin(type(TestModel).__fields__['tags'].type_) is List typing.get_origin(type(TestModel).__fields__['tags'].type_) == List  Frustratingly, this does return True:  type(TestModel).__fields__['tags'].type_ is str  What is the correct way for me to confirm a field is a List type?

Answer: Pydantic has the concept of the shape of a field. These shapes are encoded as integers and available as constants in the fields module. The more-or-less standard types have been accommodated there already. If a field was annotated with list[T], then the shape attribute of the field will be SHAPE_LIST and the type_ will be T. The type_ refers to the element type in the context of everything that is not SHAPE_SINGLETON, i.e. with container-like types. This is why you get str in your example. Thus for something as simple as list, you can simply check the shape against that constant: from pydantic import BaseModel from pydantic.fields import SHAPE_LIST class TestModel(BaseModel): tags: list[str] other: tuple[str] tags_field = TestModel.__fields__["tags"] other_field = TestModel.__fields__["other"] assert tags_field.shape == SHAPE_LIST assert other_field.shape != SHAPE_LIST  If you want more insight into the actual annotation of the field, that is stored in the annotation attribute of the field. With that you should be able to do all the typing related analyses like get_origin. That means another way of accomplishing your check would be this: from typing import get_origin from pydantic import BaseModel class TestModel(BaseModel): tags: list[str] other: tuple[str] tags_field = TestModel.__fields__["tags"] other_field = TestModel.__fields__["other"] assert get_origin(tags_field.annotation) is list assert get_origin(other_field.annotation) is tuple  Sadly, neither of those attributes are officially documented anywhere as far as I know, but the beauty of open-source is that we can just check ourselves. Neither the attributes nor the shape constants are obfuscated, protected or made private in any of the usual ways, so I'll assume these are stable (at least until Pydantic v2 drops).
Question: I have a list of dictionaries like this: [{"id": 1, "name": "Joe", "lastname": "Bloggs"}, {"id": 2, "name": "Bob", "lastname": "Wilson"}]  And I would like to transform it to a polars dataframe. I've tried going via pandas but if possible, I'd like to avoid using pandas. Any thoughts?

Answer: Just pass it to pl.DataFrame In [2]: pl.DataFrame([{"id": 1, "name": "Joe", "lastname": "Bloggs"}, {"id": 2, "name": "Bob", "lastname": "Wilson"}]) Out[2]: shape: (2, 3) ┌─────┬──────┬──────────┐ │ id ┆ name ┆ lastname │ │ --- ┆ --- ┆ --- │ │ i64 ┆ str ┆ str │ ╞═════╪══════╪══════════╡ │ 1 ┆ Joe ┆ Bloggs │ │ 2 ┆ Bob ┆ Wilson │ └─────┴──────┴──────────┘
Question: I am using a step and fill_between functions in Matplotlib and want the steps to be centred on the x points. Code import matplotlib.pyplot as plt import numpy as np xpoints=np.array([1,2,3,4]) ypoints=np.array([4,6,5,2]) ypoints_std=np.array([0.5,0.3,0.4,0.2]) plt.step(xpoints,ypoints,where='mid') plt.fill_between(xpoints,ypoints+ypoints_std,ypoints-ypoints_std,step='mid',alpha=0.2) plt.show()  Current plot:  At the moment, the step centred on 1 is only 0.5 wide, whereas the step centred on 2 is 1 wide. Wanted I actually want the step-width of 1 for all steps and also for the fill. This should include first and last step, so that they are extended compared to the current plot.  Of course I can pad the data, but that is getting messy in my actual code. Questions  Is there a way to make the first and last steps the same size as the middle ones? Or is there a way to produce a similar graph using histogram ? i.e. showing an error the size of the full width of the bar, centred on the y position of the graph? 

Answer: Using a bar plot at a height The error bands could be shown via a bar plot with a bottom at ypoints - ypoints_std and a height of 2*ypoints_std. import matplotlib.pyplot as plt import numpy as np xpoints = np.array([1, 2, 3, 4]) ypoints = np.array([4, 6, 5, 2]) ypoints_std = np.array([0.5, 0.3, 0.4, 0.2]) plt.bar(xpoints, ypoints, width=1, facecolor='none', edgecolor='dodgerblue') plt.bar(xpoints, height=2 * ypoints_std, bottom=ypoints - ypoints_std, width=1, color='dodgerblue', alpha=0.2) plt.xticks(xpoints) plt.show()   Using zero-height bars To only have horizontal lines, you could replace the first bar plot with zero-height bars. Adding the original plt.step with the same color will create the connecting lines plt.gca().use_sticky_edges = False # prevent bars from "sticking" to the bottom plt.step(xpoints, ypoints, where='mid', color='dodgerblue') plt.bar(xpoints, height=0, bottom=ypoints, width=1, facecolor='none', edgecolor='dodgerblue') plt.bar(xpoints, height=2 * ypoints_std, bottom=ypoints - ypoints_std, width=1, color='dodgerblue', alpha=0.2)   Extending the points You could add dummy values to repeat the first and last point. And then use plt.xlim(...) to limit the plot between 0.5 and 4.5. import matplotlib.pyplot as plt import numpy as np xpoints = np.array([1, 2, 3, 4]) ypoints = np.array([4, 6, 5, 2]) ypoints_std = np.array([0.5, 0.3, 0.4, 0.2]) xpoints = np.concatenate([[xpoints[0] - 1], xpoints, [xpoints[-1] + 1]]) ypoints = np.pad(ypoints, 1, mode='edge') ypoints_std = np.pad(ypoints_std, 1, mode='edge') plt.step(xpoints, ypoints, where='mid') plt.fill_between(xpoints, ypoints + ypoints_std, ypoints - ypoints_std, step='mid', alpha=0.2) plt.xlim(xpoints[0] + 0.5, xpoints[-1] - 0.5) plt.show()
Question: I have a dictionary setup like this: company = {'Honda': {} ,'Toyota':{} ,'Ford':{} }  I have a list containing data like this: years = ['Year 1', 'Year 2', 'Year 3']  Finally, I also have a list of lists containing data like this: sales = [[55,9,90],[44,22,67],[83,13,91]]  I am trying to achieve a final result that looks like this: {'Honda': {'Year 1':55,'Year 2':9,'Year 3':90} ,'Toyota':{'Year 1':44,'Year 2':22,'Year 3':67} ,'Ford':{'Year 1':83,'Year 2':13,'Year 3':91} }  I can access the sub-list if sales like this: for i in sales: for j in i: #this would give me a flat list of all sales  I can't seem to wrap my head around constructing the final dictionary that would tie everything together. Any help is appreciated!

Answer: You can use a dict comprehension with zip. res = {k : dict(zip(years, sale)) for k, sale in zip(company, sales)}
Question: I'm trying to create a class which has an attribute that should be a constant. This attribute could have different types depending on were the class is used in the codebase. Moreover, the type of this attribute is used in various type hints throughout the class¹, so I decided to convert the class to a Generic, like so: from typing import TypeVar, Generic, Final T = TypeVar("T") class Foo(Generic[T]): bar: Final[T] def __init__(self, bar: T) -> None: self.bar = bar  However, MyPy complains saying that  Final name declared in class body cannot depend on type variables  while if I remove the Final annotation MyPy doesn't raise any errors. I can't find any logical errors in my code: it simply says that, regardless of its type, the bar attribute should always be referencing the same object. Am I missing something or is this some limitation of Python and/or of MyPy? ¹In the example I'm showing only one such usage to keep things simple. P.S. I'm using Python 3.10.8 and MyPy 0.991.

Answer: Your code is correct per PEP591, mypy applies the rules in a wrong order, annotate in __init__ to solve the issue. Here are the links to the docs and PEP591. mypy should've checked the presence of initializer and then decide whether the initializer is missing, but it doesn't in fact and thinks that you define a final class attribute despite missing initializer. Type variables make no sense in types of class attributes (because type variables are bound to instances, not classes), so mypy errors here. Thus we need to help mypy resolve the kind of Final properly. To do so, we can annotate attribute in __init__ instead: from typing import TypeVar, Generic, Final _T = TypeVar("_T") class Foo(Generic[_T]): def __init__(self, bar: _T): self.bar: Final[_T] = bar  This typechecks now (playground).
Question: I am not very familiar with python, I only done automation with so I am a new with packages and everything. I am creating an API with Flask, Gunicorn and Poetry. I noticed that there is a version number inside the pyproject.toml and I would like to create a route /version which returns the version of my app. My app structure look like this atm: ├── README.md ├── __init__.py ├── poetry.lock ├── pyproject.toml ├── tests │ └── __init__.py └── wsgi.py  Where wsgi.py is my main file which run the app. I saw peoples using importlib but I didn't find how to make it work as it is used with:  __version__ = importlib.metadata.version("__package__") But I have no clue what this package mean.

Answer: You should not use __package__, which is the name of the "import package" (or maybe import module, depending on where this line of code is located), because this is not what is expected here. importlib.metadata.version() expects the name of the "distribution package" (the thing that you pip-install), which is the one you write in the [project] table of pyproject.toml as name = "my-distribution-package-name". So if the application or library whose version string you want to get is named my-foo-lib, then you only need to call the following import importlib.metadata version_string_of_foo = importlib.metadata.version('my-foo-lib')
Question: I am using project.toml file to package my module, I want to extract the version from git tag using setuptools_scm module. When I run python setup.p y --version command it gives this output 0.0.1.post1.dev0. How will I get only 0.0.1 value and omit the .post.dev0 value? Here is project.toml file settings: [build-system] requires = ["setuptools>=46.1.0", "setuptools_scm[toml]>=5"] build-backend = "setuptools.build_meta" [tool.setuptools_scm] version_scheme = "no-guess-dev" local_scheme="no-local-version" write_to = "src/showme/version.py" git_describe_command = "git describe --dirty --tags --long --match v* --first-parent" [tool.setuptools.dynamic] version = {attr = "showme.__version__"}  output:  python setup.py --version setuptools/config/pyprojecttoml.py:108: _BetaConfiguration: Support for `[tool.setuptools]` in `pyproject.toml` is still *beta*. warnings.warn(msg, _BetaConfiguration) 0.0.1.post1.dev0  Thanks

Answer: setuptools_scm out-of-the-box generates development and post-release versions. To generate a release version like 0.0.1, you can pass a callable into use_scm_version: # content of setup.py def my_version(): from setuptools_scm.version import SEMVER_MINOR, guess_next_simple_semver, release_branch_semver_version def my_release_branch_semver_version(version): v = release_branch_semver_version(version) if v == version.format_next_version(guess_next_simple_semver, retain=SEMVER_MINOR): return version.format_next_version(guess_next_simple_semver, fmt="{guessed}", retain=SEMVER_MINOR) return v return { 'version_scheme': my_release_branch_semver_version, 'local_scheme': 'no-local-version', } setup(use_scm_version=my_version)  Reference: https://github.com/pypa/setuptools_scm#importing-in-setuppy
Question: I have a polars DataFrame for example: >>> df = pl.DataFrame({'A': ['a', 'b', 'c', 'd'], 'B': ['app', 'nop', 'cap', 'tab']}) >>> df shape: (4, 2) ┌─────┬─────┐ │ A ┆ B │ │ --- ┆ --- │ │ str ┆ str │ ╞═════╪═════╡ │ a ┆ app │ │ b ┆ nop │ │ c ┆ cap │ │ d ┆ tab │ └─────┴─────┘  I'm trying to get a third column C which is True if strings in column B starts with the strings in column A of the same row, otherwise, False. So in the case above, I'd expect: ┌─────┬─────┬───────┐ │ A ┆ B ┆ C │ │ --- ┆ --- ┆ --- │ │ str ┆ str ┆ bool │ ╞═════╪═════╪═══════╡ │ a ┆ app ┆ true │ │ b ┆ nop ┆ false │ │ c ┆ cap ┆ true │ │ d ┆ tab ┆ false │ └─────┴─────┴───────┘  I'm aware of the df['B'].str.starts_with() function but passing in a column yielded: >>> df['B'].str.starts_with(pl.col('A')) ... # Some stuff here. TypeError: argument 'sub': 'Expr' object cannot be converted to 'PyString'  What's the way to do this? In pandas, you would do: df.apply(lambda d: d['B'].startswith(d['A']), axis=1) 

Answer: Expression support was added for .str.starts_with() in pull/6355 as part of the Polars 0.15.17 release. df.with_columns(pl.col("B").str.starts_with(pl.col("A")).alias("C"))  shape: (4, 3) ┌─────┬─────┬───────┐ │ A | B | C │ │ --- | --- | --- │ │ str | str | bool │ ╞═════╪═════╪═══════╡ │ a | app | true │ │ b | nop | false │ │ c | cap | true │ │ d | tab | false │ └─────┴─────┴───────┘
Question: I have a polars dataframe: df = pl.DataFrame({'schema_name': ['test_schema', 'test_schema_2'], 'table_name': ['test_table', 'test_table_2'], 'column_name': ['test_column, test_column_2','test_column']})     schema_name table_name column_name     test_schema test_table test_column, test_column_2   test_schema_2 test_table_2 test_column    I have a string: date_field_value_max_query = ''' select '{0}' as schema_name, '{1}' as table_name, greatest({2}) from {0}.{1} group by 1, 2 '''  I would like to use polars to add a column by using string formatting. The target dataframe is this:    schema_name table_name column_name query     test_schema test_table test_column, test_column_2 select test_schema, test_table, greatest(test_column, test_column_2) from test_schema.test_table group by 1, 2   test_schema_2 test_table_2 test_column select test_schema_2, test_table_2, greatest(test_column) from test_schema_2.test_table_2 group by 1, 2    In pandas, I would do something like this: df.apply(lambda row: date_field_value_max_query.format(row['schema_name'], row['table_name'], row['column_name']), axis=1)  For polars, I tried this: df.map_rows(lambda row: date_field_value_max_query.format(row[0], row[1], row[2]))  ...but this returns only the one column, and I lose the original three columns. I know this approach is also not recommended for polars, when possible. How can I perform string formatting across multiple dataframe columns with the output column attached to the original dataframe?

Answer: Another option is to use polars.format to create your string. For example: date_field_value_max_query = ( '''select {} as schema_name, {} as table_name, greatest({}) from {}.{} group by 1, 2 ''' ) ( df .with_columns( pl.format(date_field_value_max_query, 'schema_name', 'table_name', 'column_name', 'schema_name', 'table_name') ) )  shape: (2, 4) ┌───────────────┬──────────────┬────────────────────────────┬─────────────────────────────────────────────┐ │ schema_name ┆ table_name ┆ column_name ┆ literal │ │ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ str ┆ str ┆ str │ ╞═══════════════╪══════════════╪════════════════════════════╪═════════════════════════════════════════════╡ │ test_schema ┆ test_table ┆ test_column, test_column_2 ┆ select test_schema as schema_name, │ │ ┆ ┆ ┆ test_table as table_name, │ │ ┆ ┆ ┆ greatest(test_column, test_column_2) │ │ ┆ ┆ ┆ from test_schema.test_table │ │ ┆ ┆ ┆ group by 1, 2 │ │ ┆ ┆ ┆ │ │ ┆ ┆ ┆ │ │ test_schema_2 ┆ test_table_2 ┆ test_column ┆ select test_schema_2 as schema_name, │ │ ┆ ┆ ┆ test_table_2 as table_name, │ │ ┆ ┆ ┆ greatest(test_column) │ │ ┆ ┆ ┆ from test_schema_2.test_table_2 │ │ ┆ ┆ ┆ group by 1, 2 │ │ ┆ ┆ ┆ │ │ ┆ ┆ ┆ │ └───────────────┴──────────────┴────────────────────────────┴─────────────────────────────────────────────┘
Question: I am using Poetry of version 1.3.2 (currently the last version), and added group to .toml file as below: [tool.poetry.group.dev.dependencies]. And following official documentation tried to add library to this group using command: poetry add pytest --group dev. But always getting error that says: The "--group" option does not exist. (Using python of version 3.9.15) I tried to change version of poetry to 1.2.x, but it did not help.

Answer: Just type: poetry add --group dev 'package_name'  That should work
Question: I see in many places an approach for getting SqlAlchemy session just like this one below: async def get_session() -> AsyncSession: async with async_session() as session: yield session  It used together with Depends: @app.post("/endpoint") async def vieww(session: AsyncSession = Depends(get_session)): session.execute(some_statement)  So my question is how to get a session from get_session ooutside Depends? I had a lot of attempts and got a headache.. I tried with s = await get_session() s.execute(stmt)  And I get AttributeError: 'async_generator' object has no attribute 'execute'

Answer: get_session is an asynchronous generator function. It returns an asynchronous iterator. You can't await those. You can await their __anext__ method though. from asyncio import run from collections.abc import AsyncIterator async def get_generator() -> AsyncIterator[int]: yield 1 async def main() -> None: generator = get_generator() print(await generator.__anext__()) if __name__ == "__main__": run(main())  Output: 1 Since Python 3.10 we can use the built-in anext, which makes this look less hacky: ... async def main() -> None: generator = get_generator() print(await anext(generator) ...  So in your case, you could use s = await anext(get_session()). But I struggle to see the use of this. This get_session function is designed as an asynchronous generator for the specific use case in the FastAPI Depends construct. Outside of that, you may as well just use the context manager normally: async with async_session() as session: ... # do things with the session session.execute()  Otherwise you'll have to remember to call __anext__ on that generator again (and catch the StopAsyncIteration error), otherwise the session will never be closed.
Question: I am working on an animation of a 3D plot using mpl_toolkits.mplot3d (Matplotlib 3.6.3) and need to set the view distance. It seems that earlier versions of Matplotlib allowed the elevation, azimuth, and distance of the viewpoint "camera" to be set for 3D plots using methods like this: ax.elev = 45 ax.azim = 10 ax.dist = 2  but the distance attribute appears to have been deprecated for some reason: Warning (from warnings module): ax.dist = 2 MatplotlibDeprecationWarning: The dist attribute was deprecated in Matplotlib 3.6 and will be removed two minor releases later.  This still runs, but the output plots have all sorts of visual artifacts that only go away if I shut off the axes with ax.set_axis_off(). Is there an equivalent means of setting the viewpoint distance to zoom in on a 3D data set in 3.6.3?

Answer: According to the documentation, you now need to use the zoom argument of set_box_aspect: ax.set_box_aspect(None, zoom=2) Where the first argument is the aspect ratio. Use None to use the default values.