eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: What is the difference between using square brackets [ ] and using Expression APIs like select, filter, etc. when trying to access data from a polars Dataframe? Which one to use when? a polars dataframe df = pl.DataFrame( { "a": ["a", "b", "a", "b", "b", "c"], "b": [2, 1, 1, 3, 2, 1], } ) 

Answer: Usually, it is advised to use the polars expression API as most of the methods are part of polars lazy API. The API defers the evaluation of many operations, such as selection, filtering, and mutation, until the result is actually needed. This allows for the query optimizations that makes polars as efficient as it is. In contrast, accessing columns using square bracket notation only work in eager-mode. Example. As an concrete example, consider reading the first element of a rather large .csv file (~700mb). We start by creating the .csv file and write it to disk. import polars as pl pl.DataFrame({"col": [0] * 100_000_000}).write_csv("df.csv")  Using square bracket notation ([). As square bracket notation does not work on LazyFrames, we'll need to use pl.read_parquet to read the file into a DataFrame object. pl.read_csv("df.csv")["col"][0]  On my machine, this takes roughly 200ms. Using polars lazy API. The same can be achieved using polars lazy API (pl.scan_csv, select, first, item). pl.scan_csv("df.csv").select("col").first().collect().item()  The delay of the evaluation allows polars to avoid reading in the entire file and the execution only takes 300μs on my machine, i.e. we obtained a ~650x speed-up by using polars' lazy API. Of course, one could argue that much of the time here is saved by avoiding to read the entire .csv file instead of by using select over [. However, note that this is still due to the benefits of the lazy API.
Question: data = {"col1": ['2020/01/01', '2020/02/01'], "col2": ['2020/01/01', '2020/02/01']} df = pl.DataFrame(data, schema={"col1": pl.String, "col2": pl.String}) df = df.with_columns( pl.col('col1').str.to_datetime(), pl.col('col2').str.to_datetime() ) df.with_columns( pl.col(pl.DATETIME_DTYPES).dt.year() )  With the given code I want to create new columns for every column selected by pl.DATETIME_DTYPES with the extracted year. For a single column I would apply .alias() but what to do for potentially n new columns? Any generic way?

Answer: For this, you can use the methods in the pl.Expr.name namespace. The most flexible method would be pl.Expr.name.map, but there is also pl.Expr.name.prefix, pl.Expr.name.suffix, etc. df.with_columns( pl.col(pl.DATETIME_DTYPES).dt.year().name.map(lambda s: s + "_year") )  Output. shape: (2, 4) ┌─────────────────────┬─────────────────────┬───────────┬───────────┐ │ col1 ┆ col2 ┆ col1_year ┆ col2_year │ │ --- ┆ --- ┆ --- ┆ --- │ │ datetime[μs] ┆ datetime[μs] ┆ i32 ┆ i32 │ ╞═════════════════════╪═════════════════════╪═══════════╪═══════════╡ │ 2020-01-01 00:00:00 ┆ 2020-01-01 00:00:00 ┆ 2020 ┆ 2020 │ │ 2020-02-01 00:00:00 ┆ 2020-02-01 00:00:00 ┆ 2020 ┆ 2020 │ └─────────────────────┴─────────────────────┴───────────┴───────────┘
Question: I am new to polars, but have worked quite a bit with pandas and numpy. Let's say I have a dataframe df like:    Column 1 Column 2 Column 3 Column 4 Column 5 Column 6     A 1 W B 2 X   C 3 Y D 4 Z    I would like to do something like df.reshape((-1,3)) (in numpy) in order to obtain:    Column 1 Column 2 Column 3     A 1 W   B 2 X   C 3 Y   D 4 Z    however I haven't been able to find a similar function in polars. df.melt() doesn't work in this case of course and doing df = pl.DataFrame(df.to_numpy().reshape((-1,3))) doesn't seem like an optimal solution. Would appreciate any help! Looked through the documentation to no avail.

Answer: I don't think polars has a specialised reshape function for the scenario outlined above. However, you could combine pl.concat to concatenate dataframe fragments created using pl.DataFrame.select. pl.concat([ df.select("Column 1", "Column 2", "Column 3"), df.select( pl.col("Column 4").alias("Column 1"), pl.col("Column 5").alias("Column 2"), pl.col("Column 6").alias("Column 3"), ), ])  Output. shape: (4, 3) ┌──────────┬──────────┬──────────┐ │ Column 1 ┆ Column 2 ┆ Column 3 │ │ --- ┆ --- ┆ --- │ │ str ┆ i64 ┆ str │ ╞══════════╪══════════╪══════════╡ │ A ┆ 1 ┆ W │ │ C ┆ 3 ┆ Y │ │ B ┆ 2 ┆ X │ │ D ┆ 4 ┆ Z │ └──────────┴──────────┴──────────┘
Question: I am trying to assess in a sales database whether the # of advertisements has changed. The example dataframe I am using is as such: df = pd.DataFrame({"offer-id": [1,1,2,2,3,4,5], "date": ["2024-02-10","2024-02-11","2024-02-10","2024-02-11","2024-02-11","2024-02-11","2024-02-10"], "price": [30,10,30,30,20,25,20]})  And looks like the below:  I am now trying to get the # of items that were sold or newly added (I don't care which one, since once I have one the other should be failry easily computable). E.g. in a perfect case the next piece of code tells me that on 10th of February 3 offers were online (ID 1, 2, and 5) and one was sold (ID 5) Or alternatively, it tells me on 11th of February 4 offers are online, and 2 of them are new (from that, since I know the day before 5 were online I can also calculate that one must have sold) Is there a simple way of doing this? I have tried things like df.groupby(['date'])["offer-id"].agg({'nunique'})  but they are missing the "comparison to previous" timestep component.

Answer: You could aggregate as a set: offers = df.groupby('date', sort=True)['offer-id'].agg(set) date 2024-02-10 {1, 2, 5} 2024-02-11 {1, 2, 3, 4} Name: offer-id, dtype: object  Then getting the diff will give you the new items: offers.diff() date 2024-02-10 NaN 2024-02-11 {3, 4} Name: offer-id, dtype: object  Or the sold items: offers.diff(-1) date 2024-02-10 {5} 2024-02-11 NaN Name: offer-id, dtype: object  If you want the number of items, chain str.len: offers.diff().str.len().fillna(0).convert_dtypes() date 2024-02-10 0 2024-02-11 2 Name: offer-id, dtype: Int64  And to get those as new columns, map: df['new'] = df['date'].map(offers.diff().str.len().fillna(0).convert_dtypes()) df['sold'] = df['date'].map(offers.diff(-1).str.len().fillna(0).convert_dtypes()) print(df)  Output:  offer-id date price new sold 0 1 2024-02-10 30 0 1 1 1 2024-02-11 10 2 0 2 2 2024-02-10 30 0 1 3 2 2024-02-11 30 2 0 4 3 2024-02-11 20 2 0 5 4 2024-02-11 25 2 0 6 5 2024-02-10 20 0 1
Question: I have this code: import pandas as pd df = pd.DataFrame({'consumption': [10.51, 103.11, 55.48], 'co2_emissions': [37.2, 19.66, 1712]}, index=['Pork', 'Wheat Products', 'Beef']) df['Max'] = df.idxmax(axis=1, skipna=True, numeric_only=True) df  I need to find the n largest values. Here there is a technique using apply/lambda. But it returns error. df.apply(lambda s: s.abs().nlargest(2).index.tolist(), axis=1,skipna=True, numeric_only=True)   TypeError: () got an unexpected keyword argument 'numeric_only'  Is there any way to obtain top N results using idxmax? Is there any way to overcome this error got when using apply lambda method?

Answer: Your error is due to passing the skipna and numeric_only parameters to apply. You can fix it with: (df.select_dtypes('number') .apply(lambda s: s.dropna().abs().nlargest(2) .index.tolist(), axis=1) )  Output: Pork [co2_emissions, consumption] Wheat Products [consumption, co2_emissions] Beef [co2_emissions, consumption] dtype: object  A more efficient approach using numpy N = 2 tmp = df.select_dtypes('number') out = pd.Series( np.take_along_axis( tmp.columns.to_numpy()[:, None], np.argpartition(tmp, -N)[:, -N:], axis=0 )[:, ::-1].tolist(), index=df.index, )
Question: I have 107 images and I want to extract text from them, and I am using Gemini API, and this is my code till now: # Gemini Model model = genai.GenerativeModel('gemini-pro-vision', safety_settings=safety_settings) # Code images_to_process = [os.path.join(image_dir, image_name) for image_name in os.listdir(image_dir)] # list of 107 images prompt = """Carefully scan this images: if it has text, extract all the text and return the text from it. If the image does not have text return '<000>'.""" for image_path in tqdm(images_to_process): img = Image.open(image_path) output = model.generate_content([prompt, img]) text = output.text print(text)  In this code, I am just taking one image at a time and extracting text from it using Gemini. Problem - I have 107 images and this code is taking ~10 minutes to run. I know that Gemini API can handle 60 requests per minute. How to send 60 images at the same time? How to do it in batch?

Answer: 2024-10 update: I've added a Cookbook Quickstart on asynchronous requests to show how this works. The advice below is still correct.  In synchronous Python you can use something like a ThreadPoolExecutor to make your requests in separate threads. The Gemini Python SDK has an async API though, which can be a bit more natural: $ python -m asyncio >>> import asyncio >>> import google.generativeai as genai >>> import PIL >>> model = genai.GenerativeModel('gemini-pro-vision') >>> imgs = ['/path/img.jpg', ...] >>> prompt = "..." >>> async def process_image(img: str) -> str: ... r = await model.generate_content_async([prompt, PIL.Image.open(img)]) ... # TODO: error handling ... return r.text >>> jobs = asyncio.gather(*[process_image(img) for img in imgs]) >>> results = await jobs # or run_until_complete(jobs) >>> results ['text is here', ...]  This uses the implicit asyncio REPL event loop, in a real app you'll need to set up and use your own event loop. See also TaskGroups.
Question: The docs for cancel make it sound like you should usually propagate CancelledError exceptions:  Therefore, unlike Future.cancel(), Task.cancel() does not guarantee that the Task will be cancelled, although suppressing cancellation completely is not common and is actively discouraged. Should the coroutine nevertheless decide to suppress the cancellation, it needs to call Task.uncancel() in addition to catching the exception.  However, neither of the methods for detecting cancellation are awaitable: cancelling which tells you if cancelling is in progress, and cancelled tells you if cancellation is done. So the obvious way to wait for cancellation is this: foo_task.cancel() try: await foo_task except asyncio.CancelledError: pass  There are lots of examples of this online even on SO. But the docs warn you asyncio machinery will "misbehave" if you do this:  The asyncio components that enable structured concurrency, like asyncio.TaskGroup and asyncio.timeout(), are implemented using cancellation internally and might misbehave if a coroutine swallows asyncio.CancelledError  Now you might be wondering why you would wait to block until a task is fully cancelled. The problem is the asyncio event loop only creates weak references to tasks, so if as your class is shutting down (e.g. due to a cleanup method or __aexit__) and you don't await every task you spawn, you might tear down the only strong reference while the task is still running, and then python will yell at you: ERROR base_events.py:1771: Task was destroyed but it is pending!  So it seems to avoid the error I am specifically being forced into doing the thing I'm not supposed to do :P The only alternative seems to be weird unpythonic hackery like stuffing every task I make in a global set and awaiting them all at the end of the run.

Answer: You have a cleanup problem. The with statement is generally used to solve cleanup. Use asyncio.TaskGroup: async with asyncio.TaskGroup() as tg: tg.create_task(some_coro(...)).cancel()
Question: Context I am running a snakemake (v.7.32.4) pipeline using slurm task manager. I have set resources (time and memory for each rule) dynamically based on file size and # of tries, like this: rule index: resources: mem_mb = lambda wildcards, input, attempt: ( 200 * attempt ), runtime = lambda wildcards, input, attempt: ( "{minutes}min".format( minutes=max( int((input.size_mb / 5000) * attempt), 1) ) )  I have 2 related questions (should I split the post?): 1) Is it possible to set resources dinamically outside of snakefile? I tried to set that on the profile config file but didn't success (sometime ago, so cannot say exactly what I tried) 2) Having set resources dinamically inside Snakefile, how do I do a dry run or a rulegraph? If I run a dry-run I get the following error: WorkflowError: Cannot parse runtime value into minutes for setting runtime resource: <TBD>  This seems logical to me since the file doesn't exist yet. Nevertheless, I would like to know the specifics of all steps (except resources of course) to be run before actually running them, is this possible? Something similar happens if I try to do a rulegraph: snakemake --profile Config/Profiles/slurm -np --rulegraph | dot -Tsvg > rulegraph.svg  In this case, I get an empty file (probably because of the error in the dry run?).

Answer: 1) Is it possible to set resources dinamically outside of snakefile? Depends on the snakemake version. For v8.14.0, yes. For version 7.32.4, sort of. See both scenarios below. Snakemake version 7.34.4 You can call a function on snakefile and declare the function elsewhere, as proposed by @SultanOrazbayev: Snakefile: from resources import get_runtime rule index: resources: mem_mb = get_mem_mb  resources.py: def get_runtime(wildcards: dict, input: str|list[str], attempt: int) -> str: try: minutes = max(int((input.size_mb / 5_000) * attempt), 1) except FileNotFoundError: minutes = 10 # this is some test value return "{minutes}min"  Snakemake version 8.14.0 Install the slurm plug in: pip install snakemake-executor-plugin-slurm  Then you can specify resources dinamically (entirely) on your workflow profile config.yaml file: executor: slurm set-resources: index: runtime: f"{max(int((input.size_mb / 5_000) * attempt), 1)}min"  NOTE: I only tried snakemake v8 using the slurm plugin. And I found this solution on the slurm plugin documentation. Hence, I don't know if using the workflow profile as described above would work without the slurm plugin or not. 2) Having set resources dynamically inside Snakefile, how do I do a dry run or a rulegraph? This question only applies to snakemake version 7 or lower. On snakemake v8 (I specifically tried v8.14.0) dry runs and graphs works fine, even if file does not exist yet. As for snakemake v7.32.4, one way of solving it is by handling the error as described above (proposed by @SultanOrazbayev).
Question: i'm trying to validate input email from request body and i have build custom field validator at model. here is code of RegisterModel: from pydantic import BaseModel, EmailStr, Field, field_validator from repository.repository_user import UserRepository from pprint import pprint class RegisterModel(BaseModel): name:str email:EmailStr password:str = Field(..., min_length=7) @field_validator('email') @classmethod def email_must_unique(cls, v): repo = UserRepository() result = repo.find({'email': v}) pprint(result) return result  the email_must_unique method will create new instace of UserRepository class and it will call the find method from UserRepository to find specific user based on email. actualy the email_must_unique method not finished yet to validate the email, it just get specific users, but i already facing an error. here is code of UserRepository: from pymongo.database import Database from fastapi import Depends from config.db import db_conn class UserRepository: def __init__(self, db: Database = Depends(db_conn)): self.repository = db.users # users is mongo collection def find(self, filter: dict): result = self.repository.find_one(filter) return result  with this code i'm facing error like this:  self.repository = db.users ^^^^^^^^ AttributeError: 'Depends' object has no attribute 'users'  i have no idea what What caused it. can you give me the solution to solve this error or maybe alternative way to validate email uniqueness ?

Answer: As mentioned in the comments by @Chiheb Nexus, Do not call the Depends function directly. Instead, use the Annotated dependency which is more graceful. For example, the config/db.py file should look like this: # config/db.py from typing import Annotated from fastapi import Depends from pymongo import MongoClient from pymongo.database import Database def get_db() -> Database: client = MongoClient("mongodb://localhost:27017/") return client.some_database DbConn = Annotated[Database, Depends(get_db)] # <-- this is the annotated depend  You can then use it like this: from config.db import DbConn class UserRepository: def __init__(self, db: DbConn): # <- FastAPI will call `DbConn` self.repository = db.users def find(self, filter: dict): result = self.repository.find_one(filter) return result  In addition, db in the __init__ parameter is annotated via DbConn so IDE knows its type. Validating the uniqueness of data is a task best left to the database rather than Pydantic. In other words, you should insert data and handle unique errors, DuplicateKeyError in pymongo.
Question: I want an if-statement to check if x is an empty tuple. Is there any significant advantage of writing the if-statement as if isinstance(x, tuple) and not x:  vs if x == ():  given the latter is shorter to write and simpler to read?

Answer: More often than not, you rely on object interfaces rather than specific types, so just if not x is sufficient and this is the usual Pythonic expression (see comments by @СергейКох and @MarkRansom). The advantage of the first case is that tuple subclasses will be accepted, which is usually what is expected in a duck-typed language. The second one is checking whether x is comparable to True against an empty tuple, which could be true for any class that implements a valid __eq__ method. If neither of these is acceptable, then you need to first compare the strict type with type and then check if it's empty. Here is an example of the differences: class Foo(tuple): pass class Bar: def __eq__(self, other): return len(other) == 0: things = {'tuple': tuple(), 'Foo': Foo(), 'Bar': Bar() } for name, thing in things.items(): if thing == (): print(f"{name} is an empty object with __eq__ implementation against empty tuple") if isinstance(thing, tuple) and not thing: print(f"{name} is an empty tuple, or tuple subclass") if type(thing) == type(()) and thing == (): print(f"{name} is an empty tuple")  Output: tuple is an empty object with __eq__ implementation against empty tuple tuple is an empty tuple, or tuple subclass tuple is an empty tuple Foo is an empty object with __eq__ implementation against empty tuple Foo is an empty tuple, or tuple subclass Bar is an empty object with __eq__ implementation against empty tuple  So it all depends on how strict or loose you want to be with your comparison.
Question: I am exploring tree plots with the following library and code: from dsplot.graph import Graph graph = Graph( {0: [1, 4, 5], 1: [3, 4], 2: [1], 3: [2, 4], 4: [], 5: []}, directed=True ) graph.plot()  When I run the code, I get the following error: from dsplot.graph import Graph ModuleNotFoundError: No module named 'dsplot'  I have tried pip installing the dsplot library with this command on the PyCharm terminal: pip install dsplot  But this then causes another error:  Collecting dsplot Using cached dsplot-0.9.0-py3-none-any.whl (8.8 kB) Collecting pygraphviz<2.0,>=1.7 Using cached pygraphviz-1.12.tar.gz (104 kB) ERROR: Error [WinError 3] The system cannot find the path specified while executing command pip subprocess to install build dependencies Installing build dependencies ... error ERROR: Could not install packages due to an OSError: [WinError 3] The system cannot find the path specified  What is the reason for this problem installing the library and how can I resolve it?

Answer: It looks like you have not installed pygraphviz. Install pygraphviz Try downloading it from their official website and make sure its added to your path by clicking "Add Graphviz to the System PATH for the current user". In your terminal, run dot -V and if it outputs your version of pygraphviz, you should be all set to run pip install dsplot. If dot -V does not return the Version: Make sure you checked this box during installation:  If you checked the box and it is still not in your PATH:  Find where Graphviz is installed; it is likely in "C:\Program Files\Graphviz" but is dependent on installation and OS. Click on this directory, and right click on "bin" to copy the path. Open Start Search, type "env", and select "Edit the system environment variables". Click "Environment Variables" in System Properties Find "Path Variables" under "System Variables" and select "Edit" Click "New" Paste in the path of Graphviz, which is likely something like C:\Program Files\Graphviz\bin Click "Ok" on all open windows to apply the changes  After adding Graphviz to your path, open a terminal and once again try dot -V. If that returns the version, then you should be able to run pip install dsplot.
Question: i have calculated the scores of all the columns in my DF,which has 312 columns and 650 rows, using PCA with following code: all_pca=PCA(random_state=4) all_pca.fit(tt) all_pca2=all_pca.transform(tt) plt.plot(np.cumsum(all_pca.explained_variance_ratio_) * 100) plt.xlabel('Number of components') plt.grid(which='both', linestyle='--', linewidth=0.5) plt.xticks(np.arange(0, 330, step=25)) plt.yticks(np.arange(0, 110, step=10)) plt.ylabel('Explained variance (%)') plt.savefig('elbow_plot.png', dpi=1000)  and the result is the following image.  My main goal is to use only important features for Random forest regression, Gradient boosting, OLS regression and LASSO. As you can see, 100 columns describe 95.2% of the variance in my Dataframe. Can I use this threshold (100 Columns) for backward feature selection?

Answer:  As you can see, 100 columns describe 95.2% of the variance in my Dataframe.  The graph tells you that 100 PCA components capture 95% of the variance. These 100 components do not correspond to 100 individual features. Each PCA component is made by combining all features together, which gives you that one component. When 100 PCA components capture 95% of the variance, it means that your original 312 columns can be linearly combined into fewer (100) new columns, and you only lose 5% of the information in the process. It's a measure of the intrinsic dimensionality of the feature set.  Can I use this threshold (100 Columns) for backward feature selection?  The 100 PCA components that explain 95% don't really tell you which individual features (or how many of them) are important, as each PCA component is a mix of all features. Also, the 95% refers to variability of the features - it doesn't mean the 100 PCA components will be useful for the target. Perhaps you could use the 100 components to guide your choice between using forward vs. backward feature selection. In this case, the intrinsic dimensionality of the dataset is closer to 100 than it is to 312, so I'd opt for forward selection as it seems like the number of useful features might be less than the original size. If you were to run PCA before feature selection, it would create new features (PCA components) out of the original ones, and in the process you could lose interpretability, as the new features can be messy linear combinations of the original ones. One method of identifying useful features is to run forward (or backward) selection on a random forest using the original features, and stop when you hit a score threshold like 95% validation accuracy. Then you can use those selected features for other models. Feature selection is relatively time-consuming as it requires lots of repeated model fitting. Permutation importance is another way of identifying key features.