eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Say you have a Python program that you can successfully package using only pyproject.toml. What are the downsides? Why use setup.py or setup.cfg in this case?

Answer: There is no downside in not having a setup.py. It is just that in some particular cases some elements of the packaging can not be expressed in a descriptive manner (which means without code) in setup.cfg or pyproject.toml. This can range from some custom dynamic package metadata to the handling of packaging for custom non-Python code and many other things. My recommendation is:  Avoid using setup.py as much as possible, if you can do without this file completely, then do without it  Place the standard parts in pyproject.toml:  [build-system] section, it is strongly recommended to have this section independently of whether or not you have a setup.cfg file and/or setup.py script  [project] section, and avoid dynamic fields as much as possible    For the parts that are specific to setuptools, you can choose to have them in pyproject.toml (under the [tool.setuptools] section) or in setup.cfg, as you prefer   Related:  Is setup.py deprecated? How to modernize a setup.py based project?
Question: I have a dataframe as follows: df = pl.DataFrame( {"a": [([1, 2, 3], [2, 3, 4], [6, 7, 8]), ([1, 2, 3], [3, 4, 5], [5, 7, 9])]} )  Basically, each cell of a is a tuple of three arrays of the same length. I want to fully split them to separate columns (one scalar resides in one column) like the shape below: shape: (2, 9) ┌─────────┬─────────┬─────────┬─────────┬─────┬─────────┬─────────┬─────────┬─────────┐ │ field_0 ┆ field_1 ┆ field_2 ┆ field_3 ┆ ... ┆ field_5 ┆ field_6 ┆ field_7 ┆ field_8 │ │ --- ┆ --- ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ i64 ┆ i64 ┆ ┆ i64 ┆ i64 ┆ i64 ┆ i64 │ ╞═════════╪═════════╪═════════╪═════════╪═════╪═════════╪═════════╪═════════╪═════════╡ │ 1 ┆ 2 ┆ 3 ┆ 2 ┆ ... ┆ 4 ┆ 6 ┆ 7 ┆ 8 │ │ 1 ┆ 2 ┆ 3 ┆ 3 ┆ ... ┆ 5 ┆ 5 ┆ 7 ┆ 9 │ └─────────┴─────────┴─────────┴─────────┴─────┴─────────┴─────────┴─────────┴─────────┘  One way I have tried is to use list.to_struct and unnest two times to fully flatten the two nested levels. Two levels is fine here, but if there are a variety of nested levels and the number could not be determined ahead, the code will be so long. Is there any simpler (or more systematic) way to achieve this?

Answer: Piggybacking on some of the concepts in the other answers... We can see how much nesting there is by checking the inner datatype of the pl.List datatype for the column. This can be the condition for a loop. Here's an example with one further nesting of lists than the initial question: df = pl.DataFrame({"a": [([[1, 2], [2, 4], [3, 6]], [[2, 4], [3, 6], [4, 8]], [[6, 12], [7, 14], [8, 16]]), ([[1, 2], [2, 4], [3, 6]], [[3, 6], [4, 8], [5, 10]], [[5, 10], [7, 14], [9, 18]])]}) shape: (2, 1) ┌─────────────────────────────────┐ │ a │ │ --- │ │ list[list[list[i64]]] │ ╞═════════════════════════════════╡ │ [[[1, 2], [2, 4], [3, 6]], [[2… │ │ [[[1, 2], [2, 4], [3, 6]], [[3… │ └─────────────────────────────────┘  expr = pl.col('a') dtype = df.schema['a'] # alternative: dtype.base_type() == pl.List while isinstance(dtype.inner, pl.List): expr = expr.list.eval(pl.element().list.explode()) dtype = dtype.inner df.lazy().select(expr.list.to_struct()).collect().unnest('a')  shape: (2, 18) ┌─────────┬─────────┬─────────┬─────────┬───┬──────────┬──────────┬──────────┬──────────┐ │ field_0 ┆ field_1 ┆ field_2 ┆ field_3 ┆ … ┆ field_14 ┆ field_15 ┆ field_16 ┆ field_17 │ │ --- ┆ --- ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ i64 ┆ i64 ┆ ┆ i64 ┆ i64 ┆ i64 ┆ i64 │ ╞═════════╪═════════╪═════════╪═════════╪═══╪══════════╪══════════╪══════════╪══════════╡ │ 1 ┆ 2 ┆ 2 ┆ 4 ┆ … ┆ 7 ┆ 14 ┆ 8 ┆ 16 │ │ 1 ┆ 2 ┆ 2 ┆ 4 ┆ … ┆ 7 ┆ 14 ┆ 9 ┆ 18 │ └─────────┴─────────┴─────────┴─────────┴───┴──────────┴──────────┴──────────┴──────────┘
Question: I came across some weird behavior by math.cos() (Python 3.11.0): >>> import math >>> math.cos(math.pi) # expected to get -1 -1.0 >>> math.cos(math.pi/2) # expected to get 0 6.123233995736766e-17  I suspect that floating point math might play a role in this, but I'm not sure how. And if it did, I'd assume Python just checks if the parameter equaled math.pi/2 to begin with. I found this answer by Jon Skeet, who said:  Basically, you shouldn't expect binary floating point operations to be exactly right when your inputs can't be expressed as exact binary values - which pi/2 can't, given that it's irrational.  But if this is true, then math.cos(math.pi) shouldn't work either, because it also uses the math.pi approximation. My question is: why does this issue only show up when math.pi/2 is used?

Answer: Any error in math.pi vs. π (there always is some) makes very little difference in one case math.cos(math.pi) and is quite significant in math.cos(math.pi/2).  The curve is flat When math.cos(x) is very near 1.0, the curve is very flat: the slope is "close" to zero. About 47 million floating point x values near π have a cos(x) mathematically more than -1.0, yet their value is closer to -1.0 than the next encodable value of -0.99999999999999988897... The curve's slope is 1 With x near π and math.cos(x/2) near 0.0, the cosine curve has a |slope| "close" to one. Both the next smaller and next larger encodable x have a different cos(x/2). Conclusion When the |result| of sin(x) or cos(x) is near 1.0, many nearby x values will report 1.0. This would be true even if some x value was incredible close to π. For x near π (like math.pi) and y = |cos(x)|, we need about twice the precision in y to see an imprecision in x.
Question: In this example, on columns ["foo", "ham"], I want rows 1 and 4 to be removed since they match a pair in the list df = pl.DataFrame( { "foo": [1, 1, 2, 2, 3, 3, 4], "bar": [6, 7, 8, 9, 10, 11, 12], "ham": ["a", "b", "c", "d", "e", "f", "b"] } ) pairs = [(1,"b"),(3,"e"),(4,"g")]  The following worked for me but I think this will be problematic when the dataframe and list of pairs are large. for a, b in pairs: df = df.filter(~(pl.col('foo') == a) | ~(pl.col('ham') == b))  I think this is the pandas implementation for this problem Pandas: How to remove rows from a dataframe based on a list of tuples representing values in TWO columns? I am not sure what the Polars implementation of it is. (I think this problem can be generalized to any number of selected columns and any number of elements in a group. For instance, rather than a list of pairs, it can be another dataframe. You get the 'set difference', in terms of rows, of the two dataframes based on specific columns.)

Answer: It looks like an ANTI JOIN schema = ["foo", "ham"] (df.with_row_index() # just to show what "row numbers" were "removed" .join( pl.DataFrame(pairs, orient="row", schema=schema), on = schema, how = "anti" ) )  shape: (5, 4) ┌───────┬─────┬─────┬─────┐ │ index ┆ foo ┆ bar ┆ ham │ │ --- ┆ --- ┆ --- ┆ --- │ │ u32 ┆ i64 ┆ i64 ┆ str │ ╞═══════╪═════╪═════╪═════╡ │ 0 ┆ 1 ┆ 6 ┆ a │ │ 2 ┆ 2 ┆ 8 ┆ c │ │ 3 ┆ 2 ┆ 9 ┆ d │ │ 5 ┆ 3 ┆ 11 ┆ f │ │ 6 ┆ 4 ┆ 12 ┆ b │ └───────┴─────┴─────┴─────┘
Question: Is it possible to add a progress bar to a Polars apply loop with a custom function? For example, how would I add a progress bar to the following toy example: df = pl.DataFrame( { "team": ["A", "A", "A", "B", "B", "C"], "conference": ["East", "East", "East", "West", "West", "East"], "points": [11, 8, 10, 6, 6, 5], "rebounds": [7, 7, 6, 9, 12, 8] } ) df.group_by("team").map_groups(lambda x: x.select(pl.col("points").mean()))  Edit 1: After help from @Jcurious, I have the following 'tools' that can be re-used for other functions, however it does not print to console correctly. def pl_progress_applier(func, task_id, progress, **kwargs): progress.update(task_id, advance=1, refresh=True) return func(**kwargs) def pl_groupby_progress_apply(data, group_by, func, drop_cols=[], **kwargs): global progress with Progress() as progress: num_groups = len(data.select(group_by).unique()) task_id = progress.add_task('Applying', total=num_groups) return ( data .group_by(group_by) .map_groups(lambda x: pl_progress_applier( x=x.drop(drop_cols), func=func, task_id=task_id, progress=progress, **kwargs) ) ) # and using the function custom_func, we can return a table, howevef the progress bar jumps to 100% def custom_func(x): return x.select(pl.col('points').mean()) pl_groupby_progress_apply( data=df, group_by='team', func=custom_func )  Any ideas on how to get the progress bar to actually work? Edit 2: It seems like the above functions do indeed work, however if you're using PyCharm (like me), then it does not work. Enjoy non-PyCharm users!

Answer: I like the progress bars from Rich (which also comes bundled with pip) There's probably a neater way to package this up, but something like: from pip._vendor.rich.progress import ( Progress, SpinnerColumn, TimeElapsedColumn ) def polars_bar(total, title="Processing", transient=True): bar = Progress( SpinnerColumn(), *Progress.get_default_columns(), TimeElapsedColumn(), transient=transient # remove bar when finished ) def _run(func, *args, **kwargs): task_id = bar.add_task(title, total=total) def _execute(*args, **kwargs): bar.update(task_id, advance=1) return func(*args, **kwargs) return lambda self: _execute(self, *args, **kwargs) bar.run = _run return bar  Examples .map_groups() def my_custom_group_udf(group, expr): time.sleep(.7) return group.select(expr) num_groups = df["team"].n_unique() with polars_bar(total=num_groups) as bar: (df.group_by("team") .map_groups( bar.run( my_custom_group_udf, expr=pl.col("points").mean().name.suffix("_mean") ) ) )   .map_elements() def my_custom_udf(points, multiplier=1): time.sleep(.3) # simulate some work return (points + 100) * multiplier with polars_bar(total=df.height) as bar: df.with_columns( pl.col("points").map_elements( bar.run(my_custom_udf, multiplier=5), return_dtype = pl.Int64 ) .alias("udf") )   Note: tqdm also has Rich support: https://tqdm.github.io/docs/rich/
Question: I was wondering is there is a good and comprehensive way to debug python code step by step so that I can have a better idea of all the variables involved, their dimensions and values? What can be done to do step-by-step debugging?

Answer: JupyterLab is the current generation of Jupyter interface and has a debugger mode that allows you to step through code. See this post here and a related one here. Documentation is here. You can try it out without installing anything on your own system by clicking here to launch a temporary Jupyter session running on a remote computer via the MyBinder service, and then open a new notebook file from the launcher. You can step the sections 'Debug code in notebook' and 'Explore the code state' there in the documentation.  For %%debug, which is from IPython that Jupyter inherits much from, see here and other posts in that thread.
Question: My GET endpoint receives a query parameter that needs to meet the following criteria:  be an int between 0 and 10 be even number  1. is straight forward using Query(gt=0, lt=10). However, it is not quiet clear to me how to extend Query to do extra custom validation such as 2.. The documentation ultimately leads to pydantic. But, my application runs into internal server error when the second validation 2. fails. Below is a minimal scoped example from fastapi import FastAPI, Depends, Query from pydantic import BaseModel, ValidationError, validator app = FastAPI() class CommonParams(BaseModel): n: int = Query(default=..., gt=0, lt=10) @validator('n') def validate(cls, v): if v%2 != 0: raise ValueError("Number is not even :( ") return v @app.get("/") async def root(common: CommonParams = Depends()): return {"n": common.n}  Below are requests that work as expected and ones that break: # requsts that work as expected localhost:8000?n=-4 localhost:8000?n=-3 localhost:8000?n=2 localhost:8000?n=8 localhost:8000?n=99 # request that break server localhost:8000?n=1 localhost:8000?n=3 localhost:8000?n=5 

Answer: Option 1 Raise HTTPException directly instead of ValueError, as demonstrated in Option 1 of this answer. Example: from fastapi import FastAPI, Depends, Query, HTTPException from pydantic import BaseModel, validator app = FastAPI() class CommonParams(BaseModel): n: int = Query(default=..., gt=0, lt=10) @validator('n') def prevent_odd_numbers(cls, v): if v % 2 != 0: raise HTTPException(status_code=422, detail='Input number is not even') return v @app.get('/') async def root(common: CommonParams = Depends()): return {'n': common.n}  Server Response (when the input number is not even, e.g., n = 1): # 422 Error: Unprocessable Entity { "detail": "Input number is not even" }  Option 2 Use a custom exception handler, in order to handle ValueError exceptions, similar to this answer and this answer. Example: from fastapi import FastAPI, Request, Depends, Query, status from fastapi.responses import JSONResponse from fastapi.encoders import jsonable_encoder from pydantic import BaseModel, validator app = FastAPI() @app.exception_handler(ValueError) async def validation_exception_handler(request: Request, exc: ValueError): return JSONResponse( status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, content=jsonable_encoder({"detail": exc.errors(), # optionally, include the pydantic errors "custom msg": {"Your error message"}}), # optionally, return a custom message ) class CommonParams(BaseModel): n: int = Query(default=..., gt=0, lt=10) @validator('n') def prevent_odd_numbers(cls, v): if v % 2 != 0: raise ValueError('Input number is not even') return v @app.get('/') async def root(common: CommonParams = Depends()): return {'n': common.n}  Server Response (when the input number is not even, e.g., n = 1): # 422 Error: Unprocessable Entity { "detail": [ { "loc": [ "n" ], "msg": "Input number is not even", "type": "value_error" } ], "custom msg": [ "Your error message" ] }  Update Please note that in Pydantic V2, @validator has been deprecated and was replaced by @field_validator. Please have a look at this answer for more details and examples.
Question: I need to drop rows that have a nan value in any column. As for null values with drop_nulls() df.drop_nulls()  but for nans. I have found that the method drop_nans exist for Series but not for DataFrames df['A'].drop_nans()  Pandas code that I'm using: df = pd.DataFrame( { 'A': [0, 0, 0, 1,None, 1], 'B': [1, 2, 2, 1,1, np.nan] } ) df.dropna() 

Answer: Another definition is: to keep rows where all values are not NaN For that, we can use:  .is_not_nan() to test for "not nan" pl.col(pl.Float32, pl.Float64) to select only float columns .all_horizontal() to compute a row-wise True/False comparison DataFrame.filter to keep only the "True" rows  df = pl.from_repr(""" ┌─────┬─────┬─────┐ │ A ┆ B ┆ C │ │ --- ┆ --- ┆ --- │ │ f64 ┆ f64 ┆ str │ ╞═════╪═════╪═════╡ │ 0.0 ┆ 1.0 ┆ a │ │ 0.0 ┆ 2.0 ┆ b │ │ 0.0 ┆ 2.0 ┆ c │ │ 1.0 ┆ 1.0 ┆ d │ │ NaN ┆ 1.0 ┆ e │ │ 1.0 ┆ NaN ┆ g │ └─────┴─────┴─────┘ """)  df.filter( pl.all_horizontal(pl.col(pl.Float32, pl.Float64).is_not_nan()) )  shape: (4, 3) ┌─────┬─────┬─────┐ │ A ┆ B ┆ C │ │ --- ┆ --- ┆ --- │ │ f64 ┆ f64 ┆ str │ ╞═════╪═════╪═════╡ │ 0.0 ┆ 1.0 ┆ a │ │ 0.0 ┆ 2.0 ┆ b │ │ 0.0 ┆ 2.0 ┆ c │ │ 1.0 ┆ 1.0 ┆ d │ └─────┴─────┴─────┘  polars.selectors has also since been added which provides cs.float() df.filter( pl.all_horizontal(cs.float().is_not_nan()) )
Question: I have a Polars dataframe that I want to write to an external database (SQLite). How can I do it? In Pandas you have to_sql() but I couldn't find any equivalent in Polars.

Answer: You can use the DataFrame.write_database method.
Question: I downloaded the mambaforge install file for Windows, ran it, and it successfully completed. Mamba has a quickstart guide with CLI commands here: https://mamba.readthedocs.io/en/latest/user_guide/mamba.html But I can't figure out WHERE to enter the commands (e.g. mamba create -n envname). Is there supposed to be a Start Menu shortcut for Mambaforge or something similar? I checked the option to create Start Menu shortcuts during the install, but I don't see any even though the install completed without errors. I tried running mamba create -n envname from the cmd prompt and it returns: 'mamba' is not recognized as an internal or external command, operable program or batch file.  Clearly I missed a step somewhere but I can't for the life of me figure out what. What I tried already: I tried running the Mambaforge-Windows-x86_64.exe and checked the option to create Start Menu shortcuts. The install completed successfully. I found the mambaforge directory (which has a _conda.exe and python.exe among others) I was expecting it to create a "mambaforge" start menu shortcut. As far as I can tell no start menu shortcuts were created. The Windows installation instructions are literally a single line: "Download the installer and double click it on the file browser." https://github.com/conda-forge/miniforge

Answer:  I was expecting it to create a "mambaforge" start menu shortcut. As far as I can tell no start menu shortcuts were created.  The start menu shortcut is called "miniforge prompt" and behaves as you expect from Anaconda prompt. Mambaforge is based on miniforge, and the prompt stems from that.
Question: from instapy import InstaPy session = InstaPy(username='name', password='password') session.login()  (I use VSC) my code breaks at the first line with error: from clarifai.rest import ClarifaiApp, Workflow ModuleNotFoundError: No module named 'clarifai.rest' I tried reinstalling instapy, that specific module, but nothing changed. Edit: Yes, I have tried reinstalling clarifai and it didn't help.

Answer: The solution is here I tested Franchen Bao's comment and it worked for me. I had this error after the emoji error. Looks like this project is a little unstable and can only handle certain versions of each package, like Franchen Bao said. pip uninstall clarifai pip install clarifai==2.6.2
Question: I have asyncio sqlalchemy code: import asyncio from sqlalchemy.orm import sessionmaker, declarative_base from sqlalchemy import text, Column, Integer, String from sqlalchemy.ext.asyncio import create_async_engine, AsyncSession async_engine = create_async_engine(f'mysql+aiomysql://root:example@127.0.0.1:3306/spy-bot') AsyncSession = sessionmaker(async_engine, class_=AsyncSession, expire_on_commit=False) Base = declarative_base() class User(Base): __tablename__ = 'users' id = Column(Integer, primary_key=True) username = Column(String(255)) async def main(): async with AsyncSession() as session: stmt = text('SELECT * FROM `users` LIMIT 1') result = await session.execute(stmt) user = result.one() print(type(user), user) asyncio.run(main())  how do I make session query return instances on User class while still using raw sql? On sync version this would look like result = session.query(User).from_statement(text(query)) 

Answer: You would do it in the same way, albeit using 2.0-style syntax as the legacy Query class is not supported in asyncio: result = await session.scalars(select(User).from_statement(text_object))  The docs at Getting ORM Results from Textual Statements apply. The complete function might look like this: import sqlalchemy as sa ... async def main(): async with AsyncSession() as session: stmt = text('SELECT * FROM `users` LIMIT 1') result = await session.scalars(sa.select(User).from_statement(stmt)) user = result.one() print(type(user), user) await async_engine.dispose()
Question: I'm currently learning to use the librosa library and when trying to display a colorbar on an associated spectrogram, I get an inexplicable error. I'm not that familiar with matplotlib I've searched everywhere for a solution and I can't help but feel I'm missing something here. The following code: n_fft = 2048 #number of samples (window of performing an FFT) hop_length = 512 #amount shifting each fourier transform to the right stft = librosa.core.stft(signal, hop_length=hop_length, n_fft=n_fft) spectogram = np.abs(stft) log_spectogram = librosa.amplitude_to_db(spectogram) librosa.display.specshow(log_spectogram, sr=sr, hop_length=hop_length) plt.xlabel("Time") plt.ylabel("Frequency") plt.colorbar() plt.show()  causes an AttributeError: AttributeError: module 'matplotlib' has no attribute 'axes'. Did you mean: 'axis'?  However, if I remove the line plt.colorbar()  I get the following result, which is OK, but I really need the colorbar. SpectrogramNoColorBar I need this: SpectrogramWithColorBar I tried using the object oriented interface for displaying the spectrogram as in the librosa documentation : https://librosa.org/doc/main/auto_examples/plot_display.html, but to no success. Please help me. EDIT: Here is the full code of my script: import librosa, librosa.display import matplotlib.pyplot as plt import numpy as np file = "Audio_ML\\blues_sample.wav" #waveform #loading the audio file with sample rate=22050(fine for audio data) signal, sr = librosa.load(file, sr=22050) # -> signal(numpy array) containing sr*T -> 22050 * 96 # librosa.display.waveshow(signal, sr=sr) # plt.xlabel("Time") # plt.ylabel("Amplitude") # plt.show() #FTT -> spectrum (FAST FOURIER TRANSFORM TO GO FROM TIME DOMAIN TO FREQUENCY DOMAIN) fft = np.fft.fft(signal) magnitude = np.abs(fft) #magnitudes of each frequency frequency = np.linspace(0, sr, len(magnitude)) left_frequency = frequency[:int(len(frequency)/2)] left_magnitude = magnitude[:int(len(frequency)/2)] # plt.plot(left_frequency, left_magnitude) # plt.xlabel("Frequency") # plt.ylabel("Magnitude") # plt.show() #STFT -> spectogram (SHORT TIME FOURIER TRANSFORM) n_fft = 2048 #number of samples (window of performing an FFT) hop_length = 512 #amount shifting each fourier transform to the right stft = librosa.core.stft(signal, hop_length=hop_length, n_fft=n_fft) spectogram = np.abs(stft) log_spectogram = librosa.amplitude_to_db(spectogram) librosa.display.specshow(log_spectogram, sr=sr, hop_length=hop_length) plt.xlabel("Time") plt.ylabel("Frequency") plt.colorbar() plt.show() #MFCCs MFCCs = librosa.feature.mfcc(y=signal, n_fft=n_fft, hop_length=hop_length, n_mfcc=13)  I am using matplotlib 3.7 and librosa 0.10.0

Answer: It seems the problem was to do with the most recent version of matplotlib. I managed to fix the problem by downgrading matplotlib 3.7.0 to matplotlib 3.6.0 using pip install matplotlib==3.6.0  I got the desired colorbar after doing this. Thank you for trying to help!
Question: While trying to install the requirements to use magic sql, I encounter this error, No module named 'sqlalchemy.databases', with this code: import pandas as pd from sqlalchemy.engine import create_engine # Presto engine = create_engine('presto://localhost:8080/system/runtime') #Read Presto Data query into a DataFrame df = pd.read_sql('select * from queries limit 1', engine) df.head()  Thanks a lot for your reply.

Answer: ipython-sql 0.5.0 depends on sqlalchemy 2 which deprecated the sqlalchemy.databases package in favor of the sqlalchemy.dialects package (details). My solution was to pin both of these libraries to the following versions: %pip install ipython-sql==0.4.1 %pip install sqlalchemy==1.4.46