eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'm building a package on top of Polars, and one of the functions looks like this def func(x: IntoExpr, y: IntoExpr): ...  The business logic requires that x can include multiple columns, but y must be a single column. What should I do to check and validate this?

Answer: You can use the polars.selectors.expand_selector function which lets you evaluate selected columns using either selectors or simple expressions. Note that the drawback here is that you can’t pass in arbitrary expressions, or else the evaluation fails (see the final examples). import polars as pl import polars.selectors as cs from polars.selectors import expand_selector data = { "a1": [1, 2, 3], "a2": [4, 5, 6], "b1": [7, 8, 9], "b2": [10, 11, 12], } df = pl.DataFrame(data) print( expand_selector(df, cs.exclude('b1', 'b2')), # ('a1', 'a2') expand_selector(df, cs.starts_with('b')), # ('b1', 'b2') expand_selector(df, cs.matches('(a|b)1$')), # ('a1', 'b1') # use with expressions expand_selector(..., strict=False) expand_selector(df, pl.exclude('a1', 'a2'), strict=False), # ('b1', 'b2') expand_selector(df, pl.col('b1'), strict=False), # ('b1', ) expand_selector(df, pl.all(), strict=False), # ('a1', 'a2', 'b1', 'b2') sep='\n' ) # anything past an arbitrary selection expression will fail print(expand_selector(df, pl.all() + 1, strict=False)) # Traceback (most recent call last): # File "/home/cameron/.vim-excerpt", line 26, in <module> # expand_selector(df, pl.all() + 1, strict=False), # File "/home/cameron/.pyenv/versions/dutc-site/lib/python3.10/site-packages/polars/selectors.py", line 190, in expand_selector # raise TypeError(msg) # TypeError: expected a selector; found <Expr ['[(*) + (dyn int: 1)]'] at 0x7F835F943D30> instead.
Question: I am trying to write a custom logging.Logger subclass which is mostly working, but I run into issues when trying to use a logging.Formatter that includes the interpolated value %(filename) in the custom format, it prints the filename where my custom subclass is, rather than the filename of the code that called the logging function. I've found a number of tutorials for subclassing Logger but none of them address the effects this has on the filename interpolation. Is there a straightforward solution to this without having to override large sections of logging.Logger? Sample code defining my custom logger: #------------------------------------- # custom_logger.py #------------------------------------- import logging import io class CustomLogger(logging.Logger): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) self.setLevel(logging.DEBUG) # create the record format formatter = logging.Formatter(fmt = "%(filename)s - %(message)s") # create the handler self.stream = io.StringIO() handler = logging.StreamHandler(self.stream) handler.setFormatter(formatter) self.addHandler(handler) def debug(self, msg, *args, **kwargs): super().debug(msg, *args, **kwargs) # do some other stuff ...  #------------------------------------- # test.py #------------------------------------- from custom_logger import CustomLogger import logging logging.setLoggerClass(CustomLogger) myLog = logging.getLogger("myLog") myLog.debug("hello world") print(myLog.stream.getvalue())  Expected output: >>> test.py - hello world Actual output: >>> custom_logger.py - hello world

Answer: This is already answered in https://stackoverflow.com/a/59492341/2138700 The solution is to use the stacklevel keyword argument when calling the super().debug in your custom_logger code. Here is the relevant section from the documentation  The third optional keyword argument is stacklevel, which defaults to 1. If greater than 1, the corresponding number of stack frames are skipped when computing the line number and function name set in the LogRecord created for the logging event. This can be used in logging helpers so that the function name, filename and line number recorded are not the information for the helper function/method, but rather its caller.  So the modified code should be import logging import io class CustomLogger(logging.Logger): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) self.setLevel(logging.DEBUG) # create the record format formatter = logging.Formatter(fmt = "%(filename)s - %(message)s") # create the handler self.stream = io.StringIO() handler = logging.StreamHandler(self.stream) handler.setFormatter(formatter) self.addHandler(handler) def debug(self, msg, *args, **kwargs): super().debug(msg, *args, stacklevel=2, **kwargs)  Now the output will be test.py - hello world
Question: I'm building a Discord bot. The bot should store some information into some internal variables to be accessed at a later time. To do so I'm structuring it as a class (as opposed to many examples where the commands are outside a class definition). However, I discovered that when you use the @commands.command(name='test') decorator, the method becomes a kind of "static" method and no longer receives the object as first input. Given this, is there any way I can access class properties (such as an_instance_property in the example below) and/or class methods (such as a_class_method in the example below)? If this is the wrong approach, what could be a better approach for a bot with an internal state? import discord from discord.ext import commands with open('TOKEN', 'r') as f: TOKEN = f.read() class mybot(commands.Bot): def __init__(self): intents = discord.Intents.default() super().__init__(command_prefix="!", intents=intents) self.add_command(self.test) self.an_instance_property = [] # <---- def a_class_method(x): # <---- return x @commands.command(name='test') async def test(ctx, *args): # How can I access self.an_instance_property from here? # How can I call self.a_class_method from here? return bot = mybot() bot.run(TOKEN) 

Answer: My recommendation is that you avoid defining commands inside your bot class. There is a more appropriate way to do this, which is using cogs/extensions. See this topic where commands are created in a separate file (extension) and only loaded into the bot class: https://stackoverflow.com/a/78166456/14307703 Also know that the Context object always carries the instance of your bot. So you can access all the properties of the class like this: class MyBot(commands.Bot): def __init__(self): intents = discord.Intents.default() super().__init__(command_prefix="!", intents=intents) self.add_command(self.test) self.an_instance_property = [] # <---- def a_class_method(x): # <---- return x @commands.command(name='test') async def test(ctx, *args): # How can I access self.an_instance_property from here? print(ctx.bot.an_instance_property) # <---- return
Question: how to pandas fast nested for loop for "non numeric" columns? because this for loop is way to slow: for i in range(len(df1[column_A]): for j in range(len(df2[column_A]): if df1[column_A][i] == df2[column_A][j]: df1[column_B][i] = df2[column_B][j] else: pass  so any other way to do it by pandas itself or other libraries?  UPDATE:  and main goal is: input: df1: name rpm power 0 John 1500 high+ 1 Mary 1400 high- 2 Sally 300 low- 3 Doe 700 medium- 4 July 1000 medium+ df2: name age 0 Peter 77 1 Sally 44 2 Micky 22 3 Sally 34 4 July 50 5 Bob 20 required output is: but i want it df2: name age rpm power 0 Peter 77 0 NA 1 Sally 44 300 low- 2 Micky 22 0 NA 3 Sally 34 300 low- 4 July 50 1000 medium+ 5 Bob 20 0 NA  i also add question in official pandas github: https://github.com/pandas-dev/pandas/issues/59824

Answer: The nested loop you provided result in O(n^2) complexity, making it slow for larger datasets . Looping over the same range for both i and j, which is unnecessary. Instead you can use pd.merge import pandas as pd # Merge file1 and file2 on column A merged_df = pd.merge(file1, file2, on='column_A') # assuming it is a pandas dataframe # Update file1_column_B with matched values from file2 file1_column_B = merged_df['column_B_y']  pd.merge() function merges two DataFrames ( file1 and file2 ) based on a common column ( column_A ).Also by default, pd.merge() performs an inner merge, which means only rows with matching values in column_A are included in the resulting DataFrame. Time complexity of pd.merge is O(n + m) in best case scenario (concluding it) where n is the number of rows in the left DataFrame (file1) and m is the number of rows in the right DataFrame (file2). However, in the worst-case scenario (e.g., when there are many duplicate values in the merge column), the time complexity can be O(n × m). You can also use numPy argsort + searchsorted import numpy as np sorted = np.argsort(file2['column_A']) match = np.searchsorted(file2['column_A'][sorted], file1['column_A']) file1_column_B = file2['column_B'][sorted[match]]  Time Complexity of above sort: argsort: O(n log n) searchsorted: O(m log n) Total: O(n log n + m log n)
Question: I'd like to get the content of an <a> tag using BeautifulSoup (version 4.12.3) in Python. I have this code and HTML exemple: h = """ <a id="0"> <table> <thead> <tr> <th scope="col">Person</th> <th scope="col">Most interest in</th> <th scope="col">Age</th> </tr> </thead> <tbody> <tr> <th scope="row">Chris</th> <td>HTML tables</td> <td>22</td> </tr> </table> </a> """ test = bs4.BeautifulSoup(h) test.find('a') # find_all, select => same results  But it only returns : <a id="0"> </a>  I'd would expect that the content inside <table> would appear between <a> tags. (I don't know if it is common to wrap a table inside an <a> tag but the HTML code I try to read is like so) I need to parse the table content from the <a> tag since I need to link the id="0" to the content of the table. How can I achieve that ? How can I get the <a> tag content with the <table> tag ?

Answer: Specify explicitly the parser you want to use (use html.parser). By default it will use the "best" parser available - I pressume lxml which doesn't parse this document well: import bs4 h = """ <a id="0"> <table> <thead> <tr> <th scope="col">Person</th> <th scope="col">Most interest in</th> <th scope="col">Age</th> </tr> </thead> <tbody> <tr> <th scope="row">Chris</th> <td>HTML tables</td> <td>22</td> </tr> </table> </a> """ test = bs4.BeautifulSoup(h, "html.parser") # <-- define parser here out = test.find("a") print(out)  Prints: <a id="0"> <table> <thead> <tr> <th scope="col">Person</th> <th scope="col">Most interest in</th> <th scope="col">Age</th> </tr> </thead> <tbody> <tr> <th scope="row">Chris</th> <td>HTML tables</td> <td>22</td> </tr> </tbody></table> </a>
Question: How can I find duplicates of a group of rows inside of a DataFrame? Or in other words, how can I find the indices of a specific duplicated DataFrame inside of a larger DataFrame? The larger DataFrame:    index 0 1     0 0 1   1 2 3   2 4 4   3 0 1   4 2 3   5 2 3   6 0 1    The specific duplicated DataFrame (or group of rows):    index 0 1     0 0 1   1 2 3    Indices I am looking for:    index     0   1   3   4    (Note that the indices of the duplicated DataFrame do not matter, only the values). import pandas as pd # larger DataFrame lrg_df = pd.DataFrame([[0, 1], [2, 3], [4, 4], [0, 1], [2, 3], [2, 3], [0, 1]]) # group of rows (i.e., duplicated DataFrame) dup_df = pd.DataFrame([[0, 1], [2, 3]]) # get indices of lrg_df that contain dup_df indcs = lrg_df[lrg_df == dup_df].index # Doesn't work of course 

Answer: You need to check all combinations with a sliding window, using numpy.lib.stride_tricks.sliding_window_view to create a mask and extend the mask with numpy.convolve: import numpy as np from numpy.lib.stride_tricks import sliding_window_view as swv n = len(dup_df) mask = (swv(lrg_df, n, axis=0) == dup_df.to_numpy().T ).all((1,2)) out = lrg_df[np.convolve(mask, np.ones(n))>0]  Output:  0 1 0 0 1 1 2 3 3 0 1 4 2 3  And if you want the indices: indices = lrg_df.index[np.convolve(mask, np.ones(n))>0]  Output: Index([0, 1, 3, 4], dtype='int64')  Intermediates: # swv(lrg_df, n, axis=0) == dup_df.to_numpy().T array([[[ True, True], [ True, True]], [[False, False], [False, False]], [[False, False], [False, False]], [[ True, True], [ True, True]], [[False, True], [False, True]], [[False, False], [False, False]]]) # mask array([ True, False, False, True, False, False])
Question: I have a Pandas dataframe with a column that contains different types of values and I want to create a new column out of it based on the information inside that column. Every few rows there is a kind of "headline" row that should define that values for the following rows until the next headline row that then defines the values for the next rows and so on. To understand better, here is an example: import pandas as pd import pandas as pd data = {'AA': ['', '', '', 'V_525-124', 'gsdgsd', 'hdfjhdf', 'gsdhsdhsd', 'gsdgsd', 'V_535-623', 'hosdfjk', 'hjodfjh', 'hjsdfjo', 'V_563-534', 'hojhdfhjdf', 'hodfjhjdfj', 'hofoj', 'hkdfphdf']} df = pd.DataFrame(data) print(df)  I want to create a new column BB that would look like that: import pandas as pd data = {'AA': ['', '', '', 'V_525-124', 'gsdgsd', 'hdfjhdf', 'gsdhsdhsd', 'gsdgsd', 'V_535-623', 'hosdfjk', 'hjodfjh', 'hjsdfjo', 'V_563-534', 'hojhdfhjdf', 'hodfjhjdfj', 'hofoj', 'hkdfphdf'], 'BB': ['', '', '', 'V_525-124', 'V_525-124', 'V_525-124', 'V_525-124', 'V_525-124', 'V_535-623', 'V_535-623', 'V_535-623', 'V_535-623', 'V_563-534', 'V_563-534', 'V_563-534', 'V_563-534', 'V_563-534']} df = pd.DataFrame(data) print(df)  The number of rows under each "headline" varies, so the script should sort of check whether the next row is a headline-type, then add the headline value to column BB and then move on down the table until a new headline is detected. I can only think of a for-loop with indices and if-statements but I am sure Pandas offers a more elegant solution. The "headlines" all start with 'V_' if that helps.

Answer: You can use where and ffill (forward fill) without the need for loops:  df['AA'].where(df['AA'].str.startswith('V_')).ffill().fillna('')  str.startswith to identify rows where AA column starts with 'V_'. where to keep identified headline rows in BB column and set other rows to NaN. ffill to forward fill the last valid headline value down the column until the next headline is identified. fillna('') to replace remaining NaN values with empty strings    import pandas as pd data = {'AA': ['', '', '', 'V_525-124', 'gsdgsd', 'hdfjhdf', 'gsdhsdhsd', 'gsdgsd', 'V_535-623', 'hosdfjk', 'hjodfjh', 'hjsdfjo', 'V_563-534', 'hojhdfhjdf', 'hodfjhjdfj', 'hofoj', 'hkdfphdf']} df = pd.DataFrame(data) df['BB'] = df['AA'].where(df['AA'].str.startswith('V_')).ffill().fillna('') print(df)
Question: I have over 12 dataframes that I want to merge into a single dataframe, where row values match for each column (or null if they don't exist). Each dataframe has a different number of rows, but will never repeat values. The goal is to both identify common values and missing values. Eg.df1 id label 1 a-1 2 b-2 3 z-10  Eg.df2 id label 1 b-2 2 d-4 3 e-5  Eg.df3 id label 1 a-1 2 d-4 3 f-6  Desired output Eg.final id df1 df2 df3 1 a-1 null a-1 2 b-2 b-2 null 3 null d-4 d-4 4 null e-5 null 5 null null f-6 6 z-10 null null  I've investigated join, but these all seem to collapse values. insert seemed plausible, but I can't rectify the different row sizes/matching values to the same row. I want to maintain each df as it's own column.

Answer: For multiple dataframes, you can use merge with reduce: from functools import reduce reduce(lambda left, right: pd.merge(left, right, on='label', how='outer'), map(lambda d: d[1].drop(columns='id') .assign(**{ f'df{d[0]}':lambda x: x['label'] }), enumerate(dfs, 1)) ).assign(id=lambda x:range(1, 1+len(x))).drop(columns='label') # this is just to drop the existing `label` and assign new `id`  Out:  df1 df2 df3 id 0 a-1 NaN a-1 1 1 b-2 b-2 NaN 2 2 NaN d-4 d-4 3 3 NaN e-5 NaN 4 4 NaN NaN f-6 5 5 z-10 NaN NaN 6  Another method is join on the index, like: renamed_dfs = list(map(lambda d: d[1].drop(columns='id') .assign(**{ f'df{d[0]}':lambda x: x['label'] }) .set_index('label'), enumerate(dfs, 1) )) renamed_dfs[0].join(renamed_dfs[1:], how='outer').reset_index(drop=True).reset_index()  Output:  index df1 df2 df3 0 0 a-1 NaN a-1 1 1 b-2 b-2 NaN 2 2 z-10 NaN NaN 3 3 NaN d-4 d-4 4 4 NaN e-5 NaN 5 5 NaN NaN f-6
Question: The imaginary number i, or j in Python, means the square root of -1. So, i to the 4th or any multiple of 4 should be positive 1. >>> (1j)**4 (1+0j) >>> (1j)**96 (1+0j) >>> (1j)**100 (1+0j)  Up until this point all is good, but once we get past 100, Python just bugs out. For example: >>> (1j)**104 (1+7.842691359635767e-15j)  This messed up my calculations so much in an unexpected way. What explains this? I'm using Python 3.8.19.

Answer: See the code, for integer exponents up to 100 it uses a different algorithm: complex_pow(PyObject *v, PyObject *w, PyObject *z) { ... // Check whether the exponent has a small integer value, and if so use // a faster and more accurate algorithm. if (b.imag == 0.0 && b.real == floor(b.real) && fabs(b.real) <= 100.0) { p = c_powi(a, (long)b.real); } else { p = _Py_c_pow(a, b); } ...  The special algorithm for the small exponents uses exponentiation by squaring, just multiplying 1j with itself and with the multiplication results, which is accurate: c_powu(Py_complex x, long n) { Py_complex r, p; long mask = 1; r = c_1; p = x; while (mask > 0 && n >= mask) { if (n & mask) r = _Py_c_prod(r,p); mask <<= 1; p = _Py_c_prod(p,p); } return r; }  The general algorithm for exponents larger than 100 and non-integer exponents uses trigonometry and ends up involving inaccurate floats: _Py_c_pow(Py_complex a, Py_complex b) { ... vabs = hypot(a.real,a.imag); len = pow(vabs,b.real); at = atan2(a.imag, a.real); phase = at*b.real; if (b.imag != 0.0) { len /= exp(at*b.imag); phase += b.imag*log(vabs); } r.real = len*cos(phase); r.imag = len*sin(phase); ...  The at value for 1j mathematically is π/2, which isn't representable exactly as float. And then the phase and sin and thus the imaginary party of the result become inaccurate as well. See Why does math.cos(math.pi/2) not return zero? for more about that. (I just noticed you mentioned using Python 3.8.19, while I looked up the current version, 3.12.6. The code slightly differs, but irrelevantly so, and the explanation is correct for both.)
Question: I just want to ask how I can call another .py from the converted .exe file? I have a main Python code called main_2.py file that acts as the main interface for the program that I created, which in this case, is the one that will be converted into .exe. One of its main functions is to open up another python code called analysis.py which in this case, is another GUI itself that will do other tasks. My question is, can the convert .exe file called the .py directly, or should I convert the analysis.py into .exe as well?

Answer: It should be enough to import the functions from the second file, as explained here! After that, auto-py-to-exe will automatically create an exe file with everything included needed to run it! Another way is to add other py scripts as additional files!
Question: I use: duration = float(ffmpeg.probe(audio_path)["format"]["duration"])  I collect an audio/video's length and want to store it using my models. Should I use models.DecimalField() or models.FloatField()? I use it to calculate and store a credit/cost in my model using credit = models.DecimalField(max_digits=20, decimal_places=4) 

Answer: I think the most sensical way is to use a DurationField model field [Django-doc]. Django will look what database the backend uses, and try to work with the most sensical column type the database offers: class MyModel(models.Model): duration = models.DurationField() and work with: from datetime import timedelta MyModel.objects.create( duration=timedelta( seconds=float(ffmpeg.probe(audio_path)['format']['duration']) ) )
Question: I have a dataframe like: data = { "a": [[1], [2], [3, 4], [5, 6, 7]], "b": [[], [8], [9, 10], [11, 12]], } df = pl.DataFrame(data) """ ┌───────────┬───────────┐ │ a ┆ b │ │ --- ┆ --- │ │ list[i64] ┆ list[i64] │ ╞═══════════╪═══════════╡ │ [1] ┆ [] │ │ [2] ┆ [8] │ │ [3, 4] ┆ [9, 10] │ │ [5, 6, 7] ┆ [11, 12] │ └───────────┴───────────┘ """  Each pair of lists may not have the same length, and I want to "truncate" the explode to the shortest of both lists: """ ┌─────┬─────┐ │ a ┆ b │ │ --- ┆ --- │ │ i64 ┆ i64 │ ╞═════╪═════╡ │ 2 ┆ 8 │ │ 3 ┆ 9 │ │ 4 ┆ 10 │ │ 5 ┆ 11 │ │ 6 ┆ 12 │ └─────┴─────┘ """  I was thinking that maybe I'd have to fill the shortest of both lists with None to match both lengths, and then drop_nulls. But I was wondering if there was a more direct approach to this?

Answer: Here's one approach: min_length = pl.min_horizontal(pl.col('a', 'b').list.len()) out = (df.filter(min_length != 0) .with_columns( pl.col('a', 'b').list.head(min_length) ) .explode('a', 'b') )  Output: shape: (5, 2) ┌─────┬─────┐ │ a ┆ b │ │ --- ┆ --- │ │ i64 ┆ i64 │ ╞═════╪═════╡ │ 2 ┆ 8 │ │ 3 ┆ 9 │ │ 4 ┆ 10 │ │ 5 ┆ 11 │ │ 6 ┆ 12 │ └─────┴─────┘  Explanation  Get the length for the lists in both columns with Expr.list.len and get the shortest for each row with pl.min_horizontal. Now, filter out the rows where min_length == 0 (df.filter) and inside df.with_columns select the first n values of each list with Expr.list.head. Finally, apply df.explode.
Question: I have a Python package, which is distributed on PyPi. It depends on number of other packages available on PyPi and on Psi4, which is only distributed on Conda repositories (https://anaconda.org/psi4/psi4), not on PyPi. Now, my package is distributed as wheel package via hatchling, so my pyproject.toml looks similar to this: [build-system] requires = ["hatchling"] build-backend = "hatchling.build" [project] name = "My project" version = "1.0.0" authors = [ ] description = "New method" readme = "README.md" requires-python = ">=3.12" classifiers = [ "Programming Language :: Python :: 3", "License :: OSI Approved :: GNU General Public License v3 (GPLv3)", "Operating System :: POSIX :: Linux", ] dependencies = [ "qiskit==1..0", "qiskit-nature>=0.5.1", "numpy>=1.23.0", "deprecated>=1.2.0"]  Is there any way to deal with such an external dependency automatically? Ideally it'd download and install Psi4 from its repositories, but if not, is there any way to get at least a warning before the download from PyPi starts? I had a look around and found this related question, which, unfortunately, got no answers:  Distributing pip packages that have non-pypi dependencies 

Answer: If you use a setup.py instead of pyproject.toml, you can add an installation check for psi4. pyproject.toml does not natively support conda packages. For setup.py, you can approach it like: import subprocess import sys try: import psi4 except ImportError: subprocess.check_call([sys.executable, "-m", "conda", "install", "-c", "psi4", "psi4"]) from setuptools import setup setup( # Your package configuration here )  For more information on setuptools and how to create it to replicate the work of your pyproject.toml see here. Additionally, You can add in your README.md about the conda dependency. When the pypi package is downloaded, setup.py is run automatically and therefore you can also add a warning in there too if you dont want to install it directly. This approach work with sdist and not with wheels as they bypass it as mentioned in the comments by @phd. For wheels, you can add a piece of code in your project itself to check for psi4 and either warn and exit or install it. before you begin your work in the code file add something like this: try: import psi4 except ImportError: raise ImportError("Psi4 is required but not installed. Please install it via Conda: " "`conda install -c conda-forge psi4`.")  Additionally, You can add it in your README.md and also add a long description where it is stated properly that psi4 is a requirement.