eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I don't understand why the code: from typing import Optional from pydantic import Field from pydantic.dataclasses import dataclass @dataclass class Klass: field1: str = Field(min_length=1) field2: str = Field(min_length=1) field3: Optional[str]  throws the error: TypeError: non-default argument 'field3' follows default argument if by default Field default kwarg is PydanticUndefined. Why are field1 and field2 default arguments? I'm using python 3.8 and pydantic 2.6 I tried field3: Optional[str] = Field(...) and it works. I expected the code block above to work because all fields are required and none has default values.

Answer: TL;DR  field3 is a required argument with type Optional[str], not an optional argument, because you didn't assign anything to field3 in the class definition.  field1 and field2 are technically optional, because the Field object you assign to each provides a default of PydanticUndefined. That value, though, causes a validation error at runtime if you don't supply another argument in its place.    The dataclass decorator is constructing a def statement to define your class's __init__ method that looks something like def __init__(self, field1=PydanticUndefined, field2=PydanticUndefined, field3): ...  The constructed statement is then execed, which is why you get the error about a non-default argument when the class is defined, rather than when you try to instantiate the class. To make field3 optional, you have to provide a default value. field3: Optional[str] = None  This makes the defined statement something like def __init__(self, field1=PydanticUndefined, field2=PydanticUndefined, field3=None): ...   You can't (as far as I know) make field1 or field2 truly required; the PydanticUndefined value just causes __init__ to raise a ValidationError rather than a TypeError if no explicit argument is passed. >>> Klass() Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/chepner/py311/lib/python3.11/site-packages/pydantic/_internal/_dataclasses.py", line 134, in __init__ s.__pydantic_validator__.validate_python(ArgsKwargs(args, kwargs), self_instance=s) pydantic_core._pydantic_core.ValidationError: 2 validation errors for Klass field1 Field required [type=missing, input_value=ArgsKwargs(()), input_type=ArgsKwargs] For further information visit https://errors.pydantic.dev/2.5/v/missing field2 Field required [type=missing, input_value=ArgsKwargs(()), input_type=ArgsKwargs] For further information visit https://errors.pydantic.dev/2.5/v/missing  I haven't dug into the source to see exactly how that happens, but I assume it's something resembling def __init__(self, field1=PydanticUndefined, ...): if field1 is PydanticUndefined: # prepare ValidationError exception if field2 is PydanticUndefined: # prepare ValidationError exception if <ValidationError needs to be raised>: raise ValidationError(...)  If desired, you can provide "real" default values for field1 and field2 by adding the default keyword argument to Field.
Question: I have 3 files all within the same directory:  ex.awk data.csv file.py  I want to execute my awk file using subprocess so I can call it within a PyTest testcase in the future. These are my sample files: ex.awk #!/usr/bin/awk -f BEGIN { FS = ","; } { print NF; } END {print "DONE"}  data.csv 10,34,32 4,76,8304 4759,5869,2940  file.py import subprocess cmd = ['awk', '-f', 'ex.awk', 'data.csv'] subprocess.run(cmd)  On bash, the following command works: $ awk -f ex.awk data.csv 3 3 3 DONE  I'm trying to get the same result when running file.py, but I get the following FileNotFoundError: [WinError 2] The system cannot find the file specified Am I missing some args in my array? Did I need to use some other subprocess args like shell or others?

Answer: To execute the awk script from Python on Windows where you have encountered a FileNotFoundError, it's often due to the way Windows handles file paths or the execution of Unix/Linux-specific utilities that might not be directly available in the Windows command prompt environment. So you need to provide the absolute path or update Environment variable path to make it available. Try these after adjusting the 'C:/Program Files/Git/usr/bin/awk.exe' path as necessary; import subprocess cmd = ['C:/Program Files/Git/usr/bin/awk.exe', '-f', 'ex.awk', 'data.csv'] subprocess.run(cmd, shell=True)  Alternatively, to execute it within a Unix-like shell provided by Git Bash or WSL: import subprocess cmd = 'awk -f ex.awk data.csv' subprocess.run(cmd, shell=True, executable='C:/Program Files/Git/bin/bash.exe')
Question: Given a dictionary like this with some items being tuples... params = { 'a': 'static', 'b': (1, 2), 'c': ('X', 'Y') }  I need the "product" of the items into a list of dict like this, with the tuples expanded so each item in b will be matched with each item in c... [{ 'a': 'static', 'b': 1, 'c': 'X' }, { 'a': 'static', 'b': 1, 'c': 'Y' }, { 'a': 'static', 'b': 2, 'c': 'X' }, { 'a': 'static', 'b': 2, 'c': 'Y')}]  I can easily separate the initial input into a list of non-tuple items and tuple items, and apply the key of each tuple to the values as a "tag" prior to multiplication so they look like this: 'b##1', 'b##2', 'c##X', 'c##Y'. Then parse those back into the above dict after multiplication. If I would always see 2 tuple items (like b and c), I could easily pass both to itertools.products. But there could be 0..n tuple items, and product() doesn't multiply a list of lists in this way. Can anyone think of a solution? TAG = '##' # separate tuples and non-tuples from the input, and prepend the key of each tuple as a tag on the value to parse out later for key, value in params.items(): if type(value) is tuple: for x in value: tuples.append(f'{key}{TAG}{x}') else: non_tuples.append({key: value}) print(list(product(tuples)) # BUG: doesn't distribute each value of b with each value of c 

Answer: product takes multiple iterables, but the key thing to remember is that an iterable can contain a single item. In cases where a value in your original dict isn't a tuple (or maybe a list), you want to convert it to a tuple containing a single value and pass that to product: params_iterables = {} for k, v in params.items(): if isinstance(v, (tuple, list)): params_iterables[k] = v # v is already a tuple or a list else: params_iterables[k] = (v, ) # A tuple containing a single value, v  which gives: params_iterables = {'a': ('static',), 'b': (1, 2), 'c': ('X', 'Y')}  Then, simply get the product of the values in params_iterables: result = [] for values in product(*params_iterables.values()): result.append(dict(zip(params, values)))  The dict(zip(params, values)) line creates a dict where the first element of values is assigned the first key in params, and so on. This dict is then appended to result, which gives the desired output: [{'a': 'static', 'b': 1, 'c': 'X'}, {'a': 'static', 'b': 1, 'c': 'Y'}, {'a': 'static', 'b': 2, 'c': 'X'}, {'a': 'static', 'b': 2, 'c': 'Y'}]
Question: I am a beginner of Pandas. I have a dataframe like below and want to calculate the "CV" day difference since "Visit" per "ID", as the "CV_days_after_visit" in the expected result.  May I know how I can achieve this with Pandas in Python import pandas as pd data1 = {'ID': ['A2A', 'A2A', 'A2A', 'BB3', 'BB3', 'BB3', '5EE', '5EE'], 'Action': ['Visit', 'CV', 'CV', 'Visit', 'Visit', 'CV', 'Visit', 'CV'], 'date': ['2023/4/1', '2023/4/5', '2023/4/7', '2023/5/5', '2023/5/29', '2023/5/30', '2023/6/1', '2023/6/10']} df = pd.DataFrame(data1) print (df) ID Acton date 0 A2A Visit 2023/4/1 1 A2A CV 2023/4/5 2 A2A CV 2023/4/7 3 BB3 Visit 2023/5/5 4 BB3 Visit 2023/5/29 5 BB3 CV 2023/5/30 6 5EE Visit 2023/6/1 7 5EE CV 2023/6/10   Thanks in advance.

Answer: You can subtract forward filling by GroupBy.ffill dates if Visit by Series.sub and convert timedeltas to days by Series.dt.days, last if necessary remove 0 days by Series.mask: #convert column to datetimes df['date'] = pd.to_datetime(df['date']) m = df['Acton'].eq('Visit') df['CV_days_after_visit'] = (df['date'].sub(df['date'].where(m).groupby(df['ID']).ffill()) .dt.days .mask(m)) print (df) ID Acton date CV_days_after_visit 0 A2A Visit 2023-04-01 NaN 1 A2A CV 2023-04-05 4.0 2 A2A CV 2023-04-07 6.0 3 BB3 Visit 2023-05-05 NaN 4 BB3 Visit 2023-05-29 NaN 5 BB3 CV 2023-05-30 1.0 6 5EE Visit 2023-06-01 NaN 7 5EE CV 2023-06-10 9.0  How it working: df['date'] = pd.to_datetime(df['date']) m = df['Acton'].eq('Visit') print (df.assign(m1 = df['date'].where(m), ffill = df['date'].where(m).groupby(df['ID']).ffill(), sub = df['date'].sub(df['date'].where(m).groupby(df['ID']).ffill()), days = df['date'].sub(df['date'].where(m).groupby(df['ID']).ffill()) .dt.days)) ID Acton date m1 ffill sub days 0 A2A Visit 2023-04-01 2023-04-01 2023-04-01 0 days 0 1 A2A CV 2023-04-05 NaT 2023-04-01 4 days 4 2 A2A CV 2023-04-07 NaT 2023-04-01 6 days 6 3 BB3 Visit 2023-05-05 2023-05-05 2023-05-05 0 days 0 4 BB3 Visit 2023-05-29 2023-05-29 2023-05-29 0 days 0 5 BB3 CV 2023-05-30 NaT 2023-05-29 1 days 1 6 5EE Visit 2023-06-01 2023-06-01 2023-06-01 0 days 0 7 5EE CV 2023-06-10 NaT 2023-06-01 9 days 9
Question: I have a function in a submodule that normally can be imported like this: from core.somepack import my_func  Instead I would like to import it lazily by a given pythonic string core.somepack.my_func. What is the best way to do it? my_func = some_function_i_am_asking_for('core.somepack.my_func') my_func() 

Answer: Try using python built-in importlib module. It helps importing a module or a function at runtime when i.e the module name is not known until runtime. Here is a sample code for having a lazy import using importlib: import importlib def lazy_import(fn_path): module_path, fn_name = fn_path.rsplit('.', 1) module = importlib.import_module(module_path) fn = getattr(module, fn_name) return fn  You can call the lazy_import function with your 'module_name.func_name' string.
Question: I have the following data in a dataframe.    Timestamp MeasureA MeasureB MeasureC MeasureD     0.00 26.46 63.60 3.90 0.67   0.94 26.52 78.87 1.58 0.42   1.94 30.01 82.04 1.13 0.46   3.00 30.19 82.00 1.17 0.36   4.00 30.07 81.43 1.13 0.42   5.94 30.02 82.46 1.05 0.34   8.00 30.22 82.48 0.98 0.35   9.00 30.00 82.21 1.13 0.33   10.00 30.00 82.34 1.12 0.34    And I'd like to filter the entries using some non-uniform intervals. Let say that my intervals are [1.0, 1.5] What I'm trying to achieve is that, we take the first row (row0), and to get the next valid row, we look at what is the next row whose Timestamp value is greater or equal to row0 + 1.0. In this scenario, the next valid row will be the one with the 1.94 timestamp. Then, for the next valid row, we will use the next item in the intervals array. Which is 1.5. That will make the next row the one with a timestamp value of 4.00. Since 1.94 + 1.5 is equal to 3.44. For the next row, we go back and start from the beginning of the array of intervals. After going through all the data, the resulting dataframe should be:    Timestamp MeasureA MeasureB MeasureC MeasureD     0.00 26.46 63.60 3.90 0.67   1.94 30.01 82.04 1.13 0.46   4.00 30.07 81.43 1.13 0.42   5.94 30.02 82.33 1.11 0.35   8.00 30.22 82.48 0.98 0.35   9.00 30.00 82.21 1.13 0.33    Is there a way to achieve this with the existing filtering methods in pandas?

Answer: Try: from itertools import cycle # the interval: A, B = 1.0, 1.5 comparing, out, last_t = cycle([B, A]), [], float("-inf") j = next(comparing) for i, t in zip(df.index, df.Timestamp): if t >= last_t + j: out.append(i) last_t = t j = next(comparing) print(df.loc[out])  Prints:  Timestamp MeasureA MeasureB MeasureC MeasureD 0 0.00 26.46 63.60 3.90 0.67 2 1.94 30.01 82.04 1.13 0.46 4 4.00 30.07 81.43 1.13 0.42 5 5.94 30.02 82.46 1.05 0.34 6 8.00 30.22 82.48 0.98 0.35 7 9.00 30.00 82.21 1.13 0.33
Question: New to web scraping and coding in general. This is probably an easy problem for someone more experienced... maybe not... here it is: Trying to web scrape a table from wikipedia. I've located the table in the html and added that info in my code. However when I run it I get 'none' returned instead of confirmation the table has been correctly located. from bs4 import BeautifulSoup from urllib.request import urlopen url = 'https://en.wikipedia.org/wiki/List_of_songs_recorded_by_the_Beatles' html = urlopen(url) soup = BeautifulSoup(html, 'html.parser') table = soup.find('table',{'class':'wikitable sortable plainrowheaders jquery-tablesorter'}) print(table)  Return: None

Answer: Remove the jquery-tablesorter from the "class" string - this class is added by javascript and beautifulsoup doesn't see it (note: always observe the real HTML document the servers sends you, that's what is beautifulsoup seeing - press ctrl-U in your browser): from urllib.request import urlopen from bs4 import BeautifulSoup url = "https://en.wikipedia.org/wiki/List_of_songs_recorded_by_the_Beatles" html = urlopen(url) soup = BeautifulSoup(html, "html.parser") table = soup.find("table", {"class": "wikitable sortable plainrowheaders"}) print(table)  Prints: <table class="wikitable sortable plainrowheaders" style="text-align:center"> <caption>Name of song, core catalogue release, songwriter, lead vocalist and year of original release </caption> <tbody><tr> <th scope="col">Song </th> <th scope="col">Core catalogue release(s) </th> <th scope="col">Songwriter(s) </th> ...
Question: I have a Python script that runs on Windows 10 Pro x64. When I open Task Manager, it shows that Python is running as 32-bit application.  This is weird because:  Python 3.12.2 is installed as 64-bit variant (double-checked) Installer got from here This is the only Python instance installed on this PC   Script was launched via double-click on it (if this is important info) Please, explain me:  Why it is still shown as 32-bit in task manager? Or is it error of Task Manager? Is it possible to force running all python scripts as 64-bit? If yes: how?  UPDATE / ANSWER  According to this documentation it is the 32-bit launcher, but the Python will be still 64-bit version to bypass this, right click on the python scripts and select "Open with" and select there the 64-bit Python executable. Typically, something like this :C:\Program Files\Python312\python.exe, than when double-clicking the python script it will appear in task manager like this:   do this only if this is the only instance of Python installed on PC 

Answer: Your Python launcher is 32-bit, but it runs 64-bit Python interpreter.
Question: Here is a minimal example: import pandas as pd import numpy as np np.random.seed(0) idx = pd.MultiIndex.from_product([[1,2,3], ['a', 'b', 'c'], [6, 7]]) df = pd.DataFrame(np.random.randn(18), index=idx) selection = [(1, 'a'), (2, 'b')]  I would like to select all the rows in df that have as index that starts with any of the items in selection. So I would like to get the sub dataframe of df with the indices: (1, 'a', 6), (1, 'a', 7), (2, 'b', 6), (2, 'b', 7)  What is the most straightforward/pythonian/pandasian way of doing this? What I found: sel = [id[:2] in selection for id in df.index] df.loc[sel] 

Answer: You could use boolean indexing with isin: out = df[df.index.isin(selection)]  Output:  0 1 a 6 1.560268 7 0.674709 2 b 6 0.848069 7 0.130719  If you want to select other levels, drop the unused leading levels: # here we want to select on levels 1 and 2 selection = [('a', 6), ('b', 7)] df[df.index.droplevel(0).isin(selection)]  Output:  0 1 a 6 1.560268 b 7 0.137769 2 a 6 0.754946 b 7 0.130719 3 a 6 -2.275646 b 7 -2.199944
Question: I've got a dataframe keptdata. I have a for loop to search through the rows of keptdata, and at one point need to average the previous values. This is the relevant line in my code, that produces the warning: avg = np.average(keptdata.iloc[i-500:i].price[keptdata.price != 0])  Here i is the variable that's being looped over in the for loop, price is a column in the dataframe, and I'm including the row in the average only if price is not zero. Why am I getting a warning? Googling for the warning it seems to happen if there are NaNs in the dataframe, but I checked for that using keptdata.isnull.any().any() which returns false. Another possibility is that if i-500 is negative, that might be causing problems, but I rewrote the for loop to start with i = 700 and I still get the warning. Removing keptdata.price != 0 seems to resolve the issue, but that changes the line fundamentally. Can the cause of the warning be fixed? Do I have any options other than suppressing the warning?

Answer: I figured this one out. The problem was the keptdata dataframe had a series of at least five hundred 0's for price. This means that after inserting the condition keptdata.price != 0, there was no longer anything to take an average over, and numpy sensibly returns a "Mean of empty slice" warning.
Question: I have a python script that passes arguments "-i on" to argv in main define this is part of the code; def main(argv): status = "none" try: opts, args = getopt.getopt(argv,"hi:") except getopt.GetoptError: print sys.argv[0], ' -i on|off' sys.exit(2) for opt, arg in opts: if opt == '-h': print sys.argv[0]," -i on|off" print "Switches invertor on and off" sys.exit() elif opt in ("-i"): status = arg up = UPower() if (up.connect() < 0): print "Could not connect to the device" exit -2 newstatR = 0 if (status == "on"): newstatR = 1  I have another script that i would like to call this script like this; import sys from flask import Flask, request sys.path.insert(0, '/home/domoticame/epever-inverter-read') import ivctl ... ivctl.main('-i on')  This however doesnt reliably work, the execution is different than on command line It seems to execute the off command in stead of on command This however doesnt reliably work, the execution is different than on command line It seems to execute the off command in stead of on command

Answer: Have you tried to pass them as a list? import sys from flask import Flask, request sys.path.insert(0, '/home/domoticame/epever-inverter-read') import ivctl ... ivctl.main('-i on'.split())
Question: I recently installed uv on linux using the command line provided in the documentation: curl -LsSf https://astral.sh/uv/install.sh | sh  It created an executable in /home/currentuser/.cargo/bin/uv. Now, just out of curiosity, I would like to know how to remove it properly. Is it as simple as deleting a file ? Or is there any script or command line which is cleaner? Please note that I also tried with pip install/uninstall uv, and it worked perfectly, but the installation path was different.

Answer: The official documentation now has complete uninstall instructions: # Remove all data that uv has stored uv cache clean rm -r "$(uv python dir)" rm -r "$(uv tool dir)" # Remove binaries rm ~/.local/bin/uv ~/.local/bin/uvx  If your original installation was a version earlier than 0.5.2 (November 2024), then replace the last line with: rm ~/.cargo/bin/uv ~/.cargo/bin/uvx  These instructions cover macOS and Linux. See the documentation for Windows instructions.
Question: I am trying to use borutashap for feature selection of a machine learning project. I can't run: from BorutaShap import BorutaShap  always came across this error: ImportError: cannot import name 'binom_test' from 'scipy.stats'  I upgraded both borutashap and scipy to make sure they are the latest version. I am using a conda env, and I removed the whole env and recreated a new env and reinstalled all libraries, but still the same mistake. Could anyone help me with this issue? Thank you in advance.

Answer: The function scipy.stats.binom_test() was removed from SciPy 1.12.0. Source. There is an alternative, scipy.stats.binomtest(), which is called slightly differently. Here are the docs for old and new. I believe you could fix this either by downgrading SciPy to 1.11.4 or by applying a fix to BorutaShap to call the new binomial test function. I believe you'd need to install the package in editable form, then edit Boruta-Shap/src/BorutaShap.py to do this, on two lines: Line 9: Change from scipy.stats import binom_test, ks_2samp  to from scipy.stats import binomtest, ks_2samp  Line 885: Change  return [binom_test(x, n=n, p=p, alternative=alternative) for x in array]  to  return [binomtest(x, n=n, p=p, alternative=alternative).pvalue for x in array]  See also this issue on the BorutaShap issue tracker.
Question: import polars as pl df = pl.DataFrame({ "Letter": ["A", "A", "B", "B", "B", "C", "C", "D", "D", "E"], "Value': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] })  I want to group Letter and collect their corresponding Value in a List. Related Pandas question: How to group dataframe rows into list in pandas groupby I know pandas code will not work here: df.group_by("a")["b"].apply(list)   TypeError: 'GroupBy' object is not subscriptable  Output will be: ┌────────┬───────────┐ │ Letter ┆ Value │ │ --- ┆ --- │ │ str ┆ list[i64] │ ╞════════╪═══════════╡ │ A ┆ [1, 2] │ │ B ┆ [3, 4, 5] │ │ C ┆ [6, 7] │ │ D ┆ [8, 9] │ │ E ┆ [10] │ └────────┴───────────┘ 

Answer: You could do this. import polars as pl df = pl.DataFrame( { 'Letter': ['A', 'A', 'B', 'B', 'B', 'C', 'C', 'D','D','E'], 'Value': [1, 2, 3, 4, 5, 6, 7, 8, 9,10] } ) g = df.group_by('Letter', maintain_order=True).agg(pl.col('Value')) print(g)  This will print ┌────────┬───────────┐ │ Letter ┆ Value │ │ --- ┆ --- │ │ str ┆ list[i64] │ ╞════════╪═══════════╡ │ A ┆ [1, 2] │ │ B ┆ [3, 4, 5] │ │ C ┆ [6, 7] │ │ D ┆ [8, 9] │ │ E ┆ [10] │ └────────┴───────────┘  maintain_order=True is required if you want to order of the groups to be consistent with the input data.
Question: In python, I'm trying to create a polars DataFrame from a pyarrow table, like so: import pyarrow as pa import polars as pl table = pa.table( { "a": [1, 2, 3], "b": [4, 5, 6], } ) df = pl.from_arrow(table)  The return-type of pl.from_arrow() is (DataFrame | Series), even though my df really always is a DataFrame. This results in type-checker-warnings further down the line, where I want to perform actions on the data. For instance: df.select(pl.col("a"))  Cannot access member "select" for type "Series" Member "select" is unknown Pylance(reportAttributeAccessIssue)  I could try to find a work-around, but haven't found a "proper" way to do this yet. For instance, I could do: df = pl.DataFrame(table) df.select(pl.col("a")) # or maybe even if (isinstance(df, pl.Series)): raise TypeError  What would be the proper way to deal with this situation? How to assure the type-checker that pl.from_arrow() really returns a DataFrame?

Answer: You can use the cast function.  from typing import cast import pyarrow as pa import polars as pl table = pa.table( { "a": [1, 2, 3], "b": [4, 5, 6], } ) df = cast(pl.DataFrame, pl.from_arrow(table)) reveal_type(df)  Since pl.from_arrow can also return a Series, this force the typechecker to consider the type you put there if you're sure about it.
Question: I am trying to integrate Apache Arrow with Apache Spark in a PySpark application, but I am encountering an issue related to sun.misc.Unsafe or java.nio.DirectByteBuffer during the execution. import os import pandas as pd from pyspark.sql import SparkSession extra_java_options = os.getenv("SPARK_EXECUTOR_EXTRA_JAVA_OPTIONS", "") spark = SparkSession.builder \ .appName("ArrowPySparkExample") \ .getOrCreate() spark.conf.set("Dio.netty.tryReflectionSetAccessible", "true") spark.conf.set("spark.sql.execution.arrow.pyspark.enabled", "true") pdf = pd.DataFrame(["midhun"]) df = spark.createDataFrame(pdf) result_pdf = df.select("*").toPandas()  Error Message: in stage 0.0 (TID 11) (192.168.140.22 executor driver): java.lang.UnsupportedOperationException: sun.misc.Unsafe or java.nio.DirectByteBuffer.<init>(long, int) not available at org.apache.arrow.memory.util.MemoryUtil.directBuffer(MemoryUtil.java:174) at org.apache.arrow.memory.ArrowBuf.getDirectBuffer(ArrowBuf.java:229)  Environment: Apache Spark version: 3.4 Apache Arrow version: 1.5 Java version: jdk 21

Answer: Same issue with:  Apache Spark version 3.5.1 Java JDK 21  Downgrading java to test minimum supported version. Update:  Java JDK 17 resolves this issue, please see supported version of Spark: https://spark.apache.org/docs/latest/ (see Spark runs on Java)