eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: While running a program on pycharm I am getting below error while running on pycharm using python. Unable to run the program due to below error: ERROR: Failed building wheel for TA-Lib-Precompiled ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (TA-Lib-Precompiled) > Package :TA-Lib-Precompiled > Python Version : Python 3.12.1 > Cython version 3.0.11 Please help in finding the solution !!  Below are the logs : > Collecting TA-Lib-Precompiled Using cached TA-Lib-Precompiled-0.4.25.tar.gz (276 kB) Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Requirement already satisfied: numpy in c:\python312\lib\site-packages (from TA-Lib-Precompiled) (1.26.4) Building wheels for collected packages: TA-Lib-Precompiled Building wheel for TA-Lib-Precompiled (setup.py): started Building wheel for TA-Lib-Precompiled (setup.py): finished with status 'error' Running setup.py clean for TA-Lib-Precompiled Failed to build TA-Lib-Precompiled ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "Cython\\Utils.py", line 129, in Cython.Utils.cached_method.wrapper File "C:\Python312\Lib\site-packages\Cython\Build\Dependencies.py", line 574, in cimports_externs_incdirs for include in self.included_files(filename): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "Cython\\Utils.py", line 129, in Cython.Utils.cached_method.wrapper File "C:\Python312\Lib\site-packages\Cython\Build\Dependencies.py", line 556, in included_files include_path = self.context.find_include_file(include, source_file_path=filename) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Python312\Lib\site-packages\Cython\Compiler\Main.py", line 299, in find_include_file error(pos, "'%s' not found" % filename) File "C:\Python312\Lib\site-packages\Cython\Compiler\Errors.py", line 178, in error raise InternalError(message) Cython.Compiler.Errors.InternalError: Internal compiler error: '_common.pxi' not found [end of output] > note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for TA-Lib-Precompiled ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (TA-Lib-Precompiled) 

Answer: The stable release of TA-Lib-Precompiled only has wheels for Python 3.8 - 3.11 for Linux. You can install The Windows Subsystem for Linux (WSL) which provides a Linux environment on your Windows machine and then use a supported Python version such as Python 3.11. See How to install Linux on Windows with WSL for detailed instructions on this.
Question: I am saving the logs to a text file and displaying them to the console at the same time. I would like to apply a filter on the logs, so that some logs neither make it to the text file nor the console output. However, with this code, the logs that I would like to filter out are still being saved to the text file. The filter only seems to work on the console output. How can I apply the filter to the text file and the console output? Thank you very much import logging class applyFilter(logging.Filter): def filter(self, record): return not record.getMessage().startswith('Hello') logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s', filename='log_file.txt', filemode='a') console = logging.StreamHandler() console.addFilter(applyFilter()) console.setLevel(logging.INFO) formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s') console.setFormatter(formatter) logging.getLogger('').addHandler(console) logging.info('Hello world') 

Answer: basicConfig created a FileHandler and a StreamHandler was also created and added to the logger. The filter was only applied to the StreamHandler. To filter both handlers, add the filter to the logger instead: import logging class applyFilter(logging.Filter): def filter(self, record): return not record.getMessage().startswith('Hello') logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s', filename='log_file.txt', filemode='a') console = logging.StreamHandler() # console.addFilter(applyFilter()) # not here console.setLevel(logging.INFO) formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s') console.setFormatter(formatter) logger = logging.getLogger('') logger.addFilter(applyFilter()) # here logger.addHandler(console) logging.info('Hello world') logging.info('world only')  Output (console): 2024-12-30 00:44:31,702 - INFO - world only  Output (log_file.txt): 2024-12-30 00:44:31,702 - INFO - world only
Question: I'm using the metpy library to do weather calculations. I'm using the multiprocessing library to run them in parallel, but I get rare exceptions, which completely stop the program. I am not able to provide a minimal, reproducible example because I can't replicate the problems with the metpy library functions and because there is a huge amount of code that runs before the problem occurs that I can't put here. I want to know how to write multiprocessing code to tell the pool.starmap function to PASS if it encounters an error. The first step in my code produces an argument list, which then gets passed to the pool.starmap function, along with the metpy function (metpy.ccl, in this case). The argument list for metpy.ccl includes a list of pressure levels, air temperatures, and dew point values. ccl_pooled = pool.starmap(mpcalc.ccl, ccl_argument_list)  I tried to write a generalized function that would take the metpy function I pass to it and tell it to pass when it encounters an error.  def run_ccl(p,t,td): try: result = mpcalc.ccl(p,t,td) except IndexError: pass  Is there a way for me to write the "run_ccl" function so I can check for errors in my original code line - something like this: ccl_pooled = pool.starmap(run_ccl, ccl_argument_list)  If not, what would be the best way to do this? EDIT: To clarify, these argument lists are thousands of data points long. I want to pass on the data point that causes the problem (and enter a nan in the result, "ccl_pooled", for that data point), and keep going.

Answer: You can generalize run_ccl with a wrapper function that suppresses specified exceptions and returns NaN as a default value: from contextlib import suppress def suppressor(func, *exceptions): def wrapper(*args, **kwargs): with suppress(*exceptions): return func(*args, **kwargs) return float('nan') return wrapper  with which you can then rewrite the code into something like: ccl_pooled = pool.starmap(suppressor(mpcalc.ccl, IndexError), ccl_argument_list)
Question: I would like to get a value_of implementation for the StrEnum (Python 3.9.x). For example: from enum import Enum class StrEnum(str, Enum): """Enum with str values""" pass class BaseStrEnum(StrEnum): """Base Enum""" @classmethod def value_of(cls, value): try: return cls[value] except KeyError: try: return cls(value) except ValueError: return None  and then can use it like this: class Fruits(BaseStrEnum): BANANA = "Banana" PEA = "Pea" APPLE = "Apple" print(Fruits.value_of('BANANA')) print(Fruits.value_of('Banana'))  it is just that the nested try-except doesn't look amazing, is there a better more idiomatic rewrite?

Answer: Since upon success of the first try block the function will return and won't execute the code that follows, there is no need to nest the second try block in the error handler of the first try block to begin with: def value_of(cls, value): try: return cls[value] except KeyError: pass try: return cls(value) except ValueError: return None  And since both of the error handlers are really meant to ignore the respective exceptions, you can use contextlib.suppress to simply suppress those errors: from contextlib import suppress def value_of(cls, value): with suppress(KeyError): return cls[value] with suppress(ValueError): return cls(value) # return None  Note that a function returns None by default so you don't have to explicitly return None as a fallback unless you want to make it perfectly clear.
Question: def divisive_recursion(n): try: if n <= 0: return 1 else: return n + divisive_recursion(n // divisive_recursion(n - 1)) except ZeroDivisionError: return -1 finally: if n == 2: print("Finally block executed for n=2") elif n == 1: print("Finally block executed for n=1") print(divisive_recursion(5))  Here, divisive_recursion(1) results in 1 + (1 // divisive_recursion(0)), then divisive_recursion(0) returns 1 and it goes into an infinite recursion, where divisive_recursion(1) and divisive_recursion(0) gets executed repeatedly. I expected the code to give and RecursionError due to this, which it does, but the finally blocks get executed before that somehow, I know that they get executed always but for the prints to be printed n should be equal to 1 or 2, which it never does due to the infinite recursion so how come both the print statements are printed when the condition inside them is never evaluated to be True?

Answer: In one of the comments, you ask "does that mean once the program encounters the crash, it will execute all the finally blocks upward the recursion before it finally crashes". And the answer is basically "yes". An exception isn't really a "crash", or perhaps think of it as a controlled way of crashing. Here is a simple example to illustrate, in this case where the exception is caught and handled: >>> def foo(n): ... if n == 0: ... raise RuntimeError ... try: ... foo(n - 1) ... except: ... print(f'caught exception at {n=}') ... finally: ... print(f'in finally at {n=}') ... >>> foo(5) caught exception at n=1 in finally at n=1 in finally at n=2 in finally at n=3 in finally at n=4 in finally at n=5  And perhaps even more clarifying, here is a case with an uncaught exception: >>> def foo(n): ... if n == 0: ... raise RuntimeError ... try: ... foo(n - 1) ... except ZeroDivisionError: ... print(f'caught exception at {n=}') ... finally: ... print(f'in finally at {n=}') ... >>> foo(5) in finally at n=1 in finally at n=2 in finally at n=3 in finally at n=4 in finally at n=5 Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 5, in foo File "<stdin>", line 5, in foo File "<stdin>", line 5, in foo [Previous line repeated 2 more times] File "<stdin>", line 3, in foo RuntimeError
Question: Suppose I want to compute variance and/or standard deviation with non-default ddof in a groupby context, I can do: df.groupby("a")["b"].var(ddof=2)  If I want that to happen together with other aggregations, I can use: df.groupby("a").agg(b_var = ("b", "var"), c_sum = ("c", "sum"))  My understanding is that to be able to have non default ddof I should create a custom aggregation. Here what I got so far: def var(ddof: int = 1) -> dd.Aggregation: import dask.dataframe as dd return dd.Aggregation( name="var", chunk=lambda s: (s.count(), s.sum(), (s.pow(2)).sum()), agg=lambda count, sum_, sum_sq: (count.sum(), sum_.sum(), sum_sq.sum()), finalize=lambda count, sum_, sum_sq: (sum_sq - (sum_ ** 2 / count)) / (count - ddof), )  Yet, I encounter a RuntimeError: df.groupby("a").agg({"b": var(2)})  RuntimeError('Failed to generate metadata for DecomposableGroupbyAggregation(frame=df, arg={‘b’: <dask.dataframe.groupby.Aggregation object at 0x7fdfb8469910>} What am I missing? Is there a better way to achieve this? Replacing s.pow(2) with s**2 also results in an error. Full script: import dask.dataframe as dd data = { "a": [1, 1, 1, 1, 2, 2, 2], "b": range(7), "c": range(10, 3, -1), } df = dd.from_dict(data, 2) def var(ddof: int = 1) -> dd.Aggregation: import dask.dataframe as dd return dd.Aggregation( name="var", chunk=lambda s: (s.count(), s.sum(), (s.pow(2)).sum()), agg=lambda count, sum_, sum_sq: (count.sum(), sum_.sum(), sum_sq.sum()), finalize=lambda count, sum_, sum_sq: (sum_sq - (sum_ ** 2 / count)) / (count - ddof), ) df.groupby("a").agg(b_var = ("b", "var"), c_sum = ("c", "sum")) # <- no issue df.groupby("a").agg(b_var = ("b", var(2)), c_sum = ("c", "sum")) # <- RuntimeError 

Answer: As answered in Dask Discourse Forum, I don't think your custom Aggregation implementation is correct. However, a simpler solution can be applied: import dask.dataframe as dd import functools data = { "a": [1, 1, 1, 1, 2, 2, 2], "b": range(7), "c": range(10, 3, -1), } df = dd.from_dict(data, 2) var_ddof_2 = functools.partial(dd.groupby.DataFrameGroupBy.var, ddof=2) df.groupby("a").agg(b_var = ("b", var_ddof_2), c_sum = ("c", "sum"))
Question: Say I have import duckdb rel = duckdb.sql('select * from values (1, 4), (1, 5), (2, 6) df(a, b)') rel  Out[3]: ┌───────┬───────┐ │ a │ b │ │ int32 │ int32 │ ├───────┼───────┤ │ 1 │ 4 │ │ 1 │ 5 │ │ 2 │ 6 │ └───────┴───────┘  I can group by a and find the mean of 'b' by doing: rel.aggregate( [duckdb.FunctionExpression('mean', duckdb.ColumnExpression('b'))], group_expr='a', )  ┌─────────┐ │ mean(b) │ │ double │ ├─────────┤ │ 4.5 │ │ 6.0 │ └─────────┘  which works wonderfully Is there a similar way to create a "n_unique" aggregation? I'm looking for something like rel.aggregate( [duckdb.FunctionExpression('count_distinct', duckdb.ColumnExpression('b'))], group_expr='a', )  but that doesn't exist. Is there something that does?

Answer: updated. I couldn't find proper way of doing count distinct, but you could use combination of array_agg() and array_unique() functions: rel.aggregate( [duckdb.FunctionExpression( 'array_unique', duckdb.FunctionExpression( 'array_agg', duckdb.ColumnExpression('b') ) )], group_expr='a', )  ┌────────────────────────────┐ │ array_unique(array_agg(b)) │ │ uint64 │ ├────────────────────────────┤ │ 1 │ │ 2 │ └────────────────────────────┘  old. you can pre-select distinct a and b columns? ( rel.select(*[duckdb.ColumnExpression('a'), duckdb.ColumnExpression('b')]) .distinct() .aggregate( [duckdb.FunctionExpression('count', duckdb.ColumnExpression('b'))], group_expr='a', ) )
Question: I want to use an expression dictionary to perform calculations for a new column. I have this Polars dataframe: df=pl.DataFrame( "col1": ["a", "b", "a"], "x": [1,2,3], "y": [2,2,5] )  And I have an expression dictionary: expr_dict = { "a": pl.col("x") * pl.col("y"), "b": pl.col("x"), }  I want to create a column where each value is calculated based on a key in in another column, but I do not know how. I want to hhave result like this: >>> df.with_columns(r=pl.col("col1").apply(lambda x: expr_dict[X]) >>> shape: (3, 3) ┌──────┬─────┬─────┬─────┐ │ col1 ┆ x ┆ y ┆ r │ │ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ i64 ┆ i64 ┆ i64 │ ╞══════╪═════╪═════╪═════╡ │ a ┆ 1 ┆ 2 ┆ 2 │ │ b ┆ 2 ┆ 2 ┆ 4 │ │ a ┆ 3 ┆ 5 ┆ 15 │ └──────┴─────┴─────┴─────┘  Is this possible?

Answer:  pl.when() for conditional expression. pl.coalesce() to combine conditional expressions together.  df.with_columns( r = pl.coalesce( pl.when(pl.col.col1 == k).then(v) for k, v in expr_dict.items() ) )  shape: (3, 4) ┌──────┬─────┬─────┬─────┐ │ col1 ┆ x ┆ y ┆ r │ │ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ i64 ┆ i64 ┆ i64 │ ╞══════╪═════╪═════╪═════╡ │ a ┆ 1 ┆ 2 ┆ 2 │ │ b ┆ 2 ┆ 2 ┆ 2 │ │ a ┆ 3 ┆ 5 ┆ 15 │ └──────┴─────┴─────┴─────┘
Question: I am looking for a way to vectorize the following code, # Let cube have shape (N, M, M) sub_arrays = np.empty(len(cube), 3, 3) row_start = ... # Shape (N,) and are integers in range [0, M-2] row_end = ... # Shape (N,) and are integers in range [1, M-1] col_start = ... # Shape (N,) and are integers in range [0, M-2] col_end = ... # Shape (N,) and are integers in range [1, M-1] # extract sub arrays from cube and put them in sub_arrays for i in range(len(cube)): # Note that the below is extracting a (3, 3) sub array from cube sub_arrays[i] = cube[i, row_start[i]:row_end[i], col_start[i]:col_end[i]]  Instead of the loop, I would like to do something like, sub_arrays = cube[:, row_start:row_end, col_start:col_end]  But this throws the exception, TypeError: only integer scalar arrays can be converted to a scalar index  Is there instead some valid way to vectorize the loop?

Answer: I believe this question is a duplicate of the one about Slicing along axis with varying indices. However, since it may not be obvious, I think it's okay to provide the answer in a new context with a somewhat different approach. From what I can see, you want to extract data from the cube using a sliding window of a fixed size (3×3 in this case), applied to a separate slice along the first axis with varying shifts within the slices. In contrast to the previously mentioned approach using as_strided, let's use sliding_window_view this time. As a result, we get two additional axes for row_start and col_start, followed by the window dimensions. Note that row_end and col_end appear as if they are equal to the corresponding starting points increased by a fixed square window side, which is 3 in this case: from numpy.lib.stride_tricks import sliding_window_view cube_view = sliding_window_view(cube, window_shape=(3, 3), axis=(1, 2)) output = cube_view[range(cube.shape[0]), row_start, col_start].copy()  That's all. But to be sure, let's compare the output with the original code, using test data: import numpy as np from numpy.random import randint from numpy.lib.stride_tricks import sliding_window_view n, m, w = 100, 10, 3 # w - square window size row_start = randint(m-w+1, size=n) col_start = randint(m-w+1, size=n) # Test cube cube = np.arange(n*m*m).reshape(n, m, m) # Data to compare with sub_arrays = np.empty((n, w, w), dtype=cube.dtype) for i in range(cube.shape[0]): sub_arrays[i] = cube[i, row_start[i]:row_start[i]+w, col_start[i]:col_start[i]+w] # Subarrays from the sliding window view cube_view = sliding_window_view(cube, window_shape=(w, w), axis=(1, 2)) output = cube_view[range(cube.shape[0]), row_start, col_start].copy() # No exceptions should occur at this step assert np.equal(output, sub_arrays).all()
Question: I am using pyproject.toml to build a package. I use setuptools_scm to automatically determine the version number. I use python version 3.11.2, setuptools 66.1.1 and setuptools-scm 8.1.0. Here are the relevant parts of pyproject.toml # For a discussion on single-sourcing the version, see # https://packaging.python.org/guides/single-sourcing-package-version/ dynamic = ["version"] [tool.setuptools_scm] # can be empty if no extra settings are needed, presence enables setuptools-scm  I build the project with python3 -m build  When I run the build command, I see ERROR setuptools_scm._file_finders.git listing git files failed - pretending there aren't any  What I've Tried:  There is a .git directory at the root of my project. It's readable by all users. Git is installed and accessible from my PATH. I've committed changes to ensure there's Git history available.  How can I fix this error? Are there additional configurations or checks I should perform to ensure setuptools_scm can correctly interact with Git for version determination? Reproducible example cd /tmp/ mkdir setuptools_scm_example cd setuptools_scm_example git init touch .gitignore git add . git commit -m "Initial commit"  Add the following to pyproject.toml [build-system] requires = ["setuptools>=61.0", "setuptools_scm>=7.0"] build-backend = "setuptools.build_meta" [project] name = "example_package" dynamic = ["version"] [tool.setuptools_scm] # No additional configuration needed, but can add if needed  Create and build a python package mkdir -p example_package touch example_package/__init__.py echo "print('Hello from example package')" > example_package/__init__.py python3 -m build  I see the error ERROR setuptools_scm._file_finders.git listing git files failed - pretending there aren't any 

Answer: python3 -m build builds in 2 phases: 1st it builds sdist and then it builds wheel from the sdist in an isolated environment where there is no .git directory. It doesn't matter because at the wheel building phase version is already set in sdist and build gets the version from sdist, not from setuptools_scm. In short: you may safely ignore the error. Reference: https://github.com/pypa/setuptools-scm/issues/997 . Found in https://github.com/pypa/setuptools-scm/issues?q=is%3Aissue+setuptools_scm._file_finders.git Another approach to try: prevent build isolation, install build dependencies into the current environment and build sdist and wheel explicitly: python3 -m pip install build setuptools-scm python3 -m build --no-isolation --sdist --wheel
Question: I need to extract a substring which is between two other substrings. But I would like to make the border substrings optional - if no substrings found then the whole string should be extracted. patt = r"(?:bc)?(.*?)(?:ef)?" a = re.sub(patt, r"\1", "bcdef") # d - as expected a = re.sub(patt, r"\1", "abcdefg") # adg - as expected # I'd like to get `d` only without `a` and `g` # Trying to remove `a`: patt = r".*(?:bc)?(.*?)(?:ef)?" a = re.sub(patt, r"\1", "bcdef") # empty !!! a = re.sub(patt, r"\1", "abcdef") # empty !!! # make non-greedy patt = r".*?(?:bc)?(.*?)(?:ef)?" a = re.sub(patt, r"\1", "bcdef") # d - as expected a = re.sub(patt, r"\1", "abcdef") # `ad` instead of `d` - `a` was not captured # make `a` non-captured patt = r"(?:.*?)(?:bc)?(.*?)(?:ef)?" a = re.sub(patt, r"\1", "abcdef") # ad !!! `a` still not captured  I also tried to use re.search without any success. How can I extract d only (a substring between optional substrings bc and ef) from abcdefg? The same pattern should return hij when applied to hij.

Answer: By making the bc and ef patterns optional, you'll get into situations where the one is matched, while the other is not. Yet, you'd need both of them or neither. The requirement that you need the whole input to match when these delimiters are not present really overcomplicates it. Realise that if there is no match, sub will not alter the input, and so that would actually achieve the desired result. In other words, don't make these delimiter patterns optional -- make them mandatory. When there is a match, you'll want to replace all of the input with the captured group. This means you should also match what follows ef, so it gets replaced (removed) too. Bringing all that together, you could use: patt = r".*?bc(.*?)ef.*"  Be aware that this will only match the first occurrence of the bc...ef pattern. If the input string has more occurrences of those, the sub call will only return the first delimited text.