eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I want to log the output of my python file into a txt. I also want to still see it in the terminal too. I tried using sys.stdout, but it did not still have terminal output. To log it I open my log file with f = open("log.txt", "r+"), then set sys.stdout = f. When the code ended, I said f.close() to write changes. Here is some code: import sys f = open("files/log.txt", "r+") sys.stdout = f print("You will not see this in terminal") f.close() 

Answer: You can set sys.stdout to a file-like object with a write method that writes to both the original sys.stdout and a given file: import sys class Tee: def __init__(self, file): self.file = file def write(self, text): self.orig_stdout.write(text) self.file.write(text) def start(self): self.orig_stdout = sys.stdout sys.stdout = self def flush(self): self.orig_stdout.flush() def close(self): sys.stdout = self.orig_stdout __enter__ = start def __exit__(self, exc_type, exc_val, exc_tb): self.close()  so that: with open('test.txt', 'w') as file, Tee(file): print('both terminal and file') print('terminal only')  would output to the terminal: both terminal and file terminal only  and output to the file: both terminal and file  If you mean to redirect the output of an entire program to both the terminal and a file, it would be safer to make sure that the file is properly closed upon the program's termination, normal or not, by registering the file closer as an exit handler with atexit.register. On the other hand, it isn't necessary to explicitly restore sys.stdout upon exit, since the OS, if not the interpreter, will always do that for you: import atexit file = open('test.txt', 'w') atexit.register(file.close) Tee(file).start() print('both terminal and file')
Question: I'm aiming to use list comprehension to return values in a list. Specifically, if 'x' is in a list, I want to drop all other values. However, if 'x' is not in the list, I want to return the same values (not return an empty list). list1 = ['d','x','c'] list2 = ['d','b','c'] list1 = [s for s in list1 if s == 'x'] list2 = [s for s in list2 if s == 'x']  List2 would return []. Where I want it to be ['d','b','c'] list2 = [s for s in list2 if s == 'x' else list2]  Returns: list2 = [s for s in list2 if s == 'x' else list2] ^^^^ SyntaxError: invalid syntax 

Answer: This would keep all elements that aren't x list2 = [x for x in list2 if x != 'x']  However, if x is in the list, it'll still return all other elements. So, you'd need two passes to check whether x does exist since list comprehension alone cannot return that information def filter_x(lst): if 'x' in lst: return [x for x in lst if x == 'x'] else: return lst
Question: In numpy, I can do the following: >>> x = np.array([[False, True, False], [True, False, True]]) >>> z0 = np.logical_and.accumulate(x, axis=0) >>> z1 = np.logical_and.accumulate(x, axis=1)  This returns the following: >>> z0 array([[False, True, False], [False, False, False]]) >>> z1 array([[False, False, False], [ True, False, False]])  What is the equivalent of this ufunc operation in pytorch?

Answer: The logical and corresponds to a product in binary terms. You can use cumprod for that: >>> x.cumprod(dim=0).bool() tensor([[False, True, False], [False, False, False]]) >>> x.cumprod(dim=1).bool() tensor([[False, False, False], [ True, False, False]])
Question: I have a dataset like this index Date_Time Pass_ID El 0 3/30/23 05:12:36.36 A 1 1 3/30/23 05:12:38.38 A 2 1 3/30/23 05:12:40.40 A 3 1 3/30/23 05:12:42.42 A 4 1 3/30/23 05:12:44.44 A 4 1 3/30/23 12:12:50.50 B 3 1 3/30/23 12:12:52.52 B 4 1 3/30/23 12:12:54.54 B 5 1 3/30/23 12:12:56.56 B 6 1 3/30/23 12:12:58.58 B 7 1 3/30/23 12:13:00.00 B 8 1 3/30/23 12:13:02.02 B 9 1 3/31/23 20:02:02.02 C 3 1 3/31/23 20:02:05.05 C 4  The Date_Time is pandas datetime object. I'd like to group the records by Pass_ID, and then select out only those unique Pass_IDs that occur between specific hours in the day: for instance, between 10:00 and 13:00 would return B. I don't know how to get groupby and 'between_time' to work in this case... which would seem to be the best way forward. I've also tried using a lambda function after setting the Date_Time as the index, but that didn't work. And using aggregate doesn't seem to allow me to pull out the dt.hour of the Date_Time field. Anyone know how to do this concisely?

Answer: Try: # to datetime if necessary # df["Date_Time"] = pd.to_datetime(df["Date_Time"]) out = df.set_index("Date_Time").between_time("10:00", "13:00")["Pass_ID"].unique() print(out)  Prints: ['B']   OR: If you want to filter whole groups between time 10:00-13:00: out = ( df.groupby("Pass_ID") .filter( lambda x: len(x.set_index("Date_Time").between_time("10:00", "13:00")) == len(x) )["Pass_ID"] .unique() ) print(out)
Question: FastAPI added support for Annotated (and started recommending it) in version 0.95.0. Additionally, FastAPI has a very powerful but intuitive Dependency Injection system (documentation). Moreover, FastAPI support Classes as Dependencies. However, it seams like Annotated cannot be used in class dependencies, but on function dependencies. I use FastAPI version 0.110.1. from __future__ import annotations from typing import Annotated from fastapi import FastAPI, Depends, Query app = FastAPI() class ClassDependency: def __init__(self, name: Annotated[str, Query(description="foo")]): self.name = name async def function_dependency(name: Annotated[str, Query(description="foo")]) -> dict: return {"name": name} @app.get("/") async def search(c: Annotated[ClassDependency, Depends(function_dependency)]) -> dict: return {"c": c.name}  The example above works without errors, but if I replace Depends(function_dependency) with Depends(ClassDependency) an exception is raised with the following message: pydantic.errors.PydanticUndefinedAnnotation: name 'Query' is not defined  Then, if I remove Annotated from the ClassDependency, by replacing the name: Annotated[str, Query(description="foo")] with the name: str, the example works. My question: Can I use Class dependencies and put Annotated to the parameters set in the constructor? Because it seams this is not working. My need: I want to have a class hierarchy for the query params of my api endpoints and provide validation and documentation extras for each of the param.

Answer: Remove the following and it will work. from __future__ import annotations
Question: I'm on an MacBook Pro w/ M1 Pro Chip running macOS Venture 13.0.1 and have Python 3.9.6 installed. When trying to run the following command(s): pip install anonympy  pip3 install anonympy  I get the following output: Collecting anonympy Using cached anonympy-0.3.7.tar.gz (5.8 MB) Installing build dependencies ... done Getting requirements to build wheel ... error error: subprocess-exited-with-error × Getting requirements to build wheel did not run successfully. │ exit code: 1 ╰─> [1 lines of output] error in anonympy setup command: 'python_requires' must be a string containing valid version specifiers; Invalid specifier: '>=3.6*' [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: subprocess-exited-with-error × Getting requirements to build wheel did not run successfully. │ exit code: 1 ╰─> See above for output. note: This error originates from a subprocess, and is likely not a problem with pip.  I tried this on Colab and on a virtual environment running the same python version locally, getting the same build wheel error both times. I have also tried running the following command to no avail. pip(3) install --upgrade setuptools  Is there a problem with the package, or am I missing something?

Answer: It seems it's an error in the package that is not compatible with the last PyPi version. You can install it directly with pip+git : pip install git+https://github.com/ArtLabss/open-data-anonymizer.git Related Github Issue : https://github.com/ArtLabss/open-data-anonymizer/issues/26
Question: This is my DataFrame: import pandas as pd df = pd.DataFrame( { 'a': ['x', 'x', 'x', 'x', 'x', 'y', 'y', 'y', 'y', 'y', 'y', 'y'], 'b': [1, 1, 1, 2, 2, 1, 1, 1, 2, 2, 2, 2], 'c': [9, 8, 11, 13, 14, 3, 104, 106, 11, 100, 70, 7] } )  Expected output: Creating column out:  a b c out 0 x 1 9 NaN 1 x 1 8 NaN 2 x 1 11 NaN 3 x 2 13 found 4 x 2 14 NaN 5 y 1 3 NaN 6 y 1 104 found 7 y 1 106 NaN 8 y 2 11 NaN 9 y 2 100 NaN 10 y 2 70 NaN 11 y 2 7 NaN  The mask is: mask = (df.c > 10)  The process: Grouping is by column a: a) For each group, finding the first row that meets the conditions of the mask. b) For group x this condition only applies when b == 2. That is why row 3 is selected. And this is my attempt. It is getting close but it feels like this is not the way: def func(g): mask = (g.c > 10) g.loc[mask.cumsum().eq(1) & mask, 'out'] = 'found' return g df = df.groupby('a').apply(func) 

Answer: One option with groupby.idxmax: mask = (df['c'] > 10) & (df['a'].ne('x') | df['b'].eq(2)) idx = mask.groupby(df['a']).idxmax() df.loc[idx[mask.loc[idx].values], 'out'] = 'found'  Another with groupby.transform: mask = (df['c'] > 10) & (df['a'].ne('x') | df['b'].eq(2)) df.loc[mask & mask.groupby(df['a']) .transform(lambda m: (~m).shift(fill_value=True) .cummin()), 'out'] = 'found'  Output, with an extra group z that has no match:  a b c out 0 x 1 9 NaN 1 x 1 8 NaN 2 x 1 11 NaN 3 x 2 13 found 4 x 2 14 NaN 5 y 1 3 NaN 6 y 1 104 found 7 y 1 106 NaN 8 y 2 11 NaN 9 y 2 100 NaN 10 y 2 70 NaN 11 y 2 7 NaN 12 z 3 1 NaN 13 z 3 1 NaN  last match to get the last match (instead of the first one), just inver the mask: Example: mask = (df['c'] > 10) & (df['a'].ne('x') | df['b'].eq(2)) mask = mask[::-1] idx = mask.groupby(df['a']).idxmax() df.loc[idx[mask.loc[idx].values], 'out'] = 'found' a b c out 0 x 1 9 NaN 1 x 1 8 NaN 2 x 1 11 NaN 3 x 2 13 NaN 4 x 2 14 found 5 y 1 3 NaN 6 y 1 104 NaN 7 y 1 106 NaN 8 y 2 11 NaN 9 y 2 100 NaN 10 y 2 70 found 11 y 2 7 NaN 12 z 3 1 NaN 13 z 3 1 NaN
Question: I have encountered the issue in one particular stage of my project. Replicate with following: import pandas as pd # Recreated a sample data data = { "FailCodes": ['4301,4090,5003(1)'], } df = pd.DataFrame(data) # Want to replace the '(1)' with 'p1q' print(df.FailCodes.str.replace('(1)','p1q'),'\n') # Not giving expected result # compare to string object method from standard python, which gives desired result print('4301,4090,5003(1)'.replace('(1)','p1q'),'\n') # Can get wanted result with following longer code,but would like explanation why first approach giving the unexpected result. print(df.FailCodes.str.replace('(','p').str.replace(')','q' ))   Unexpected result end like: 430p1q,4090,5003(p1q) Expected result: 4301,4090,5003p1q 

Answer: You probably use older version of Pandas, where default argument is regex=True (and (1) is regular pattern). Put regex=False to .str.replace: print(df.FailCodes.str.replace("(1)", "p1q", regex=False))  Prints: 0 4301,4090,5003p1q Name: FailCodes, dtype: object
Question: When I run the stable diffusion on colab https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/stable_diffusion.ipynb with no modification, it fails on the line from diffusers import StableDiffusionPipeline  The error log is AttributeError: module 'jax.random' has no attribute 'KeyArray' How can I fix this or any clue ? The import should work, the ipynb should run with no error.

Answer: jax.random.KeyArray was deprecated in JAX v0.4.16 and removed in JAX v0.4.24. Given this, it sounds like the HuggingFace stable diffusion code only works JAX v0.4.23 or earlier. You can install JAX v0.4.23 with GPU support like this: pip install "jax[cuda12_pip]==0.4.23" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html  or, if you prefer targeting a local CUDA installation, like this: pip install "jax[cuda12_local]==0.4.23" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html  For more information on GPU installation, see JAX Installation: NVIDIA GPU. From the colab tutorial, update the second segment into: !pip install "jax[cuda12_local]==0.4.23" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html !pip install diffusers==0.11.1 !pip install transformers scipy ftfy accelerate
Question: import numba from typing import List @numba.njit def test(a: List[int]) -> int: return 1 test([i for i in range(2_000_000)])  takes 2s and scales linearly with the size of the list. Wrapping the input argument with numba.typed.List takes even longer. (all the time is spent on the numba.typed.List call. The timings don't get better if the function is called multiple times (while only being defined once), i.e., this is not a matter of compilation time. Is there a way to tell numba to just use the list as is? In my actual application, the raw data comes from an external library that cannot return numpy arrays or numba lists directly, only Python lists. I'm using numba 0.59.1 and Python 3.12 on a 4 core Ubuntu22 laptop with 16GB RAM.

Answer: Numba only operates on typed variables. It needs not only to check the types of all the items but also convert the whole list into a typed list. This implicit conversion can be particularly expensive since CPython lists, a.k.a. reflected lists contain pointers on allocated objects and each objects is reference counted. Typed lists of Numba are homogeneous and they do not contain references but directly the value in it. This is far more efficient and similar to a Numpy array with additional features like resizing.  Is there a way to tell numba to just use the list as is?  AFAIK, no. Reflected lists are not supported any-more in Numba code. Operating on reflected lists is not only inefficient, but also break the type system. The best option is to create directly a typed list. Here is an example: import numba as nb # Quite fast part (less than 0.1 seconds) reflected_lst = [i for i in range(2_000_000)] # Slow part (3 seconds) typed_lst = nb.typed.typedlist.List(reflected_lst) # Very fast part (less than 2 µs) since `lst` is already a typed-list test(typed_lst)  As mentioned by @roganjosh, note the list generation is included in your benchmark, but this only takes a tiny fraction of the execution time (<5% on my machine). Note the conversion process is particularly expensive in Numba (as opposed to Numpy, see the below comments). There is an opened issue on this topic. To quote the developers:  [...] we came to the conclusion that there is probably room for improvement. [...] this problem is all down to implementation details.  As of now, the issue is still opened and there is a patch to improve a bit the performance of the conversion, but it is still rather slow with it.
Question: I am unfortunately having to work with some nested data in a polars dataframe. (I know it is bad practice) Consider data: data = { "positions": [ { "company": { "companyName": "name1" }, }, { "company": { "companyName": "name2" }, }, { "company": { "companyName": "name3" }, } ] }  positions is a column in the dataframe. I have explored the polars python api docs but cannot figure out how to extract out just the companyName fields into a separate list column. I want to achieve the same that this comprehension does:  names = ( [ p["company"]["companyName"] for p in data["positions"] if p.get("company") and p.get("company").get("companyName") ] if data.get("positions") else None )  Note the null checks. I get a sense that I have to use the pl.list.eval function along with pl.element but I am a bit foggy on the api. Before: shape: (3, 1) ┌─────────────┐ │ positions │ │ --- │ │ struct[1] │ ╞═════════════╡ │ {{"name1"}} │ │ {{"name2"}} │ │ {{"name3"}} │ └─────────────┘ After: shape: (3, 1) ┌───────┐ │ names │ │ --- │ │ str │ ╞═══════╡ │ name1 │ │ name2 │ │ name3 │ └───────┘ 

Answer: Structs You can use .struct.field() or .struct[] syntax to extract struct fields.  https://docs.pola.rs/user-guide/expressions/structs/#extracting-individual-values-of-a-struct  df = pl.DataFrame(data) df.with_columns( pl.col("positions").struct["company"].struct["companyName"] )  shape: (3, 2) ┌─────────────┬─────────────┐ │ positions ┆ companyName │ │ --- ┆ --- │ │ struct[1] ┆ str │ ╞═════════════╪═════════════╡ │ {{"name1"}} ┆ name1 │ │ {{"name2"}} ┆ name2 │ │ {{"name3"}} ┆ name3 │ └─────────────┴─────────────┘  Alternatively, you can work at the frame-level and .unnest() the structs into columns. df.unnest("positions").unnest("company")  shape: (3, 1) ┌─────────────┐ │ companyName │ │ --- │ │ str │ ╞═════════════╡ │ name1 │ │ name2 │ │ name3 │ └─────────────┘  List of structs If working with a list of structs you could use the .list.eval() API: df = pl.DataFrame([data]) df.with_columns( pl.col("positions").list.eval( pl.element().struct["company"].struct["companyName"] ) )  shape: (1, 1) ┌─────────────────────────────┐ │ positions │ │ --- │ │ list[str] │ ╞═════════════════════════════╡ │ ["name1", "name2", "name3"] │ └─────────────────────────────┘  Or at the frame-level using .explode() and .unnest() df.explode("positions").unnest("positions").unnest("company")  shape: (3, 1) ┌─────────────┐ │ companyName │ │ --- │ │ str │ ╞═════════════╡ │ name1 │ │ name2 │ │ name3 │ └─────────────┘
Question: I have a dict with item\column name and a df with columns from dict and other columns. How can I add column to df with min value for every item just from columns corresponding from dict? import pandas as pd my_dict={'Item1':['Col1','Col3'], 'Item2':['Col2','Col4'] } df=pd.DataFrame({ 'Col0':['Item1','Item2'], 'Col1':[20,25], 'Col2':[89,15], 'Col3':[26,30], 'Col4':[40,108], 'Col5':[55,2] }) df['min']=?  I tried df['min']=df[df.columns[df.columns.isin(my_dict)]].min(axis=1),  but it didn't work.

Answer: You can use apply with a function that reads the appropriate column names out of the dictionary (returning an empty list if there is no match) and then takes the minimum of the specified columns: my_dict = { 'Item1': ['Col1', 'Col3'], 'Item2': ['Col2', 'Col4'] } df['min'] = df.apply(lambda r:r[my_dict.get(r['Col0'], [])].min(), axis=1)  Output:  Col0 Col1 Col2 Col3 Col4 Col5 min 0 Item1 20 89 26 40 55 20 1 Item2 25 15 30 108 2 15  If it's possible my_dict may contain column names that don't exist in the dataframe, you can check for that in the function. For example: my_dict = { 'Item1': ['Col1', 'Col3'], 'Item2': ['Col4', 'Col6'] } df['min'] = df.apply( lambda r:r[[col for col in my_dict.get(r['Col0'], []) if col in r]].min(), axis=1 )  Output:  Col0 Col1 Col2 Col3 Col4 Col5 min 0 Item1 20 89 26 40 55 20 1 Item2 25 15 30 108 2 108  You can even get the column names if you want: my_dict = { 'Item1': ['Col1', 'Col3'], 'Item2': ['Col2', 'Col4'] } df[['min', 'name']] = df.apply( lambda r:min((r[col], col) for col in my_dict.get(r['Col0'], []) if col in r), axis=1, result_type='expand' )  Output:  Col0 Col1 Col2 Col3 Col4 Col5 min name 0 Item1 20 89 26 40 55 20 Col1 1 Item2 25 15 30 108 2 15 Col2
Question: Context We develop a Python library that contains a function expecting a numlike parameter. We specify this in our signature and make use of python type hints: def cool(value: float | int | List[float | int])  🏳 Problem & Goal During runtime, we noticed it's fine to pass in numpy number types as well, e.g. np.float16(1.2345). So we thought: why not incorporate "numpy number types" into our signature as this would be beneficial for the community that will use our library. However, we don't want numpy as dependency in our project. We'd like to only signify in the method signature that we can take a float, int, a list of them OR any "numpy number type". If the user hasn't installed numpy on their system, they should still be able to use our library and just ignore that they could possibly pass in a "numpy number type" as well. We don't want to depend on numpy as we don't use it in our library (except for allowing their types in our method signature). So why include it in our dependency graph? There's no reason to do so. One dependency less is better. Additional requirements/context  We search for an answer that is compatible with all Python versions >=3.8. (The answer should work with setuptools>=69.0.) The answer should be such that we get proper IntelliSense (Ctrl + Space) when typing cool( in an IDE, e.g. VSCode. This is what our pyproject.toml looks like.  Efforts  We've noticed the option [project.optional-dependencies] for the pyproject.toml file, see here. However, it remains unclear how this optional dependencies declaration helps us in providing optional numpy datatypes in our method signatures. numpy provides the numpy.typing type annotations. Is it somehow possible to only depend on this subpackage? We did search on search engines and found this SO question, however our question is more specific with regards to how we can only use types from another module. We also found this SO question, but despite having "optional" in its title, it's not about optional numpy types. 

Answer: Defer evaluation of annotations, and only import numpy conditionally. from __future__ import annotations import typing as t if t.TYPE_CHECKING: import numpy as np def cool(value: int | np.floating | etc ...): ...  Now the numpy dependency is only necessary when type-checking. See PEP 563 – Postponed Evaluation of Annotations Side-questions..  numpy provides the numpy.typing type annotations. Is it somehow possible to only depend on this subpackage?  No, this is not possible.  We've noticed the option [project.optional-dependencies] for the pyproject.toml file ...  It doesn't really help you much here. It could still be useful if you wanted "extra" dependencies which the user can opt-in for, e.g.: pip install mypkg # install with required dependencies pip install mypkg[typing] # install with extra dependencies such as numpy  Then you could use this to easily install the package along with the soft-deps in your CI, for example.
Question: This code snippet describes a problem I have been having. For some reason rounded_data seems to be rounded, but once passed in np.unique and np.column_stack the result_array seems to be unrounded, meanwhile the rounded_data is still rounded. rounded_data = data_with_target_label.round(decimals=2) unique_values, counts = np.unique(rounded_data, return_counts=True) result_array = np.column_stack((unique_values, counts)) print(rounded_data) print(result_array)  Result: 443392 0.01 443393 0.00 443394 0.00 443395 0.00 443396 0.11 ... 452237 0.04 452238 0.00 452239 0.00 452240 0.00 452241 0.00 Name: values, Length: 8850, dtype: float32 [[0.00000000e+00 4.80000000e+01] [9.99999978e-03 2.10000000e+01] [1.99999996e-02 1.10000000e+01] ... [3.29000015e+01 1.00000000e+00] [3.94099998e+01 1.00000000e+00] 

Answer: this is because your dataframe is in float32 while default number format in numpy is float64. So the number that is rounded in float32 won't be visibly rounded in float64, because number representation is a bit different. Solution is to convert either input array to float64 or the result_array into float 32. Solution 1 Converting numpy array to float32: rounded_data = data_with_target_label.round(decimals=2) unique_values, counts = np.unique(rounded_data, return_counts=True) result_array = np.column_stack((unique_values, counts)) result_array = np.float32(result_array)  Solution 2 Converting input data. For example input is pd.DataFrame (or pd.Series): df = pd.DataFrame({'vals': np.array([0.013242, 3.94099998, 9.99999978, 0.03234, 0.05532, 33.22, 33.44, 55.66])}, dtype = 'float32') rounded_data = df['vals'].astype('float64').round(decimals=2) unique_values, counts = np.unique(rounded_data, return_counts=True) result_array = np.column_stack((unique_values, counts))