eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I already saw one old post regarding this topic - An error while trying to execute tests on python 3.10 with pytest, I am having the same problem, Python 3.10.4 and pytest 7.1.2, when I start command: $ pipenv run pytest I get an error: $ pipenv run pytest ============================= test session starts ============================= platform win32 -- Python 3.10.4, pytest-4.0.0, py-1.7.0, pluggy-0.8.0 rootdir: **DIR**, inifile: collected 0 items / 1 errors =================================== ERRORS ==================================== ____________________ ERROR collecting test/test_person.py _____________________ <frozen importlib._bootstrap>:939: in _find_spec ??? E AttributeError: 'AssertionRewritingHook' object has no attribute 'find_spec' During handling of the above exception, another exception occurred: **LOCAL_PATH**\.virtualenvs\iamdb-2ZawZA6J\lib\site-packages\py\_path\local.py:668: in pyimport __import__(modname) <frozen importlib._bootstrap>:1027: in _find_and_load ??? <frozen importlib._bootstrap>:1002: in _find_and_load_unlocked ??? <frozen importlib._bootstrap>:941: in _find_spec ??? <frozen importlib._bootstrap>:915: in _find_spec_legacy ??? **LOCAL_PATH**\.virtualenvs\iamdb-2ZawZA6J\lib\site-packages\_pytest\assertion\rewrite.py:162: in find_module source_stat, co = _rewrite_test(self.config, fn_pypath) **LOCAL_PATH**\.virtualenvs\iamdb-2ZawZA6J\lib\site-packages\_pytest\assertion\rewrite.py:412: in _rewrite_test co = compile(tree, fn.strpath, "exec", dont_inherit=True) E TypeError: required field "lineno" missing from alias !!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!! =========================== 1 error in 0.16 seconds ===========================  Anyone has a solution?

Answer: As per comment from Marco Bonelli, pytest had no correct version. So command: pipenv update pytest  fixed the issue.
Question: from statistics import mean import pandas as pd df = pd.DataFrame(columns=['A', 'B', 'C']) df["A"] = [1, 2, 3, 4, 4, 5, 6] df["B"] = ["Feb", "Feb", "Feb", "May", "May", "May", "May"] df["C"] = [10, 20, 30, 40, 30, 50, 60] df1 = df.groupby(["A","B"]).agg(mean_err=("C", mean)).reset_index() df1["threshold"] = df1["A"] * df1["mean_err"]  Instead of the last line of code, how can I do it as in Pyspark .withColumn() ?  This code wont work. I would like to create new column by using output of operation on the fly similarly like we do in Pyspark withColumn method. Can anybody have any idea how to do this?

Answer: Option 1: DataFrame.eval (df.groupby(['A', 'B'], as_index=False) .agg(mean_err=('C', 'mean')) .eval('threshold = A * mean_err'))  Option 2: DataFrame.assign (df.groupby(['A', 'B'], as_index=False) .agg(mean_err=('C', 'mean')) .assign(threshold=lambda x: x['A'] * x['mean_err']))    A B mean_err threshold 0 1 Feb 10.0 10.0 1 2 Feb 20.0 40.0 2 3 Feb 30.0 90.0 3 4 May 35.0 140.0 4 5 May 50.0 250.0 5 6 May 60.0 360.0
Question: I have the following variables: a = [1, 2, 3] b = "de" # <-- not a (usual) list ! c = 5 # <-- not a list ! d = [4, 5, 23, 11, 5] e = ["dg", "kuku"]  Now I want to concat all a, b, c, d, e to one list: [1, 2, 3, "de", 5, 4, 5, 23, 11, 5, "dg", "kuku"]  I have tried itertools.chain but it didn't work. Please advise how can I make the concatenation?

Answer: chain works with iterables. What you mean is: concatenate these lists and raw values. I see two steps: def ensure_list(x): if isinstance(x, list): return x return [x] lists = map(ensure_list, (a, b, c, d, e)) concatenated = list(itertools.chain.from_iterable(lists))
Question: Is it possible to save numpy arrays on disk in boolean format where it takes only 1 bit per element? This answer suggests to use packbits and unpackbits, however from the documentation, it seems that this may not support memory mapping. Is there a way to store 1bit arays on disk with memmap support? Reason for memmap requirement: I'm training my neural network on a database of full HD (1920x1080) images, but I crop out randomly a 256x256 patch for each iteration. Since reading the full image is time consuming, I use memmap to read the only the required patch. Now, I want to use a binary mask along with my images and hence this requirement.

Answer: numpy does not support 1 bit per element arrays, I doubt memmap has such a feature. However, there is a simple workaround using packbits. Since your case is not bitwise random access, you can read it as 1 byte per element array. # A binary mask represented as an 1 byte per element array. full_size_mask = np.random.randint(0, 2, size=[1920, 1080], dtype=np.uint8) # Pack mask vertically. packed_mask = np.packbits(full_size_mask, axis=0) # Save as a memmap compatible file. buffer = np.memmap("./temp.bin", mode='w+', dtype=packed_mask.dtype, shape=packed_mask.shape) buffer[:] = packed_mask buffer.flush() del buffer # Open as a memmap file. packed_mask = np.memmap("./temp.bin", mode='r', dtype=packed_mask.dtype, shape=packed_mask.shape) # Rect where you want to crop. top = 555 left = 777 width = 256 height = 256 # Read the area containing the rect. packed_top = top // 8 packed_bottom = (top + height) // 8 + 1 packed_patch = packed_mask[packed_top:packed_bottom, left:left + width] # Unpack and crop the actual area. patch_top = top - packed_top * 8 patch_mask = np.unpackbits(packed_patch, axis=0)[patch_top:patch_top + height] # Check that the mask is cropped from the correct area. print(np.all(patch_mask == full_size_mask[top:top + height, left:left + width]))  Note that this solution could (and likely will) read extra bits. To be specific, 7 bits maximum at both ends. In your case, it will be 7x2x256 bits, but this is only about 5% of the patch, so I believe it is negligible. By the way, this is not an answer to your question, but when you are dealing with binary masks such as labels for image segmentation, compressing with zip may drastically reduce the file size. It is possible that it could be reduced to less than 8 KB per image (not per patch). You might want to consider this option as well.
Question: The following fragment of code comes from my github repository found here. It opens a binary file, and extracts the text within <header> tags. These are the crucial lines: gbxfile = open(filename,'rb') gbx_data = gbxfile.read() gbx_header = b'(<header)((?s).*)(</header>)' header_intermediate = re.findall(gbx_header, gbx_data)  The script works BUT it receives the following Deprecation Warning: DeprecationWarning: Flags not at the start of the expression b'(<header)((?s).*)(</' (truncated) header_intermediate = re.findall(gbx_header, gbx_data)  What is the correct use of the regular expression in gbx_header, so that this warning is not displayed?

Answer: You can check the Python bug tacker Issue 39394, the warning was introduced in Python 3.6. The point is that the Python re now does not allow using inline modifiers not at the start of string. In Python 2.x, you can use your pattern without any problem and warnings as (?s) is silently applied to the whole regular expression under the hood. Since it is not always an expected behavior, the Python developers decided to produce a warning. Note you can use inline modifier groups in Python re now, see restrict 1 word as case sensitive and other as case insensitive in python regex | (pipe). So, the solutions are  Putting (?s) (or any other inline modifier) at the start of the pattern: (?s)(<header)(.*)(</header>) Using the re option, re.S / re.DOTALL instead of (?s), re.I / re.IGNORECASE instead of (?i), etc. Using workarounds (instead of ., use [\w\W]/[\d\D]/[\s\S] if you do not want to use (?s) or re.S/re.DOTALL).
Question: I'm running some python scripts on some Linux clusters using SGE or SLURM. I already have my conda environment set up properly using the login node. I have been writing something like source ~/.bashrc module purge #Kill all active modules conda init bash conda deactivate conda deactivate conda activate my_env python my_script.py  to activate the environment properly. (I have done a lot of work to figure this out) However, I just found some example codes like /anaconda3/envs/my_env/bin/python my_script.py  seems to do the same thing without the need for tedious deactivation and activation. Are they actually doing the same thing? If so, which would be the better practice?

Answer: Programmatic execution with an environment is usually better done through the conda run subcommand. E.g., my_slurm_script.sh #!/bin/bash -l conda run -n my_env python my_script.py  Read the conda run --help for details.
Question: In pandas, we can just assign directly: import pandas as pd df = pd.DataFrame({"a": [1, 2]}) # add a single value df["b"] = 3 # add an existing Series df["c"] = pd.Series([4, 5])   a b c 0 1 3 4 1 2 3 5  Notice that the new numerical Series is not in the original df, it is a result of some computation. How do we do the same thing in polars? import polars as pl df = pl.DataFrame({"a": [1, 2]}) df = df.with_columns(...) # ???? 

Answer: Let's start with this DataFrame: import polars as pl df = pl.DataFrame( { "col1": [1, 2, 3, 4, 5], } )  shape: (5, 1) ┌──────┐ │ col1 │ │ --- │ │ i64 │ ╞══════╡ │ 1 │ │ 2 │ │ 3 │ │ 4 │ │ 5 │ └──────┘  To add a scalar (single value) Use polars.lit. my_scalar = -1 df.with_columns(pl.lit(my_scalar).alias("col_scalar"))  shape: (5, 2) ┌──────┬────────────┐ │ col1 ┆ col_scalar │ │ --- ┆ --- │ │ i64 ┆ i32 │ ╞══════╪════════════╡ │ 1 ┆ -1 │ │ 2 ┆ -1 │ │ 3 ┆ -1 │ │ 4 ┆ -1 │ │ 5 ┆ -1 │ └──────┴────────────┘  You can also choose the datatype of the new column using the dtype keyword. df.with_columns(pl.lit(my_scalar, dtype=pl.Float64).alias("col_scalar_float"))  shape: (5, 2) ┌──────┬──────────────────┐ │ col1 ┆ col_scalar_float │ │ --- ┆ --- │ │ i64 ┆ f64 │ ╞══════╪══════════════════╡ │ 1 ┆ -1.0 │ │ 2 ┆ -1.0 │ │ 3 ┆ -1.0 │ │ 4 ┆ -1.0 │ │ 5 ┆ -1.0 │ └──────┴──────────────────┘  To add a list To add a list of values (perhaps from some external computation), use the polars.Series constructor and provide a name to the Series constructor. my_list = [10, 20, 30, 40, 50] df.with_columns(pl.Series(name="col_list", values=my_list))  shape: (5, 2) ┌──────┬──────────┐ │ col1 ┆ col_list │ │ --- ┆ --- │ │ i64 ┆ i64 │ ╞══════╪══════════╡ │ 1 ┆ 10 │ │ 2 ┆ 20 │ │ 3 ┆ 30 │ │ 4 ┆ 40 │ │ 5 ┆ 50 │ └──────┴──────────┘  You can use the dtype keyword to control the datatype of the new series, if needed. df.with_columns(pl.Series(name="col_list", values=my_list, dtype=pl.Float64))  shape: (5, 2) ┌──────┬──────────┐ │ col1 ┆ col_list │ │ --- ┆ --- │ │ i64 ┆ f64 │ ╞══════╪══════════╡ │ 1 ┆ 10.0 │ │ 2 ┆ 20.0 │ │ 3 ┆ 30.0 │ │ 4 ┆ 40.0 │ │ 5 ┆ 50.0 │ └──────┴──────────┘  To add a Series If you already have a Series, you can just provide a reference to it. my_series = pl.Series(name="my_series_name", values=[10, 20, 30, 40, 50]) df.with_columns(my_series)  shape: (5, 2) ┌──────┬────────────────┐ │ col1 ┆ my_series_name │ │ --- ┆ --- │ │ i64 ┆ i64 │ ╞══════╪════════════════╡ │ 1 ┆ 10 │ │ 2 ┆ 20 │ │ 3 ┆ 30 │ │ 4 ┆ 40 │ │ 5 ┆ 50 │ └──────┴────────────────┘  If your Series does not already have a name, you can provide one using the alias Expression. my_series_no_name = pl.Series(values=[10, 20, 30, 40, 50]) df.with_columns(my_series_no_name.alias('col_no_name'))  shape: (5, 2) ┌──────┬─────────────┐ │ col1 ┆ col_no_name │ │ --- ┆ --- │ │ i64 ┆ i64 │ ╞══════╪═════════════╡ │ 1 ┆ 10 │ │ 2 ┆ 20 │ │ 3 ┆ 30 │ │ 4 ┆ 40 │ │ 5 ┆ 50 │ └──────┴─────────────┘
Question: When I try to run the azure function defined below, I get the following error log The 'my_function' function is in error: The binding name my_function_timer is invalid. Please assign a valid name to the binding.  What is the format of a valid binding name for Azure Function ? Function definition I have two files in my_function directory:  __init__.py contains the python code of the function function.json contains the configuration of the function  Here is the content of those two files __init__.py import azure.functions as func import logging def main(my_function_timer: func.TimerRequest) -> None: logging.info("My function starts") print("hello world") logging.info("My function stops")  function.json { "scriptFile": "__init__.py", "bindings": [ { "name": "my_function_timer", "type": "timerTrigger", "direction": "in", "schedule": "0 0 1 * * *" } ] }  I deploy this function using Azure/functions-action@v1 github action

Answer: I couldn't find anything in the documentation either, but looking at the source code of azure-functions-host(which contains code for the runtime host used by the Azure Functions service), it uses following regex to validate the binding name. ^([a-zA-Z][a-zA-Z0-9]{0,127}|\$return)$  This means that, for a valid binding name  First character must be a letter and can be followed by letters or digits(at most 127 characters). OR  Literal string $return   Since your binding name contains an underscore(_), the above regex does not match which will result in validation error.
Question: I have a function with multiple kwargs with defaults. One of them (in the middle somewhere) is a boolean toggle that controls the return type. I would like to create two overloads for this method with Literal[True/False] but keeping the default value. My idea was the following: from typing import overload, Literal @overload def x(a: int = 5, t: Literal[True] = True, b: int = 5) -> int: ... @overload def x(a: int = 5, t: Literal[False] = False, b: int = 5) -> str: ... def x(a: int = 5, t: bool = True, b: int = 5) -> int | str: if t: return 5 return "asd"  But mypy raises:  error: Overloaded function signatures 1 and 2 overlap with incompatible return types  I assume that is because x() will conflict. But I cannot remove the default = False value in the second overload since it is preceded by arg a with a default. How can I overload this properly such that  x() → int x(t=True) → int x(t=False) → str 

Answer: It is an old problem. The reason is that you specify default value in both branches, so x() is possible in both and return type is undefined. I have the following pattern for such cases: from typing import overload, Literal @overload def x(a: int = ..., t: Literal[True] = True, b: int = ...) -> int: ... @overload def x(a: int = ..., *, t: Literal[False], b: int = ...) -> str: ... @overload def x(a: int, t: Literal[False], b: int = ...) -> str: ... def x(a: int = 5, t: bool = True, b: int = 1) -> int | str: if t: return 5 return "asd"  Why and how? You have to think about ways to call your function. First, you can provide a, then t can be given as kwarg (#2) or arg (#3). You can also leave a default, then t is always a kwarg (#2 again). This is needed to prevent putting arg after kwarg, which is SyntaxError. Overloading on more than one parameter is more difficult, but possible this way too: @overload def f(a: int = ..., b: Literal[True] = ..., c: Literal[True] = ...) -> int: ... @overload def f(a: int = ..., *, b: Literal[False], c: Literal[True] = ...) -> Literal['True']: ... @overload def f(a: int = ..., *, b: Literal[False], c: Literal[False]) -> Literal['False']: ... @overload def f(a: int, b: Literal[False], c: Literal[True] = ...) -> Literal['True']: ... @overload def f(a: int, b: Literal[False], c: Literal[False]) -> Literal['False']: ... def f(a: int = 1, b: bool = True, c: bool = True) -> int | Literal['True', 'False']: return a if b else ('True' if c else 'False') # mypy doesn't like str(c)  You can play with overloading here. Ellipsis (...) in overloaded signatures default values means "Has a default, see implementation for its value". It is no different from the actual value for the type checker, but makes your code saner (default values are defined only in actual signatures and not repeated).
Question: For work I have to use Python 2.7. But when I use the "debug my python file" function in VS Code, I get an error. Even with a simple program, like : print() 

Answer: As rioV8 said in a comment, you have to install a previous version of the Python extension, because in the meanwhile support for Python 2 has been dropped. To install a previous version you have to:  Open the Extensions pane from the bar on the left and find Python Click on the gear icon and select "Install another version" Choose 2021.9.1246542782. After it's finished, restart VS Code.   If you want to understand why you need version 2021.9.1246542782: The component that provides support to the language is Jedi, and the release notes of version 0.17.2 (2020-07-17) say that  This will be the last release that supports Python 2 and Python 3.5. 0.18.0 will be Python 3.6+.  And according to the release notes of the Python extension, the latest version that was based on Jedi 0.17 was 2021.9.3 (20 September 2021), because the following one (2021.10.0, 7 October 2021) says  Phase out Jedi 0.17  Is that all? No, because the selection that VS Code offers when selecting previous versions uses a different numbering scheme. Anyway, the latest one of the v2021.9.* branch is v2021.9.1246542782, which I suppose corresponds to 2021.9.3, so it's the one you need.
Question: I need to search a string and check if it contains numbers in its name. If it does, I want to replace it with nothing. I've started doing something like this but I didn't find a solution for my problem. table = "table1" if any(chr.isdigit() for chr in table) == True: table = table.replace(chr, "_") print(table) # The output should be "table"  Any ideas?

Answer: You could do this in many different ways. Here's how it could be done with the re module: import re table = 'table1' table = re.sub(r'\d+', '', table)
Question: I have a python project in which I have python files and notebooks. I use strict typing in my project but I would like to remove it only on notebooks. I use VScode with setting: "python.analysis.typeCheckingMode": "strict"  I know how to ignore type on a python file:  But it seems it does not work on notebooks:  I get the following type error:  "Type of "y" is partially unknown Type of "y" is "Unknown | None (pylance)"   How can I ignore type checking on notebooks ?

Answer: That is a Pylance error. You can create a pyrightconfig.json file at the root of your workspace and define the files to be exclude-d from analysis or completely ignore-d: { "ignore": [ "**/*.ipynb", ], }  You can even list up specific filenames: { "ignore": [ "notimportant.ipynb", "test.ipynb", ], }   Historical Notes: It initially didn't work for Jupyter Notebooks (.ipynb): https://github.com/microsoft/pylance-release/issues/2135  This happens because pyright doesn't see the file as a "*.ipynb". The file is being preprocessed (to combine all of the cells) in the notebook by the VS Code Python extension, and the resulting combined file is then passed to pyright for analysis. The pylance team is actively working on changing the way this works. I'm going to transfer this bug to the pylance-release repo so it gets the attention it deserves.  That Github issue has since been resolved the fix was deployed as part of pylance 2022.8.51: https://github.com/microsoft/pylance-release/blob/main/CHANGELOG.md#2022851-31-august-2022-prerelease  Notable changes:  ... Bug Fix: Ignoring *.ipynb files does not work (pylance-release#2135)   If it somehow still does not work, check the version of pylance on your VS Code.
Question: I'm using Intel and Apple Silicon chip Mac. When I build dockerfile on Intel Mac, it is success. How ever when I build same file on M1 Mac I get This error.  #10 3.254 ERROR: Could not find a version that satisfies the requirement google-python-cloud-debugger (from versions: none) #10 3.255 ERROR: No matching distribution found for google-python-cloud-debugger ------ executor failed running [/bin/sh -c python -m pip install --upgrade pip && pip install -r requirements.txt]: exit code: 1  This is not only google-python-cloud-debugger. If I remove it, another modules hits same error. How to solve this problem? My requirements.txt and dockerfile is below. gunicorn==19.9.0 selenium==3.141.0 webdriver-manager==3.5.2 google-python-cloud-debugger google-cloud-storage django==3.2 mysqlclient==2.1.0 pysqlite3==0.4.6 django-storages==1.12.3 Pillow==9.0.0 requests==2.27.1  FROM python:3.9-buster # Install manually all the missing libraries RUN apt-get update && apt-get install -y gconf-service libasound2 libatk1.0-0 libcairo2 libcups2 libfontconfig1 libgdk-pixbuf2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libxss1 fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget fonts-takao-* # Install Chrome RUN wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb RUN dpkg -i google-chrome-stable_current_amd64.deb; apt-get -fy install # Install Python dependencies. COPY requirements.txt requirements.txt RUN python -m pip install --upgrade pip && pip install -r requirements.txt # Copy chrome driver COPY drivers /root/.wdm/drivers # Copy local code to the container image. COPY . /app ENV APP_HOME /app/website WORKDIR $APP_HOME # Run the web service on container startup. Here we use the gunicorn CMD exec gunicorn --bind :$PORT --workers 1 --threads 8 website.wsgi 

Answer: I solved this problem! When we build dockerfile on M1 Mac, we need specify platform. So you need to add --platform=linux/x86_64 after FROM on dockerfile. Example FROM --platform=linux/x86_64 python:3.9-buster  or you can also add the option when you build. docker build . --platform=linux/x86_64 -t my_tag  If linux/x86_64 is not good. You can try linux/amd64 it's also efective.
Question: I have the following line of code  end_df['Soma Internet'] = end_df.iloc[:,end_df.columns.get_level_values(1) == 'Internet'].drop('site',axis=1).sum(axis=1)  It basically, filts my multi index df by a specific level 1 column. Drops a few not wanted columns. And does the sum, of all the other ones. I took a glance, at a few of the documentation and other asked questions. But i didnt quite understood what causes the warning, and i also would love to rewrite this code, so i get rid of it.

Answer: Let's try with an example (without data for simplicity): import pandas as pd # Column MultiIndex. idx = pd.MultiIndex(levels=[['Col1', 'Col2', 'Col3'], ['subcol1', 'subcol2']], codes=[[2, 1, 0], [0, 1, 1]]) df = pd.DataFrame(columns=range(len(idx))) df.columns = idx print(df)   Col3 Col2 Col1 subcol1 subcol2 subcol2  Clearly, the column MultiIndex is not sorted. We can check it with: print(df.columns.is_monotonic_increasing)  False  This matters because Pandas performs index lookup and other operations much faster if the index is sorted, because it can use operations that assume the sorted order and are faster. Indeed, if we try to drop a column: df.drop('Col1', axis=1)  PerformanceWarning: dropping on a non-lexsorted multi-index without a level parameter may impact performance. df.drop('Col1', axis=1)  Instead, if we sort the index before dropping, the warning disappears: print(df.sort_index(axis=1)) # Index is now sorted in lexicographical order. Col1 Col2 Col3 subcol2 subcol2 subcol1  # No warning here. df.sort_index(axis=1).drop('Col1', axis=1)  EDIT (see comments): As the warning suggests, this happens when we do not specify the level from which we want to drop the column. This is because to drop the column, pandas has to traverse the whole index (happens here). By specifying it we do not need such traversal: # Also no warning. df.drop('Col1', axis=1, level=0)  However, in general this problem relates more on row indices, as usually column multi-indices are way smaller. But definitely to keep it in mind for larger indices and dataframes. In fact, this is in particular relevant for slicing by index and for lookups. In those cases, you want your index to be sorted for better performance.