eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a wide dataframe, and I'm applying some custom logic to some columns to generate a new column. This works, and returns a dataframe with a single column with my desired values. How can I get this as a new column in the original dataframe? I tried various forms of .with_columns but none did the trick; and without row identifiers I don't feel at ease doing a concatenation. Any ideas? I'm trying to solve for a generic UDF, not one that I plan to express as a polars expression. df = pl.DataFrame({ "foo": [16, 28, 0 ], "bar": [None, 4,17 ], "yat": [41, 174,15 ], "tar": [None, 4,0 ], }) def udf(row: tuple[float])->str: '''This code is illustrative - meant to be a row-wise UDF''' return ' + '.join([f'{x}x{"^"+str(i) if i>0 else ""}' for i,x in enumerate(row) if x!=0 and x is not None]) df.select('foo','bar', 'yat', 'tar').map_rows(udf).rename({'map':'poly'})  shape: (3, 1) ┌────────────────────────────┐ │ poly │ │ --- │ │ str │ ╞════════════════════════════╡ │ 16x + 41x^2 │ │ 28x + 4x^1 + 174x^2 + 4x^3 │ │ 17x^1 + 15x^2 │ └────────────────────────────┘  My desired output is shape: (3, 5) ┌─────┬──────┬─────┬──────┬────────────────────────────┐ │ foo ┆ bar ┆ yat ┆ tar ┆ poly │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ i64 ┆ i64 ┆ str │ ╞═════╪══════╪═════╪══════╪════════════════════════════╡ │ 16 ┆ null ┆ 41 ┆ null ┆ 16x + 41x^2 │ │ 28 ┆ 4 ┆ 174 ┆ 4 ┆ 28x + 4x^1 + 174x^2 + 4x^3 │ │ 0 ┆ 17 ┆ 15 ┆ 0 ┆ 17x^1 + 15x^2 │ └─────┴──────┴─────┴──────┴────────────────────────────┘  In pandas you'd do this as: df.assign(poly = lambda x: x.apply(udf, axis=1)) SOLVED: (df.with_columns( pl.struct('foo','bar','yat','tar') .map_elements(lambda x: udf(x.values())) .alias('poly')) )  Or (df.with_columns( pl.struct(pl.all()) .map_elements(lambda x: udf(x.values())) .alias('poly')) ) 

Answer: If you want to apply a function over multiple columns you need to pack them into a struct type. This packing is free, but is needed to suffice the expression rules, that every expressions input only consist of a single datatype. E.g. an expression is Fn(Expr) -> Expr. Below shows an example of using map_elements to compute the horizontal sum and the more idiomatic way to compute a horizontal sum. df = pl.DataFrame({ "foo": [1, 2], "bar": [.1, .2], }) def mysum(row: tuple[float])->float: '''This code is illustrative - meant to be a row-wise UDF''' return sum(row) df.with_columns( # horizontal sum with a custom UDF pl.struct("foo", "bar").map_elements(lambda x: mysum((x["foo"], x["bar"]))).alias("foo+bar (slow)"), # idiomatic way to do a horizontal sum pl.sum_horizontal("foo", "bar").alias("foo+bar (fast)") )  Folds If you want to do more complicated horizontal aggregations, but want to keep the code fast (as using a python function in map_elements is not), you can use fold. Below I show how to compute a horizontal sum with a fold. df.with_columns( pl.fold( acc=0, function=lambda a, b: a + b, exprs=pl.all() ).alias("foo+bar") )
Question: I have an array of strings arr in which I want to search for elements and get the index of element. Numpy has a method where to search element and return index in a tuple form. arr = numpy.array(["string1","string2","string3"]) print(numpy.where(arr == "string1")  It prints: (array([0], dtype=int64),)  But I only want the index number 0. I tried this: i = numpy.where(arr == "string1") print("idx = {}".format(i[0]))  which has output: i = [0]  Is there any way to get the index number without using replace or slicing method?

Answer: TL;DR Use: try: i = numpy.where(arr == "string1")[0][0] except IndexError: # handle the case where "string1" was not found in arr  or indices = list(numpy.where(arr == "string1")[0])  Details Finding elements in NumPy arrays is not intuitive the first time you try to do it. Let's decompose the operation: >>> arr = numpy.array(["string1","string2","string3"]) >>> arr == "string1" array([ True, False, False])  Notice how just doing arr == "string1" is already doing the search: it's returning an array of booleans of the same shape as arr telling us where the condition is true. Then, you're using numpy.where which, when used with only one parameter (the condition), returns where its input is non-zero. With booleans, that means non false. >>> numpy.where(numpy.array([ True, False, False])) (array([0], dtype=int64),) >>> numpy.where(arr == "string1") (array([0], dtype=int64),)  It's not quite clear to my where this gives you a tuple of arrays for a 1-D input, but when you use this syntax with a 2-d input, it makes more sense. In any case, what you're getting here is a tuple containing a list of indices where the condition matches. Notice it has to be a list, because you might have multiple matches. For your code, you want numpy.where(arr == "string1")[0][0], because you know "string1" occurs in the list, but the inner list may also contain zero or more than one values, depending on how many times the string is found. >>> arr2 = numpy.array(["string1","string2","string3","string1", "string3"]) >>> numpy.where(arr2 == "foo") (array([], dtype=int64),) >>> numpy.where(arr2 == "string3") (array([2, 4], dtype=int64),)  So when you want to use these indices, you should simply treat numpy.where(arr == "string1")[0] as a list (it's really a 1-D array, though) and continue from there. Now, just using numpy.where(arr == "some string")[0][0] is risky, because it will throw an IndexError exception if the string is not found in arr. If you really want to do that, do it in a try/except block. If you need the list of indices as a Python list, you can do this: indices = list(numpy.where(arr == "string1")[0])
Question: I am trying to type a a function which receives a Series from typing import Any from pandas import Series def func(w: Series[Any], v: Series[Any]) -> int:  However I got the error  TypeError: 'type' object is not subscriptable  What I am doing wrong?

Answer: Adding quotes around the type will fix this issue. In the code you provided: from typing import Any from pandas import Series def func(w: 'Series[Any]', v: 'Series[Any]') -> int: # your code pass  Modern editors will identify the type and raise appropriate warnings or errors if violated. You are not doing anything "wrong", since this won't have raised any warnings or error for builtin types i.e. list, dict, set, etc. However, using quotes is the safer choice since it will work in all these cases and yours as well.
Question: How to create a mocked mongo db object to test my software using python? I tried https://pytest-mock-resources.readthedocs.io/en/latest/mongo.html but got error. First, i tried the code below: def insert_into_customer(mongodb_connection): collection = mongodb_connection['customer'] to_insert = {"name": "John", "address": "Highway 37"} collection.insert_one(to_insert) from pytest_mock_resources import create_mongo_fixture mongo = create_mongo_fixture() def test_insert_into_customer(mongo): insert_into_customer(mongo) collection = mongo['customer'] returned = collection.find_one() assert returned == {"name": "John", "address": "Highway 37"} test_insert_into_customer(mongo)  I got the error below: Traceback (most recent call last): File "/home/ehasan-karbasian/Desktop/NationalEliteFoundation/serp_matcher/src/mock_mongo.py", line 19, in <module> test_insert_into_customer(mongo) File "/home/ehasan-karbasian/Desktop/NationalEliteFoundation/serp_matcher/src/mock_mongo.py", line 11, in test_insert_into_customer insert_into_customer(mongo) File "/home/ehasan-karbasian/Desktop/NationalEliteFoundation/serp_matcher/src/mock_mongo.py", line 2, in insert_into_customer collection = mongodb_connection['customer'] TypeError: 'function' object is not subscriptable  And then i tried the code: def insert_into_customer(mongodb_connection): collection = mongodb_connection['customer'] to_insert = {"name": "John", "address": "Highway 37"} collection.insert_one(to_insert) from pymongo import MongoClient from pytest_mock_resources import create_mongo_fixture mongo = create_mongo_fixture() def test_create_custom_connection(mongo): client = MongoClient(**mongo.pmr_credentials.as_mongo_kwargs()) db = client[mongo.config["database"]] collection = db["customers"] to_insert = [ {"name": "John"}, {"name": "Viola"}, ] collection.insert_many(to_insert) result = collection.find().sort("name") returned = [row for row in result] assert returned == to_insert test_create_custom_connection(mongo)  and got the error: Traceback (most recent call last): File "/home/ehasan-karbasian/Desktop/NationalEliteFoundation/serp_matcher/src/mock_mongo.py", line 30, in <module> test_create_custom_connection(mongo) File "/home/ehasan-karbasian/Desktop/NationalEliteFoundation/serp_matcher/src/mock_mongo.py", line 14, in test_create_custom_connection client = MongoClient(**mongo.pmr_credentials.as_mongo_kwargs()) AttributeError: 'function' object has no attribute 'pmr_credentials'  It looks like mongo is a function not a MongoClient. How can i use the library pytest_mock_resources to mock mongo? Is there a better library to mock monodb to test with pytest?

Answer: import mongomock client = mongomock.MongoClient() database = client.__getattr__(database_name) collection = database.__getattr__(collection_name)  You can define as many as client, database and collections you need at the same time.
Question: I used the BeeWare environment to create a simple MahJong game (find & click pairs to remove them) using Python (with Toga as layout tool) for Android. Now I would like to have some buttons give a "click sound" when pressed: Anyone have a helping hint (or even working example)?

Answer: If you're using Briefcase 0.3.10 or newer (which uses Chaquopy to support Python on Android), then you could use the Chaquopy Python API to play audio files using SoundPool. For example, the code from this answer could be written in Python as follows: from android.media import AudioManager, SoundPool from os.path import dirname, join soundPool = SoundPool(5, AudioManager.STREAM_MUSIC, 0) soundId = soundPool.load(join(dirname(__file__), "filename.mp3"), 1) soundPool.play(soundId, 1, 1, 0, 0, 1)  This will play the file "filename.mp3" from the same directory as the Python source file.
Question: I'm trying to find a way to get the UPC plus the 5 number supplement barcode using Python or NodeJS. So far I've tried using pyzbar in Python via this code.  img = Image.open(requests.get(url, stream=True).raw) img = ImageOps.grayscale(img) results = decode(img)  That only returns the main UPC code. Not the supplemental code. This is an example of an image I'm trying to read this from. 

Answer: Install the dependencies (on mac + python): $ brew install zbar $ pip install pyzbar $ pip install opencv-python  You can define the symbols to ZBar. Please try this python code: import cv2 import pyzbar.pyzbar as pyzbar from pyzbar.wrapper import ZBarSymbol img = cv2.imread('barcode.png') gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY) barcodes = pyzbar.decode(gray, [ZBarSymbol.EAN13, ZBarSymbol.EAN5]) for barcode in barcodes: print('{}: {}'.format(barcode.type, barcode.data))  I've tested it with this image:  The result is: EAN5: b'07900' EAN13: b'9791234567896'  (BTW, for your image this code scans just the EAN13 part. Maybe you have to use a bigger image, or increase the contrast)
Question: I need to send a premium emoji on the user's account using Pyrogram. I tried to send with send_message() a list of MessageEntityCustomEmoji and MessageEntity. The first one gave the error 'MessageEntityCustomEmoji' object has no attribute '_client', and the second one sent a message without an emoji. How do I send them without errors?

Answer: After lots of research and suffering, the answer was: ... my_emoji_str = "<emoji id=5310129635848103696>✅</emoji> And this is custom emoji in the text" await app.send_message(message.chat.id, my_emoji_str)  This is basically a text formatting, here it is in HTML ParseMode of Pyrogram, but Pyrogram supports both Markdown and HTML formatting simultaneously out-of-the-box, and you also don't need to specify it in the parse_mode argument. And the id is the custom_emoji_id that you can find in the message object if you don't know it yet. To know what to look for: ... "text": "✅ Hello", "entities": [ { "_": "MessageEntity", "type": "MessageEntityType.CUSTOM_EMOJI", "offset": 0, "length": 1, "custom_emoji_id": 5310129635848103696 # <-- here you go } ], ...  And.. That's it!
Question: I have an app that uses a FastAPI backend and a Next.js frontend. In development and on production with stable origins, I am able to use the CORSMiddleware with no issues. However, I have deployed the Next.js frontend with Vercel, and want to take advantage of the automatic Preview deployments that Vercel makes with each git commit to allow for staging-type qualitative testing and sanity checks. I'm running into CORS issues on the Preview deployments: since each Preview deployment uses an auto-generated URL of the pattern: <project-name>-<unique-hash>-<scope-slug>.vercel.app, I can't add them directly to the allow_origins argument of the CORSMiddleware. Instead I am trying to add the pattern to the allow_origin_regex argument. I am very new to regex, but was able to figure out a pattern that I've tested to work in REPL. However, because I'm having issues, I've switched to use an ultra-permissive regex of '.*' just to get anything to work but that has failed also. main.py (relevant portions) from fastapi import FastAPI from fastapi.middleware.cors import CORSMiddleware app = FastAPI() origins = [ "http://localhost", "http://localhost:8080", "http://localhost:3000", "https://my-project-name.vercel.app" ] app.add_middleware( CORSMiddleware, allow_origins=origins, allow_origin_regex=".*", allow_credentials=True, allow_methods=["*"], allow_headers=["*"], )  I've looked at the FastAPI/Starlette cors.py file to see how it ingests and uses the origin regex and don't see where the problem would be. I've tested the same methods in REPL with no issues. I'm at a loss as to the next avenue to investigate in order to resolve this issue. Any assistance or pointers or "hey dummy you forgot this" comments are welcome.

Answer: Whenever a new deployment is created, Vercel will automatically generate a unique URL that is publicly available, and which is composed of the following pieces: <project-name>-<unique-hash>-<scope-slug>.vercel.app  To allow requests from any Vercel deployment, use: allow_origin_regex='https://.*\.vercel\.app'  To allow requests from a specific Vercel project, use: allow_origin_regex='https://<project-name>-.*\.vercel\.app'  for instance: allow_origin_regex = 'https://my-site-.*\.vercel\.app'  The example below is based on how FastAPI/Starlette's CORSMiddleware works internally (see implementation here). The example shows that using the above regex, a match for an origin such as https://my-site-xadvghg2z-acme.vercel.app is found. import re origin = 'https://my-site-xadvghg2z-acme.vercel.app' allow_origin_regex = 'https://my-site-.*\.vercel\.app' compiled_allow_origin_regex = re.compile(allow_origin_regex) if (compiled_allow_origin_regex is not None and compiled_allow_origin_regex.fullmatch(origin)): print('Math found') else: print('No match found')  Please make sure to specify the correct protocol (e.g., http, https) and port(80, 8000, 3000) in allow_origin_regex.
Question: I am having troubles with my conda installation on a cluster. It seems that i can't deactivate any of my environments. it goes so far, that i have to close my terminal because it froze. I am working on multiple server with a common home directory, so i can access the same conda installation from different servers. Interestingly, I can deactivate the conda envs on a different server, but not on my main server, which i work most of the time. I use conda 22.9.0 I there a way to solve this without reinstalling everything? (Can I reinstall conda itself without loosing the already set environments?) Thanks 

Answer: thanks to @merv I could easily solve the problem. Running conda init bash and restarting the terminal seems to clean something and now it works as well. I can see again my prompt and the environments are closing again. yeroslaviz@hpcl8001:~$ conda init bash modified /fs/home/yeroslaviz/miniconda3/condabin/conda modified /fs/home/yeroslaviz/miniconda3/bin/conda modified /fs/home/yeroslaviz/miniconda3/bin/conda-env no change /fs/home/yeroslaviz/miniconda3/bin/activate no change /fs/home/yeroslaviz/miniconda3/bin/deactivate no change /fs/home/yeroslaviz/miniconda3/etc/profile.d/conda.sh no change /fs/home/yeroslaviz/miniconda3/etc/fish/conf.d/conda.fish no change /fs/home/yeroslaviz/miniconda3/shell/condabin/Conda.psm1 no change /fs/home/yeroslaviz/miniconda3/shell/condabin/conda-hook.ps1 no change /fs/home/yeroslaviz/miniconda3/lib/python3.7/site-packages/xontrib/conda.xsh no change /fs/home/yeroslaviz/miniconda3/etc/profile.d/conda.csh no change /fs/home/yeroslaviz/.bashrc ==> For changes to take effect, close and re-open your current shell. <== yeroslaviz@hpcl8001:~$ exit yeroslaviz@hpcl8001:~$ conda activate base (base) yeroslaviz@hpcl8001:~$ (base) yeroslaviz@hpcl8001:~$ conda deactivate yeroslaviz@hpcl8001:~$
Question: As mentioned on the title, CMake seems to be broken after upgrading to MacOS 13.0. Trying to install something that requires Cmakes takes unusually long then the following pop-up shows up. “CMake” is damaged and can’t be opened. You should move it to the Trash. This file was downloaded on an unknown date. # this txt is grey and smaller font Pop-up Options 1. Move to Trash 2. Cancel  Steps to reproduce Err  Cloned EasyOCR 1.1 git clone ...  Made Python venv 2.0. cd EasyOCR/ 2.1. python3 -m venv venv 2.2. source venv/bin/activate 2.3. venv info python --version && pip --version # output Python 3.10.6 pip 22.3 from ... # path to venv dir   pip install -r requirements.txt   # requirements.txt content torch torchvision>=0.5 opencv-python-headless<=4.5.4.60 scipy numpy Pillow scikit-image python-bidi PyYAML Shapely pyclipper ninja   After a while, the aforementioned pop-up, shows up. Clicking on either option will result in the following error.  Building wheels for collected packages: opencv-python-headless Building wheel for opencv-python-headless (pyproject.toml) ... error error: subprocess-exited-with-error × Building wheel for opencv-python-headless (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [9 lines of output] File "/private/var/folders/5h/36chnb_s3b5fpqmqgt_7cz_m0000gn/T/pip-build-env-ea_5u80v/overlay/lib/python3.10/site-packages/skbuild/setuptools_wrap.py", line 613, in setup cmkr = cmaker.CMaker(cmake_executable) File "/private/var/folders/5h/36chnb_s3b5fpqmqgt_7cz_m0000gn/T/pip-build-env-ea_5u80v/overlay/lib/python3.10/site-packages/skbuild/cmaker.py", line 141, in __init__ self.cmake_version = get_cmake_version(self.cmake_executable) File "/private/var/folders/5h/36chnb_s3b5fpqmqgt_7cz_m0000gn/T/pip-build-env-ea_5u80v/overlay/lib/python3.10/site-packages/skbuild/cmaker.py", line 95, in get_cmake_version raise SKBuildError( Traceback (most recent call last): Problem with the CMake installation, aborting build. CMake executable is cmake [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for opencv-python-headless Failed to build opencv-python-headless ERROR: Could not build wheels for opencv-python-headless, which is required to install pyproject.toml-based projects  Any thoughts on how to fix or get around this? This is the first time I see this pop up.

Answer: The pip package is broken on macOS 13 prior to CMake 3.24.2 due to improper code signing. You should upgrade CMake in your virtual environment by running: $ python -m pip install -U pip setuptools wheel $ python -m pip install -U 'cmake>=3.24.2'  As CMake is extremely backwards compatible, it should be safe. You can also add cmake>=3.24.2 to your requirements.txt.
Question: I would like to add overloaded type annotations to an existing API that has semantics something like this: def f(x: Tuple[int, ...]) -> Union[int, List[int]]: if len(x) == 1: return x[0] return list(x)  The argument is a tuple, and the return type is either int or List[int] depending on whether the tuple has length 1. To encode the output type's dependence on the input type, I've tried the following: from typing import overload, List, Tuple @overload def f(x: Tuple[int]) -> int: ... @overload def f(x: Tuple[int, ...]) -> List[int]: ... def f(x: Tuple[int, ...]) -> Union[int, List[int]]: if len(x) == 1: return x[0] return list(x)  But when type-checked with mypy, this gives the following error: script.py:4: error: Overloaded function signatures 1 and 2 overlap with incompatible return types Found 1 error in 1 file (checked 1 source file)  To address this I would need the second overload to have a type that indicates "tuples of any length except 1", but it seems that this sort of exclusive type doesn't exist (PEP 484: exclusive type for type hint) What would be the best way to define overloaded annotations for this function?

Answer: The "overlap with incompatible return types" error message can sometimes be a little bit of a lint, as opposed to something broken. If you # type: ignore, mypy will still do what you want. (I'm a maintainer of mypy)
Question: Given that I know the, Max, Min and Average of sample (I don't have access to the sample itself). I would like to write a generic function to generate a sample with the same characteristics. From this answer I gather that this is no simple task since many distribuitions can be found with the same characteristics. max, min, average = [411, 1, 20.98]  I'm trying to use scipy.norm but unsuccessfully. I can't seem to understand if I can pass the arguments mentioned above or if they are just returned values from an already generated function. I'm pretty new to python stats so this might be something quite easy to solve.

Answer: Triangular distribution should perform your desired task since it takes three parameters (min, mode, max) as inputs that match your criteria. You can think of other distributions such as standard, uniform, and so on; however, all of their input parameters either lack or partially take one of the three input parameters mentioned by you above. If I were in your position, I would consider triangular distribution because even partial exclusion of a single parameter can incur information loss. import numpy as np import matplotlib.pyplot as plt h = plt.hist(np.random.triangular(-3, 0, 8, 100000), bins=200, density=True) plt.show()  Numpy - Triangular Distribution
Question: Consider the following data set: How can I generate the expected value, ExpectedGroup such that the same value exists when True, but changes and increments by 1, when we run into a False statement in case_id. df = pd.DataFrame([ ['A', 'P', 'O', 2, np.nan], ['A', 'O', 'O', 5, 1], ['A', 'O', 'O', 10, 1], ['A', 'O', 'P', 4, np.nan], ['A', 'P', 'P', 300, np.nan], ['A', 'P', 'O', 2, np.nan], ['A', 'O', 'O', 5, 2], ['A', 'O', 'O', 10, 2], ['A', 'O', 'P', 4, np.nan], ['A', 'P', 'P', 300, np.nan], ['B', 'P', 'O', 2, np.nan], ['B', 'O', 'O', 5, 3], ['B', 'O', 'O', 10, 3], ['B', 'O', 'P', 4, np.nan], ['B', 'P', 'P', 300, np.nan], ], columns = ['ID', 'FromState', 'ToState', 'Hours', 'ExpectedGroup'])  # create boolean mask df['case_id'] = ( (df.FromState == 'O') & (df.ToState == 'O') ) 0 False 1 True 2 True 3 False 4 False 5 False 6 True 7 True 8 False 9 False 10 False 11 True 12 True 13 False 14 False Name: case_id, dtype: boo  # but how to get incrementing groups? np.where(df['case_id'] != False, df['case_id'].cumsum(), np.nan) 

Answer: You can use diff to select only the first item of each stretch of True: df['ExpectedGroup'] = (df['case_id'].diff() &df['case_id'] ).cumsum().where(df['case_id'])  If you don't want the intermediate column: s = (df.FromState == 'O') & (df.ToState == 'O') # or # s = df[['FromState', 'ToState']].eq('O').all(axis=1) df['ExpectedGroup'] = (s.diff()&s).cumsum().where(s) # or # df.loc[s, 'ExpectedGroup'] = (s.diff()&s).cumsum()  Output:  ID FromState ToState Hours ExpectedGroup case_id 0 A P O 2 NaN False 1 A O O 5 1.0 True 2 A O O 10 1.0 True 3 A O P 4 NaN False 4 A P P 300 NaN False 5 B P O 2 NaN False 6 B O O 5 2.0 True 7 B O O 10 2.0 True 8 B O P 4 NaN False 9 B P P 300 NaN False