eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: What I have is two Pandas dataframes of coordinates in xyz-format. One of these contains points that should be masked in the other one, but the values are slightly offset from each other, meaning a direct match with drop_duplicates is not possible. My idea was to round the values to the nearest significant number, but this also does not always work, since if some values are rounded to different numbers, they won't match and won't be removed. For example, if one point lies at x = 149 and another at x = 151, rounding them to the nearest hundred gives different values. My code looks something like this: import pandas as pd import numpy as np df_test_1 = pd.DataFrame(np.array([[123, 449, 756.102], [406, 523, 543.089], [140, 856, 657.24], [151, 242, 124.42]]), columns = ['x', 'y', 'z']) df_test_2 = pd.DataFrame(np.array([[123, 451, 756.099], [404, 521, 543.090], [139, 859, 657.23], [633, 176, 875.76]]), columns = ['x', 'y', 'z']) df_test_3 = pd.concat([df_test_1, df_test_2]) df_test_3['xr'] = df_test_3.x.round(-2) df_test_3['yr'] = df_test_3.y.round(-2) df_test_3['zr'] = df_test_3.z.round(1) df_test_3 = df_test_3.drop_duplicates(subset=['xr', 'yr', 'zr'], keep=False)  What I want is to remove duplicates if the columns 'xr' and 'yr' are duplicates +-100 and 'zr' duplicates +-0.1. For example, if two coordinates are rounded to (100, 300, 756.2) and (200, 400, 756.1), they should be considered duplicates and should be removed. Any ideas are appreciated, thanks!

Answer: You can numpy broadcasting: # Convert to numpy vals1 = df_test_1.values vals2 = df_test_2.values # Remove from df_test_1 arr1 = np.abs(vals1 - vals2[:, None]) msk1 = ~np.any(np.all(arr1 < [100, 100, 0.1], axis=2), axis=1) # Remove from df_test_2 arr2 = np.abs(vals2 - vals1[:, None]) msk2 = ~np.any(np.all(arr1 < [100, 100, 0.1], axis=2), axis=1) out = pd.concat([df_test_1[msk1], df_test_2[msk2]], ignore_index=True)  Output: >>> out x y z 0 151.0 242.0 124.42 1 633.0 176.0 875.76  Comment of @James  This removes left vs right and right vs left, but not duplicates within left vs left or right vs right.  In this case: df_test_3 = pd.concat([df_test_1, df_test_2]) arr = df_test_3.values msk = np.abs(arr - arr[:, None]) < [100, 100, 0.1] out = df_test_3[np.sum(np.all(msk, axis=2), axis=1) == 1] print(out) # Output x y z 3 151.0 242.0 124.42 3 633.0 176.0 875.76
Question: I have an Excel file (which can optionally be loaded into a database, and into an array of arrays of course) with values such as:     A B C     NULL NULL zxy   xyz xzy NULL   xyz xzy xyy   yzy yyx yxy   NULL NULL xyx   xyz NULL yxx     and so on. There are thousands of values. Is there any known algorithm to come up with all possible combinations of rows where values are not "NULL"? For example for the table above the result would be:     A B C Number of occurrences     NULL NULL * 2   * * NULL 1   * * * 2   * NULL * 1     I feel like it is a typical task, but cannot find the algorithm anywhere. Would appreciate your help a lot.

Answer: If you are tempted to use pandas : #pip install pandas import pandas as pd df = pd.read_excel("file.xlsx") out = ( df.replace(".+", "*", regex=True).fillna("NULL") .groupby(list(df), group_keys=False, sort=False) .size().reset_index(name="Number of occurrences") )  Output : print(out) A B C Number of occurrences 0 NULL NULL * 2 1 * * NULL 1 2 * * * 2 3 * NULL * 1
Question: I am trying to write a Python regex pattern that will allow me to capture words in a given text that have letters separated by the same symbol or space. For example, in the text "This is s u p e r and s.u.p.e.r and s👌u👌p👌e👌r and s!u.p!e.r", my goal is to extract the words "s u p e r", "s.u.p.e.r", and s👌u👌p👌e👌r. However, I want to exclude "s!u.p!e.r" because it does not have the same consistent separating symbol within the word. I'm currently using the following: x="This is s u p e r and s.u.p.e.r and s👌u👌p👌e👌r and s!u.p!e.r" pattern = r"(?:\b\w[^\w\d]){2,}" re.findall(pattern, x) ['s u p e r ', 's.u.p.e.r ', 's👌u👌p👌e👌r ', 's!u.p!e.']  I'm just curious if it's possible to exclude the cases that do not have the same symbol.

Answer: You may consider using pattern = r"(?<!\S)\w(?=(\W))(?:\1\w)+(?!\S)" results = [m.group() for m in re.finditer(pattern, x)]  See the Python demo and the regex demo. import re x="This is s u p e r and s.u.p.e.r and s👌u👌p👌e👌r and s!u.p!e.r" pattern = r"(?<!\S)\w(?=(\W))(?:\1\w)+(?!\S)" print([m.group() for m in re.finditer(pattern, x)]) # => ['s u p e r', 's.u.p.e.r', 's👌u👌p👌e👌r']  Pattern details  (?<!\S) - left-hand whitespace boundary \w - a word char (?=(\W)) - a positive lookahead that requires the next char to e a non-word char capturing it into Group 1 (\1) (?:\1\w)+ - one or more repetitions of the same char as captured in Group 1 and then a single word char (?!\S) - right-hand whitespace boundary
Question: I am trying to find 9 raise to power 19 using numpy. I am using numpy 1.24.3 This is the code I am trying: import numpy as np np.long(9**19)  This is the error I am getting: AttributeError: module 'numpy' has no attribute 'long' 

Answer: Sadly, numpy.long was deprecated in numpy 1.20 and it is removed in numpy 1.24 If you wan the result you have to try numpy.longlong import numpy as np np.longlong(9**19) #output 1350851717672992089  https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
Question: Calculate the most right non-zero digit of n factorial efficiently I want to calculate the right most digit of a given number's factorial and print it. What I've done so far is: import math n = int(input()) fact = math.factorial(n) print(str(fact).rstrip('0')[-1])  but I still get time limits and I look for faster solutions. It's worth noting that I must use python to solve this problem. Also, I shall point out that n is from 1 to 65536, the time limit is 0.5 seconds and I have 256 megabytes of memory.

Answer: There is a neat recursive formula you can use: let D(n) be the last non-zero digit in n!  If n<10, use a lookup table If the second last digit of n is odd, D(n) = 4 * D(n//5) * D(unit digit of n) If the second last digit of n is even, D(n) = 6 * D(n//5) * D(Unit digit of n)  See this math stackexchange post for a proof. Translating it into code: def last_nonzero_factorial_digit(n): lookup = [1, 1, 2, 6, 4, 2, 2, 4, 2, 8] if n < 10: return lookup[n] if ((n // 10) % 10) % 2 == 0: return (6 * last_nonzero_factorial_digit(n // 5) * lookup[n % 10]) % 10 else: return (4 * last_nonzero_factorial_digit(n // 5) * lookup[n % 10]) % 10  On my laptop, this version runs ~14,000 times faster on a 5-digit number.
Question: Python setuptools allows you to specify optional dependencies, but does it allow you to do something in the inverse? For example, let's say I have a list of dependencies in my_package like below:  numpy pandas  So if I installed the package with pip install my_package, it would also install these two dependencies. However, for certain use cases, a user may not need pandas. So I would want to do something like pip install my_package[~pandas] or something like that to instruct pip to not install pandas. Is this something that is currently supported?

Answer: It is not currently supported - extras are strictly additive. It has been proposed several times, but the discussions never seem to get anywhere. The latest discussion is here: Proposal - expanding optional dependencies to support opt-out of recommended/default installables. As a workaround, you can use: pip install --no-deps my_package  But this would exclude every dependency, including pandas. You'd have to find and install the other dependencies manually.
Question: In this code that uses zlib to encode some data, but with level=0 so it's not actually compressed: import zlib print('zlib.ZLIB_VERSION', zlib.ZLIB_VERSION) total = 0 print('Total 1', total) compress_obj = zlib.compressobj(level=0, memLevel=9, wbits=-zlib.MAX_WBITS) total += len(compress_obj.compress(b'-' * 1000000)) print('Total 2', total) total += len(compress_obj.flush()) print('Total 3', total)  Python 3.9.12 outputs zlib.ZLIB_VERSION 1.2.12 Total 1 0 Total 2 983068 Total 3 1000080  but Python 3.10.6 (and Python 3.11.0) outputs zlib.ZLIB_VERSION 1.2.13 Total 1 0 Total 2 1000080 Total 3 1000085  so both a different final size, and a different size along the way. Why? And how can I get them to be identical? (I'm writing a library where I would prefer identical behaviour between Python versions)

Answer: zlib 1.2.12 and 1.2.13 behave identically in this regard. The Python library must be making different deflate() calls with different amounts of data, and possibly introducing a flush in the later version. You can look in the Python source code to find out. You should be able to force identical output if you feed smaller amounts of data to .compress() each time, e.g. less than 64K-1, and use .flush() after each. The output will be larger, but should be identical across versions. A quick look turned up this commit, which is likely the culprit.
Question: When using typer to parse CLI arguments, I get very verbose and colorful error messages. How can I get a normal Python traceback? See screenshot for an example traceback (just the first few lines) for illustration of the verbose style: ❯ python scripts/add_priors.py ╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮ │ /Users/corneliusromer/code/nextclade_data_workflows/sars-cov-2/scripts/add_priors.py:26 in main │ │ │ │ 23 │ import polars as pl │ │ 24 │ │ │ 25 │ priors = ( │ │ ❱ 26 │ │ pl.scan_ndjson(ndjson, infer_schema_length=10000) │ │ 27 │ │ .select( │ │ 28 │ │ │ [ │ │ 29 │ │ │ │ pl.col("nearestNodes"), │ │ │ │ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │ │ │ json = <module 'json' from │ │ │ │ 

Answer: You can disable it on a one-off basis by setting the environment variable _TYPER_STANDARD_TRACEBACK=1. Disabling rich exceptions is possible by passing the kwarg pretty_exceptions_enable=False when initializing typer: import typer app = typer.Typer(pretty_exceptions_enable=False) @app.command() def main(): raise Exception("test") if __name__ == "__main__": app()  See the documentation for more options
Question: I have two instances of a class that compete in a simulation where they try to shoot at each other. The class contains a position variable and a target variable. The second instance's target variable references the first instance's position object, and vice-versa. I have a chicken-and-egg problem when creating the two instances, and when trying to bind the references after instantiating, they don't update properly This is a simplified version of my code: class Thing(): def __init__(self, position, target): self.position = position self.target = target def move(self): self.position += 10 ## thing1 = Thing(position = 0, target = thing2.position) # Ideally this line would work... thing1 = Thing(position = 0, target = 0) thing2 = Thing(position = 100, target = thing1.position) print(thing1.target) thing1.target = thing2.position print(thing1.target) thing2.move() print(thing1.target)  The output I get is 0,100,100, and the output I want is 0,100,110.

Answer: I think there are two parts to your question: How to I get my references to another object's position to stay synced up, and how do you initialize objects that reference each other in a cycle. For the first part, I'd suggest a slightly simpler approach that the other answers: Don't reference the target position directly, target the Thing. You can get the position via self.target.position (or the equivalent) whenever you need it. For the second part, you need some way to set up the reference cycle. The simplest approach is to start the way you have so far, initializing one object without a reference to its target, and then passing a reference to this first object to the second object. Then in another step, give a reference to the second object to the first. You're kind of doing this amid your print calls where you do thing1.target = thing2.position, but because you're referencing the position directly, you don't see updates. I'd solve both problems like this: class Thing(): def __init__(self, position, target=None): # target is now optional self.position = position self.target = target def move(self): self.position += 10 thing1 = Thing(0) # no target passed, so it defaults to None (for now) thing2 = Thing(100, thing1) # initialize thing 2 to immediately target thing1 thing1.target = thing2 # update the target of thing1, now that thing2 exists print(thing1.target.position) # get thing2's position via thing1's target reference thing2.move() print(thing1.target.position)
Question: Using Pydantic, how can I specify an attribute that has an input type different from its actual type? For example I have a systems field that contains a list of systems (so a list of strings) and the user can provide this systems list as a comma separated string (e.g. "system1,system2"); then I use a validator to split this string into a list of strings. The code below is doing that and it's working but the type hinting is wrong as the systems field is actually a list of strings, not a string; the validator is splitting the original string into a list of strings. How can I fix this? import typing from pydantic import BaseSettings, Field, validator class Config(BaseSettings): systems: str = Field([], description="list of systems as a comma separated list (e.g. 'sys1,sys2')") @validator("systems") def set_systems(cls, v) -> typing.List[str]: if v == "": return [] systems = list(filter(None, v.split(","))) return systems if __name__ == "__main__": c = Config(**{"systems": "foo,bar"}) print(c) 

Answer: Always annotate model fields with the types you actually want in your schema! If you want the field systems to be a list of strings, then annotate it accordingly. A comma-separated string is the exception after all. To allow it, use a mode='before' validator to intercept that string before the default field validators get to it (and raise an error). Then you can split it and return the list as you wish: from pydantic import BaseSettings, Field, field_validator class Config(BaseSettings): systems: list[str] = Field(default_factory=list) @field_validator("systems", mode="before") @classmethod def split_comma_separated(cls, v: object) -> object: if isinstance(v, str): v = v.strip() return [] if v == "" else v.split(",") return v if __name__ == "__main__": print(Config.parse_obj({"systems": "foo,bar"})) print(Config.parse_obj({"systems": ""})) print(Config())  Output: systems=['foo', 'bar'] systems=[] systems=[]
Question: I am working with Polars and need to drop columns that contain only null values during my data preprocessing. However, I am having trouble using the Lazy API to accomplish this. For instance, given the table below, how can I drop column "a" using Polars' Lazy API? df = pl.DataFrame( { "a": [None, None, None, None], "b": [1, 2, None, 1], "c": [1, None, None, 1], } ) df  shape: (4, 3) ┌──────┬──────┬──────┐ │ a ┆ b ┆ c │ │ --- ┆ --- ┆ --- │ │ f64 ┆ i64 ┆ i64 │ ╞══════╪══════╪══════╡ │ null ┆ 1 ┆ 1 │ │ null ┆ 2 ┆ null │ │ null ┆ null ┆ null │ │ null ┆ 1 ┆ 1 │ └──────┴──────┴──────┘  I am aware of Issue #1613 and the solution of filtering columns where all values are null, but this is not Lazy API. FYI, # filter columns where all values are null df[:, [not (s.null_count() == df.height) for s in df]]  I am also aware of the drop_nulls function in Polars, which can only drop all rows that contain null values, unlike the dropna function in Pandas that can take two arguments, axis and how. Can someone provide an example of how to drop columns with all null values in Polars using the Lazy API?

Answer: You can't, at least not in the way you want. polars doesn't know enough about the lazyframe to tell which columns are only nulls until you collect. That means you need a collect in order to get the columns you want and then another one to materialize the columns you wanted. Let's turn your df=df.lazy() Step 1: (df.select(pl.all().is_null().all()) .unpivot() .filter(pl.col('value')==False) .select('variable') .collect() .to_series() .to_list())  Those are your columns that have no nulls so now you wrap it in its own select Step 2: (df.select( df.select(pl.all().is_null().all()) .unpivot() .filter(pl.col('value')==False) .select('variable') .collect() .to_series() .to_list()) .collect())
Question: So i have a polars column/series that is strings of digits. s = pl.Series("a", ["111","123","101"]) s shape: (3,) Series: 'a' [str] [ "111" "123" "101" ]  I would like to convert each string into a list of integers. I have found a working solution but i am not sure if it is optimal. s.str.split("").list.eval(pl.element().str.to_integer(base=10)) shape: (3,) Series: 'a' [list[i32]] [ [1, 1, 1] [1, 2, 3] [1, 0, 1] ]  This seems to be working but id like to know if there are better ways to do this or any of the individual steps.

Answer: Update: .str.split("") no longers inserts leading/trailing empty strings in the result.  https://github.com/pola-rs/polars/pull/15922  So you can just .cast() the resulting list directly. s.str.split("").cast(pl.List(pl.Int64))  shape: (3,) Series: 'a' [list[i64]] [ [1, 1, 1] [1, 2, 3] [1, 0, 1] ]
Question: I'm trying to return list of operations and getting error @router.get("/") async def get_specific_operations(operation_type: str, session: AsyncSession = Depends(get_async_session)): query = select(operation).where(operation.c.type == operation_type) result = await session.execute(query) return result.all()  Error: ValueError: [TypeError('cannot convert dictionary update sequence element #0 to a sequence'), TypeError('vars() argument must have __dict__ attribute')] 

Answer: For some simple cases, if you want to return a list of Pydantic models, just use response_model:  response_model receives the same type you would declare for a Pydantic model field, so, it can be a Pydantic model, but it can also be, e.g. a list of Pydantic models, like List[Item].  Just like @sergei klinov mentioned. For other cases, such as returning a JSONResponse that contains the query results from SQLAlchemy (sqlmodel): e.g., I used the below code (sqlmodel) to return custom data to the frontend for datatables: results = session.exec(statement).all() head= ["headerName1", "headerName2", "headerName3"] data = { "head": [{"title": column} for column in head], "rows": [list(result) for result in results] }  in the above snippet code, in order to convert the SQLAlchemy Sequence object (results), we can use a List Comprehension (or a for loop) and convert the type to list using list(result). For cases such as dynamic Pydantic models that are defined at runtime (e.g. within the endpoint), or if you only want to return certain columns: e.g., In my case (sqlmodel), I only want to select certain columns (fields) and return them with some manipulation.  results = session.exec(statement).all() if results: data = [] for r in results: data.append({ "full_name": f"{r.firstname} {r.lastname}", "value": r.value, "x": r.x, "y": r.y, } ) return JSONResponse(content=data, status_code=200)  You can access the object's attributes and append to the list where you want to return. Note: I am using sqlmodel, as @Eugene said, replace .all() to .scalars().all(), see the difference here in the sqlmodel official documentation
Question: I need to make docker of my server, but it works only with cuda, how can i add it in my Dockerfile? FROM python:3.10 ENV FLASK_RUN_PORT=5000 RUN sudo nvidia-ctk runtime configure # Here COPY . /app WORKDIR /app RUN pip install --no-cache-dir -r requirements.txt EXPOSE 5000 CMD ["python", "server.py"]  I try to do it bellow, but it doesn't work, please, help

Answer: You can start with a CUDA Docker image and then install Python, for example: FROM nvidia/cuda:12.1.1-runtime-ubuntu20.04 # Install Python RUN apt-get update && \ apt-get install -y python3-pip python3-dev && \ rm -rf /var/lib/apt/lists/*  Note: User @chronoclast has suggested additionally installing python-is-python3 to fix the broken symlink to the default Python, in which case the Python installation step would instead be: RUN apt-get update && \ apt-get install -y python3-pip python3-dev python-is-python3 && \ rm -rf /var/lib/apt/lists/*
Question: Django 4.1.9 requires_system_checks issue with manage.py - is this a bug or not we are upgrading our wagtail 4.2.2 app to from django 3.1.9 to django 4.1.9 getting the error TypeError: requires_system_checks must be a list or tuple. when running python manage.py runserver_plus 0.0.0.0:8020 --keep-meta-shutdown  Is this a bug and / or can i be worked around Could I monkey patch BaseCommand.init() to set requires_system_checks = []

Answer: In 4.1, support for setting a boolean to requires_system_checks on a management command was dropped (release notes). You'll need to check your django "app" dependencies to see which define management commands, and which could be affected. Here's some examples, but the list is not exhaustive:  graphene-django fixed in 3.0.0 (technically, fixed in 3.0.0b8) django-extensions (noted by @DavidU) fixed in 3.2.1
Question: I tried using the below code, but the OpenAI API doesn't have the AuthenticationError method in the library. How can I effectively handle such error. import openai # Set up your OpenAI credentials openai.api_key = 'YOUR_API_KEY' try: # Perform OpenAI API request response = openai.some_function() # Replace with the appropriate OpenAI API function # Process the response # ... except openai.AuthenticationError: # Handle the AuthenticationError print("Authentication error: Invalid API key or insufficient permissions.") # Perform any necessary actions, such as displaying an error message or exiting the program 

Answer: Error handling with the OpenAI Python SDK v1.0.0 or newer • If you don't want to handle error types individually: import os from openai import OpenAI, OpenAIError client = OpenAI() OpenAI.api_key = os.getenv('OPENAI_API_KEY') try: # Make your OpenAI API request here response = client.completions.create( model="gpt-3.5-turbo-instruct", prompt="Say this is a test" ) print(response) except OpenAIError as e: # Handle all OpenAI API errors print(f"Error: {e}")  • If you want to handle error types individually: Note: Because there are a lot of classes for error handling, it might not be so elegant to import them individually. Instead, use import openai and all classes for error handling will be imported automatically. But the code is a bit different now. import os import openai # Import openai from openai import OpenAI # But don't import OpenAIError client = OpenAI() OpenAI.api_key = os.getenv('OPENAI_API_KEY') try: # Make your OpenAI API request here response = client.completions.create( model="gpt-3.5-turbo-instruct", prompt="Say this is a test" ) print(response) except openai.BadRequestError as e: # Don't forget to add openai # Handle error 400 print(f"Error 400: {e}") except openai.AuthenticationError as e: # Don't forget to add openai # Handle error 401 print(f"Error 401: {e}") except openai.PermissionDeniedError as e: # Don't forget to add openai # Handle error 403 print(f"Error 403: {e}") except openai.NotFoundError as e: # Don't forget to add openai # Handle error 404 print(f"Error 404: {e}") except openai.UnprocessableEntityError as e: # Don't forget to add openai # Handle error 422 print(f"Error 422: {e}") except openai.RateLimitError as e: # Don't forget to add openai # Handle error 429 print(f"Error 429: {e}") except openai.InternalServerError as e: # Don't forget to add openai # Handle error >=500 print(f"Error >=500: {e}") except openai.APIConnectionError as e: # Don't forget to add openai # Handle API connection error print(f"API connection error: {e}")  See the official OpenAI GitHub Python repository.  Error handling with the OpenAI Python SDK v0.28.0 Your code isn't correct. Change this... except openai.AuthenticationError  ...to this. except openai.error.AuthenticationError