eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am trying to convert some pandas DataFrame operations to Polars in Python, but I am running into difficulties, particularly with row-wise operations and element-wise comparisons. Here is the pandas code I am working with: df_a = pd.DataFrame({ "feature1": [1, 2, 3], "feature2": [7, 8, 9], }) df_b = pd.DataFrame({ "feature1": [3, 8, 2], "feature2": [7, 4, 9], }) if selection_mode == 'option1': max_values = df_a.max(axis=1) selected_features = df_a.eq(max_values, axis=0) final_result = selected_features.mul(df_b).sum(axis=1) / selected_features.sum(axis=1) elif selection_mode == 'option2': above_avg = df_a.ge(df_a.mean(axis=1), axis=0) combined_df = above_avg.mul(df_a).mul(df_b) sum_combined = combined_df.sum(axis=1) sum_above_avg = above_avg.mul(df_a).sum(axis=1) final_result = sum_combined / sum_above_avg  Any guidance on translating this pandas code to Polars would be greatly appreciated!

Answer: Polars has dedicated horizontal functions for "row-wise" operations. df_a.max_horizontal()  shape: (3,) Series: 'max' [i64] [ 7 8 9 ]  For DataFrames, Polars will "broadcast" the operation across all columns if the right-hand side is a Series. df_a == df_a.max_horizontal() # df_a.select(pl.all() == pl.Series([7, 8, 9]))  shape: (3, 2) ┌──────────┬──────────┐ │ feature1 ┆ feature2 │ │ --- ┆ --- │ │ bool ┆ bool │ ╞══════════╪══════════╡ │ false ┆ true │ │ false ┆ true │ │ false ┆ true │ └──────────┴──────────┘  Option #1 max_values = df_a.max_horizontal() selected_features = df_a == max_values final_result = ( (selected_features * df_b).sum_horizontal() / selected_features.sum_horizontal() )  Option #2 above_avg = df_a >= df_a.mean_horizontal() combined_df = above_avg * df_a * df_b sum_combined = combined_df.sum_horizontal() sum_above_avg = (above_avg * df_a).sum_horizontal() final_result = sum_combined / sum_above_avg
Question: I've already tried everything posted here but nothing is working, so please don't mark this as duplicate because I think the problem is different. I have a json like this: [{'Id': 1, 'Design': ["09", '10', '13' ], 'Research': ['Eng', 'Math'] }]  Plus other non-list colums. This is repeated for 500 ids. I need to explode the list columns. the final output should be an excel file, I don't care if the explosion is done directly in json or in pandas. Already tried:  def lenx(x): return len(x) if isinstance(x,(list, tuple, np.ndarray, pd.Series)) else 1 def cell_size_equalize2(row, cols='', fill_mode='internal', fill_value=''): jcols = [j for j,v in enumerate(row.index) if v in cols] if len(jcols)<1: jcols = range(len(row.index)) Ls = [lenx(x) for x in row.values] if not Ls[:-1]==Ls[1:]: vals = [v if isinstance(v,list) else [v] for v in row.values] if fill_mode=='external': vals = [[e] + [fill_value]*(max(Ls)-1) if (not j in jcols) and (isinstance(row.values[j],list)) else e + [fill_value]*(max(Ls)-lenx(e)) for j,e in enumerate(vals)] elif fill_mode == 'internal': vals = [[e]+[e]*(max(Ls)-1) if (not j in jcols) and (isinstance(row.values[j],list)) else e+[e[-1]]*(max(Ls)-lenx(e)) for j,e in enumerate(vals)] else: vals = [e[0:min(Ls)] for e in vals] row = pd.Series(vals,index=row.index.tolist()) return row  Leads to index error df.explode(['B', 'C', 'D', 'E']).reset_index(drop=True)  Columns must have same lenght df1 = pd.concat([df[x].explode().to_frame() .assign(g=lambda x: x.groupby(level=0).cumcount()) .set_index('g', append=True) for x in cols_to_explode], axis=1)  Somehow it creates lots of rows, I think it just explodes a column after another, and it leads to memory error. Desired Output: Id Design Research 1 09 Eng 1 10 Math 1 13 

Answer: You can use json_normalize and a deduplication-explode (as presented here): tmp = pd.json_normalize(json) def explode_dedup(s): s = s.explode() return s.set_axis( pd.MultiIndex.from_arrays([s.index, s.groupby(level=0).cumcount()]) ) ids = ['Id'] cols = tmp.columns.difference(ids) out = (tmp[ids] .join(pd.concat({c: explode_dedup(tmp[c]) for c in cols}, axis=1) .droplevel(-1) )[tmp.columns] )  NB. if you know the columns to explode, you can alternatively use: cols = ['Design', 'Research'] ids = tmp.columns.difference(cols)  Output:  Id Design Research 0 1 09 Eng 0 1 10 Math 0 1 13 NaN
Question: When I run a recursive function and it exceeds the recursion depth limit, the below error is displayed: Python 3.12.4+ (heads/3.12:99bc8589f0, Jul 27 2024, 11:20:07) [GCC 12.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> def f(): f() ... >>> f() Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 1, in f File "<stdin>", line 1, in f File "<stdin>", line 1, in f [Previous line repeated 996 more times] RecursionError: maximum recursion depth exceeded  From what I understand, because the traceback is all the same File "<stdin>", line 1, in f, it does not show it all (because obviously it's not really helpful) and only tells me that this line was repeated 996 times more. When I manually change the recursion limit, I expect that the traceback size grows as well. But it does not: >>> sys.setrecursionlimit(2000) >>> >>> f() Traceback (most recent call last): File "<stdin>", line 1, in f File "<stdin>", line 1, in f File "<stdin>", line 1, in f [Previous line repeated 997 more times] RecursionError: maximum recursion depth exceeded  I doubled the recursion limit, so now I expect that traceback size doubles, but it says that the previous line is repeated 997 times. Why is this the case? Note I also found this question which seems same as my question, but it isn't. My question is specifically about the size of traceback. Why does increasing the recursion depth result in stack overflow error?

Answer: I figured out that this relates to sys.tracebacklimit variable. It limits the traceback size. >>> def f(): f() ... >>> >>> import sys >>> sys.setrecursionlimit(2000) >>> >>> f() Traceback (most recent call last): File "<stdin>", line 1, in f File "<stdin>", line 1, in f File "<stdin>", line 1, in f [Previous line repeated 997 more times] RecursionError: maximum recursion depth exceeded >>> >>> sys.tracebacklimit = 1500 >>> >>> f() Traceback (most recent call last): File "<stdin>", line 1, in f File "<stdin>", line 1, in f File "<stdin>", line 1, in f [Previous line repeated 1497 more times] RecursionError: maximum recursion depth exceeded
Question: I have column list - col_list = ['Subsidiary','State of Jurisdiction of Incorporation','Jurisdiction ofIncorporationor Organization','Jurisdiction of Incorporation or Organization', 'Subsidiaries','State or Other Jurisdiction of Organization','Jurisdiction ofIncorporation orOrganization', 'Company Name', 'State of Incorporation', 'Legal Name','Entity','Name of Company/Jurisdiction of Incorporation or Formation','Place of Formation','Name of Company']  Can I match if my dataframe has these columns using regex or any other library in python like Subsidiary,Subsidiaries,Company Name,Legal Name,Entity,Name of Company/Jurisdiction of Incorporation or Formation,Name of Company , then rename it as 'entity_name' and another column like Jurisdiction, State of Jurisdiction,Place of Formation then rename it as 'entity_place' ? Example - df1 - |Subsidiary|Jurisdiction| |A1|X1| |A2|X2| |A3|X3| expected output - |entity_name|entity_place| |A1|X1| |A2|X2| |A3|X3|  df2- |Legal Name|Place of Formation| |A1|X1| |A2|X2| |A3|X3| Expected output- |entity_name|entity_place| |A1|X1| |A2|X2| |A3|X3|  Any help would be appreciated.

Answer: If there are 2 lists for entity_name and entity_place create dictionary and pass to rename: L1 = ['Subsidiary','Subsidiaries','Company Name','Legal Name', 'Entity', 'Name of Company/Jurisdiction of Incorporation or Formation','Name of Company'] L2 = ['Jurisdiction', 'State of Jurisdiction','Place of Formation'] d = {**dict.fromkeys(L1, 'entity_name'), **dict.fromkeys(L2, 'entity_place')} out = df.rename(columns=d)
Question: I'm working on a project where, in a first stage, I pull some raw data, do a bunch of processing of it in duckdb, and end up with a bunch of tables that are used by a bunch of downstream components that also operate in duckdb. I'd like to have the outputs from the first stage persist on disk, and not be modified by the downstream components, but the downstream components need to at least be able to create views and temporary tables. Further, there's no reason to have the downstream components operate on-disk... the data is small enough to fit in memory. I'd like a magical solution like conn = duckdb.connect(":memory:") conn.load_from_disk(path_to_on_disk)  but nothing like that seems to exist. I can read each table from the on disk connection, convert to pandas, then load into the in memory connection, but that takes forever. Any ideas? Example of that inefficient approach:  def load_disk_duck_to_mem_duck(path: pathlib.Path) -> duckdb.DuckDBPyConnection: """Slow and ugly!""" source_db = duckdb.connect((path / "duck.db").as_posix()) in_memory_db = duckdb.connect(":memory:") tables = source_db.execute("SHOW TABLES").fetchall() # Copy each table from on-disk to in-memory for table in tables: table_name = table[0] temp_df = source_db.table(table_name).df() # Load the table from on-disk and create a copy in the in-memory database in_memory_db.from_df(temp_df).create(table_name) return in_memory_db 

Answer:  I'd like to have the outputs from the first stage persist on disk  This can be accomplished using a DuckDB EXPORT DATABASE statement.  ... the downstream components need to at least be able to create views and temporary tables. Further, there's no reason to have the downstream components operate on-disk... the data is small enough to fit in memory.  So each downstream component should read the exported database, e.g. using the IMPORT DATABASE statement.  Alternatively, if you want a single file, you can use ATTACH and COPY DATABASE, like so: attach 'test.db' as test; copy from database memory to test;  This assumes the specified file ('test.db' in the example) does not already exist.
Question: With the following code, I can not display an image in a tkinter cell: from tkinter import * from tkinter import filedialog from PIL import Image, ImageTk root = Tk() root.geometry=("1000x1000") def orig(): orig_image = filedialog.askopenfilename(filetypes=[("Image file", "*.jpg"), ("All files", "*.")]) my_img = ImageTk.PhotoImage(Image.open(orig_image)) lbl = Label(image=my_img) lbl.grid(row=0, column=0) orig() root.mainloop()  But, by taking this out of the method, it works fine: from tkinter import * from tkinter import filedialog from PIL import Image, ImageTk root = Tk() root.geometry=("1000x1000") orig_image = filedialog.askopenfilename(filetypes=[("Image file", "*.jpg"), ("All files", "*.")]) my_img = ImageTk.PhotoImage(Image.open(orig_image)) lbl = Label(image=my_img) lbl.grid(row=0, column=0) root.mainloop()  What am I missing ? This is part of a larger project where I want to display an "original" OCR scan image and then with other methods, display a "corrected image" next to the original image (in another column) to show if that correction was an improvement or not.

Answer: there are some issues of your code  Unfortunately, the geometry method is being assigned incorrectly. It should be invoked instead of assigned.  To the first code block, you made a definition of a function orig() but let it remain unevaluated.  The PhotoImage object needs to be held on to as an attribute to prevent it from being lost to garbage collection.   try this way: from tkinter import * from tkinter import filedialog from PIL import Image, ImageTk root = Tk() root.geometry("1000x1000") # Corrected: called as a method def orig(): global my_img # Important: keep a reference to prevent garbage collection orig_image = filedialog.askopenfilename(filetypes=[("Image file", "*.jpg"), ("All files", "*.*")]) img = Image.open(orig_image) my_img = ImageTk.PhotoImage(img) lbl = Label(root, image=my_img) lbl.grid(row=0, column=0) orig() # Call the function to open file dialog and display image root.mainloop()
Question: Python does slice-by-copy on strings: Does Python do slice-by-reference on strings? Is this something that all implementations of Python need to respect, or is it just a detail of the CPython implementation?

Answer: Copying underlying memory on str slices is chosen because reference counting / garbage collection becomes complicated otherwise, and reference counting itself is an implementation detail of CPython. Therefore copying string slices is also an implementation detail. Copying could, in theory, be avoided in the implementation without violating any requirements: some years ago a patch was proposed attempting to avoid copying, but it was rejected by Guido because of performance concerns (ref). Note that already, some particular slices do not copy in CPython. This is also an implementation detail/optimization and not documented in the language reference.
Question: I have a generator that does not use send() values. Should I type its send_value as Any or None? import typing as t def pi_generator() -> t.Generator[int, ???, None]: pi = "3141592" for digit in pi: yield int(digit) pi_gen = pi_generator() next(pi_gen) # 3 pi_gen.send('foo') # 1 pi_gen.send(pi_gen) # 4  Reasons I see for Any:  The generator works perfectly fine with send() for any type, so if somebody had a reason to use .send(1) with this generator, it's totally fine. Methods' arguments' types should be general, and .send(x: Any) is more general than .send(x: None).  Reasons I see for None:  Return types should be specific, and "Generator that never uses send" is a more specific type than "Any kind of generator". If someone is using .send() to this generator, it's likely they're misunderstanding what it does and the type hint should inform them. 

Answer: Quoting the docs for typing.Generator:  If your generator will only yield values, set the SendType and ReturnType to None: def infinite_stream(start: int) -> Generator[int, None, None]: while True: yield start start += 1   Setting SendType to None is the recommended way to communicate that the generator does not expect callees to send() values.
Question: I have Python code that has the following shape to it: from dataclasses import dataclass @dataclass class Foo_Data: foo: int class Foo_Processor: def process(self, data: Foo_Data): ... class Foo_Loader: def load(self, file_path: str) -> Foo_Data: ... #---------------------------------------------------------------- @dataclass class Bar_Data: bar: str class Bar_Processor: def process(self, data: Bar_Data): ... class Bar_Loader: def load(self, file_path: str) -> Bar_Data: ...  I have several instances of this sort of Data/Processor/Loader setup, and the classes all have the same method signatures modulo the specific class family (Foo, Bar, etc.). Is there a pythonic way of formalizing this relationship among classes to enforce a similar structure if I decide to create a Spam_Data, Spam_Processor, and Spam_Loader family of classes? For instance, I want something to enforce that Spam_Processor have a process method which takes an argument of type Spam_Data. Is there a way of achieving this standardization somehow with abstract classes, generic types, or some other structure? I tried using abstract classes, but mypy correctly points out that having all *_Data classes be subclasses of an abstract Data class and similarly having all *_Processor classes be subclasses of an abstract Processor class violates the Liskov substitution principle, since each processor is only designed for its respective Data class (i.e., Foo_Processor can't process Bar_Data, but one would expect that it could if these classes have superclasses Processor and Data which are compatible in this way).

Answer: You can use abstract base classes (ABCs) with Generics. This way you can define a common interface while ensuring type safety: from abc import ABC, abstractmethod from dataclasses import dataclass from typing import Generic, TypeVar # generic type variable for Data T = TypeVar('T', bound='BaseData') @dataclass class BaseData(ABC): pass class BaseProcessor(ABC, Generic[T]): @abstractmethod def process(self, data: T) -> None: pass class BaseLoader(ABC, Generic[T]): @abstractmethod def load(self, file_path: str) -> T: pass  Now you can define your specific classes @dataclass class Foo_Data(BaseData): foo: int class Foo_Processor(BaseProcessor[Foo_Data]): def process(self, data: Foo_Data) -> None: ... class Foo_Loader(BaseLoader[Foo_Data]): def load(self, file_path: str) -> Foo_Data: ... @dataclass class Bar_Data(BaseData): bar: str class Bar_Processor(BaseProcessor[Bar_Data]): def process(self, data: Bar_Data) -> None: ... class Bar_Loader(BaseLoader[Bar_Data]): def load(self, file_path: str) -> Bar_Data: ...  Writing your code this way combines the benefits of a common interface with type safety.  ABCs ensure that subclasses implement required methods, promoting a consistent structure.  Generics allow for type-specific operations, enhancing code readability and maintainability.   As a confirmation with mypy: mypy script.py Success: no issues found in 1 source file
Question: I'm looking for a thread-safe implementation of a Semaphore I can use in Python. The standard libraries asyncio.Semaphore isn't thread-safe. The standard libraries threading.Semaphore doesn't have awaitable interface. I am using sanic which has multiple threads (workers) but also an asynchronous loop on each thread. I want to be able to yield execution back to the event loop on each of the workers whenever it encounters a blocked semaphore, while it waits. UPDATE: I meant to say process here, not threads. So these should be that Sanic splits across processes, and multiprocessing.Semaphore. I believe the answer given is still relevant to where I can apply a similar solution.

Answer: If you are saying that you need a single semaphore instance to be used across all the threads, then create a threading.Semaphore instance and am async function acquire_semaphore such as the following: import asyncio import threading from concurrent.futures import ThreadPoolExecutor executor = ThreadPoolExecutor() semaphore = threading.Semaphore() async def acquire_semaphore(): """Make a threading.Semaphore awaitable.""" def acquirer(): semaphore.acquire() loop = asyncio.get_running_loop() await loop.run_in_executor(executor, acquirer) async def test(): """Acquire and release a semaphore sharable across threads.""" await acquire_semaphore() print('acquired') await asyncio.sleep(1) semaphore.release() # This never blocks print('released') def worker(): asyncio.run(test()) def main(): threads = [ threading.Thread(target=worker) for _ in range(3) ] for thread in threads: thread.start() for thread in threads: thread.join() if __name__ == '__main__': main()  Prints: acquired released acquired released acquired released
Question: I'm trying to multiply an input * 5 raised to the power of the input. I tried this: def multiply(n): return 5 ** len(str(n)) * n  I tried (n = -2), but instead of giving me -10, which is the correct answer, it gave me -50 Why doesn't this output the correct numbers when n is negative?

Answer: you are directly checking the length of your input by casting it to string: >>> n = -2 >>> str(n) '-2' # String with length of two -> '-' and '2' (string 2) >>> len(str(n)) 2  Maybe you can try this (not sure if it's good or bad, but will work as you expected): # The dirty way.. def multiply(num: int): # Initialize a variable to keep track of length length = 0 # Convert the input to string and iterate over it for n in str(num): # If the current character is int, increment the length variable try: if isinstance(int(n), int): length += 1 # Above code will raise error for '-', catch here except ValueError: pass return 5 ** length * num # Try with the input -2 again print(multiply(-2))
Question: I want to get a list of names of pokemon from the first 150 that have a hp less than a particular value. Here's what I've got so far: def get_pokemon_with_similar_hp(max_hp): pokemon_names = [] poke_data = [] for i in range(1, 151): api_url = f"https://pokeapi.co/api/v2/pokemon/{i}" pokemon_response = requests.get(api_url) pokemon_data = pokemon_response.json() poke_data.append(pokemon_data) for j in range(0, 150): if poke_data[j]['stats'][0]['base_stat'] < max_hp: pokemon_names.append(poke_data[j]['name']) return pokemon_names  This works, and gives me the data I want, but it's currently taking 8.49 s to process and send through to my frontend. Is there a way of improving my code to speed things up? Many thanks in advance :)

Answer: According to the API documentation you can use GraphQL query, so you can do this in one request. E.g.: import requests graphql_url = "https://beta.pokeapi.co/graphql/v1beta" # https://beta.pokeapi.co/graphql/console/ payload = { "operationName": "samplePokeAPIquery", "query": r"query samplePokeAPIquery($maxhp: Int) {pokemon_v2_pokemon(where: {pokemon_v2_pokemonstats: {base_stat: {_lt: $maxhp}}}) {id name}}", "variables": {"maxhp": 100}, } data = requests.post(graphql_url, json=payload) print(json.dumps(data.json(), indent=4))  Prints: { "data": { "pokemon_v2_pokemon": [ { "id": 1, "name": "bulbasaur" }, { "id": 2, "name": "ivysaur" }, { "id": 3, "name": "venusaur" }, ...
Question: I would expect this to only hold when the last bit of the mantissa is 0. Otherwise, in order to subtract them (since their exponents differ by 1), x would lose a bit of precision first and the result would either end up being rounded up or down. But a quick experiment shows that it seems to always hold (assuming x and 2x are finite) for any random number (including those with a a trailing 1 bit). import random import struct from collections import Counter def float_to_bits(f: float) -> int: """ Convert a double-precision floating-point number to a 64-bit integer. """ # Pack the float into 8 bytes, then unpack as an unsigned 64-bit integer return struct.unpack(">Q", struct.pack(">d", f))[0] def check_floating_point_precision(num_trials: int) -> float: true_count = 0 false_count = 0 bit_counts = Counter() for _ in range(num_trials): x = random.uniform(0, 1) if 2 * x - x == x: true_count += 1 else: false_count += 1 bits = float_to_bits(x) # Extract the last three bits of the mantissa last_three_bits = bits & 0b111 bit_counts[last_three_bits] += 1 return (bit_counts, true_count / num_trials) num_trials = 1_000_000 (bit_counts, proportion_true) = check_floating_point_precision(num_trials) print(f"The proportion of times 2x - x == x holds true: {proportion_true:.6f}") print("Distribution of last three bits (mod 8):") for bits_value in range(8): print(f"{bits_value:03b}: {bit_counts[bits_value]} occurrences")  The proportion of times 2x - x == x holds true: 1.000000 Distribution of last three bits (mod 8): 000: 312738 occurrences 001: 62542 occurrences 010: 125035 occurrences 011: 62219 occurrences 100: 187848 occurrences 101: 62054 occurrences 110: 125129 occurrences 111: 62435 occurrences 

Answer: If we had to do arithmetic only in the floating-point format, even for internal values during the arithmetic, then, yes, 2*x - x would not always yield x. For example, with four-bit significands, we could have:    Expression Value/Calculation     x 1.001•20 (9)   2*x 1.001•21 (18)   2*x - x 1.001•21 − 1.001•20  = 1.001•21 − 0.100•21 (shifted right operand and lost a bit)  = 0.101•21 = 1.010•20 (10).    However, that is not how floating-point implementations work. To get the right results, they use either more digits internally or more sophisticated algorithms or both. IEEE-754 specifies that the result of an elementary operation is the value you would get by computing the exact real-number arithmetic result with no error or rounding or limitations on digits and then rounding that real number to the nearest value representable in the destination format, using the rounding rule in effect for the operation. (Most commonly, that rounding rule is to round to the nearest value, breaking ties in favor of the one with an even low digit in its significand.) A consequence of this requirement is that, if the mathematical result is representable in the format, the computed result must be that mathematical result. When the mathematical result is representable, there is never any rounding error in a properly implemented elementary operation.