eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I would like to subclass a generic class that is using a constrained type variable. This is fairly easy with pre-3.12 syntax as shown in this minimal example: from typing import Generic, TypeVar T = TypeVar("T", int, float) class Base(Generic[T]): pass class Sub(Base[T]): pass  But I am struggling with writing this in the Python 3.12 syntax. I cannot find a way to not have to specify T: (int, float) in the Sub class as well. Below is another minimal example which passes mypy check (version 1.15.0). class Base[T: (int, float)]: pass class Sub[T: (int, float)](Base[T]): pass  But when I change the subclass definition to class Sub[T](Base[T]):, mypy returns error: Type variable "T" not valid as type argument value for "Base" [type-var]. I would expect such definition to be meaningful and I find it unfortunate that I have to define the constraints again, especially when the classes can be in different files and subclassed multiple times. Am I missing something or is this not fully doable in pure Python 3.12+ syntax?

Answer: Unfortunately, you cannot reuse type variables defined with PEP 695 syntax. The type variable must have the correct bound to parametrize the superclass (in other words, bounds and constraints can't be inferred or magically "pushed down" from the superclass, that has soundness problems if you consider multiple inheritance). So, you need some way to spell "reuse type variable with these {constraints,bound,default}"... And as you noted in you question, typing.TypeVar is doing exactly that. Please note that typing.TypeVar is not deprecated (only typing.TypeAlias is) - new generics syntax is just another way, not a replacement. It will likely stay alive for a very long time if not indefinitely: Eric Traut's comment on that  It’s also important to note that this PEP doesn’t deprecate the current mechanisms for defining TypeVars or generic classes, functions, and type aliases. Those mechanisms can live side by side with the new syntax. If you find that those mechanisms are more flexible for exploring extensions to the type system, PEP 695 does not prevent you from leveraging these existing mechanisms.  There are other problems with PEP 695 typevars like inability to specify variance explicitly, so some corner cases would become impossible to express if typing.TypeVar was ever deprecated. I'm not aware of any recent proposals or discussions that would enable reusing PEP 695 type variables, and for the reasons mentioned above I doubt that will ever be possible.
Question: Hi I'm observing a strange behaviour with python multiprocessing Queue object. My environment: OS: Windows 10 python: 3.13.1  but I observed the same with: OS: Windows 10 python: 3.12.7  and: OS: Windows 10 python: 3.10.14  while I could not reproduce it in Linux Redhat. I have this short script: from multiprocessing import Queue a = list(range(716)) queue: Queue = Queue() for item in a: queue.put(item) raise ValueError(f"my len: {len(a)}")  If I run it, everything is ok, it raises the error and exits: Traceback (most recent call last): File "C:\Users\uXXXXXX\AppData\Roaming\JetBrains\PyCharmCE2024.3\scratches\scratch_1.py", line 7, in <module> raise ValueError(f"my len: {len(a)}") ValueError: my len: 716 Process finished with exit code 1  but if i change the number from 716 to 717 or any other number above it, it raises the error but doesn't exit, the script hangs there. and when I forcefully stop the script it exits with code -1 Traceback (most recent call last): File "C:\Users\uXXXXXX\AppData\Roaming\JetBrains\PyCharmCE2024.3\scratches\scratch_1.py", line 7, in <module> raise ValueError(f"my len: {len(a)}") ValueError: my len: 717 Process finished with exit code -1  Can you please help me solve this strange behaviour? i would like it to always automatically exit with code 1

Answer: when you put an item in the queue, you are only putting it in an internal list, and a worker thread is incrementally adding items to the IPC pipe, the process won't exit until the worker empties this list into the pipe, the pipe has a small internal buffer, which is why small sizes work fine, this is done so that queue.put is not blocking, and also makes the queue faster. you need to make sure all items in the queue are consumed for the processes to exit, you should consume all the remaining items in the queue from the main process, you can do that in a worker thread while joining the workers. try: while True: item = queue.get_nowait() # process item except Empty: pass  if there is no one going to read the queue and you just want to throw away the data in it then use queue.cancel_join_thread, which allows the process to exit without the list being drained, potentially leaving the queue in a broken state. only the main process should ever call it. from multiprocessing import Queue a = list(range(2000)) queue: Queue = Queue() for item in a: queue.put(item) queue.cancel_join_thread() raise ValueError(f"my len: {len(a)}")  ValueError: my len: 2000 Process finished with exit code 1
Question: import asyncio async def worker(): print("worker start") await asyncio.sleep(1) print("worker done") async def main(): # Schedule the coroutine asyncio.create_task(worker()) # Give control back to the event loop for a moment await asyncio.sleep(0.1) print("main exiting") asyncio.run(main())  Expected behavior I thought calling asyncio.create_task(worker()) would start worker() right away, so I expected to see: worker start main exiting worker done  Actual behavior Instead, nothing from worker() is printed until after main() finishes: main exiting worker start worker done  What I’ve tried  Increasing the await asyncio.sleep(0.1) delay Replacing asyncio.create_task with await worker() Running on Python 3.11 and 3.12 (same outcome)  Questions  Why doesn’t the task begin execution immediately after create_task()? Is there a recommended pattern to ensure the task gets a chance to run before main() exits (without awaiting it directly)?  Environment  Python 3.12.2 macOS 14.4 (Sonoma)  Any insights appreciated!

Answer: asyncio.create_task(coro) does not result in the immediate execution of coroutine coro. According to the docs:  asyncio.create_task(coro, *, name=None, context=None) Wrap the coro coroutine into a Task and schedule its execution. Return the Task object.  asyncio.create_task(coro) only schedules the task for execution. This is not the same thing as starts the task for execution. The new task will only start execution after a task switch occurs due to an await statement or a return statement being executed. Since you are not saving a reference to the newly created task and awaiting it, that task is not guaranteed to complete. When I run the code, I see: worker start main exiting  It is only when main executes await asyncio.sleep(0.1) that the new task starts. But when that worker task executes await asyncio.sleep(1), the asyncio.sleep(0.1) in main completes and returns back to the asyncio.run(main()) statement, which completes. Thus, the program terminates before worker wakes up from its sleep. You should change your program to: import asyncio async def worker(): print("worker start") await asyncio.sleep(1) print("worker done") async def main(): # Schedule the coroutine task = asyncio.create_task(worker()) # save reference to the task # Give control back to the event loop for a moment await asyncio.sleep(0.1) await task # wait for the task to complete print("main exiting") asyncio.run(main())  Prints: worker start worker done main exiting  Or if you have main sleep long enough, worker will wake up and finish: import asyncio async def worker(): print("worker start") await asyncio.sleep(1) print("worker done") async def main(): # Schedule the coroutine asyncio.create_task(worker()) # Give control back to the event loop for a longer period await asyncio.sleep(2) print("main exiting") asyncio.run(main())
Question: I am looking for guidance on how to check for missing values in a DataFrame that are not the typical "NaN" or "np.nan" in Python. I have a dataset/DataFrame that has a string literal "?" representing missing data. How can I identify this string as a missing value? When I run usual commands using Pandas like: missing_values = df.isnull().sum() print(missing_values[missing_values > 0]) Python doesn't pick up on these cells as missing and returns 0s for the sum of null values. It also doesn't return anything for printing missing values > 0.

Answer: You can use df.replace("?", pd.NA) to properly encode "?" as missing value. This will ensure that those are properly handled in all operations. import pandas as pd data = {"x": [1, 2, "?"], "y": [3, "?", 5]} df = pd.DataFrame(data) print(df.isnull().sum()) # x 0 # y 0 df = df.replace("?", pd.NA) print(df.isnull().sum()) # x 1 # y 1
Question: See my minimal example below for reproducing the issue: import logging import logging.config def test_logging(caplog): LOGGING_CONFIG = { "version": 1, "disable_existing_loggers": False, "formatters": { "default": { "format": "%(asctime)s - %(name)s - %(levelname)s - %(message)s", }, }, "handlers": { "console": { "class": "logging.StreamHandler", "formatter": "default", "level": "DEBUG", }, }, "loggers": { "": { # root logger "handlers": ["console"], "level": "DEBUG", "propagate": True, }, }, } logging.config.dictConfig(LOGGING_CONFIG) logger = logging.getLogger("root_module.sub1.sub2") logger.setLevel(logging.DEBUG) assert logger.propagate is True assert logger.getEffectiveLevel() == logging.DEBUG with caplog.at_level(logging.DEBUG): logger.debug("🔥 DEBUG msg") logger.info("📘 INFO msg") logger.warning("⚠️ WARNING msg") logger.error("❌ ERROR msg") logger.critical("💀 CRITICAL msg") print("🔥 caplog.messages:", caplog.messages) # Final assertion assert any("CRITICAL" in r or "💀" in r for r in caplog.messages)  Running pytest -s outputs: tests/test_logs.py 2025-04-16 17:06:03,983 - root_module.sub1.sub2 - DEBUG - 🔥 DEBUG msg 2025-04-16 17:06:03,983 - root_module.sub1.sub2 - INFO - 📘 INFO msg 2025-04-16 17:06:03,983 - root_module.sub1.sub2 - WARNING - ⚠️ WARNING msg 2025-04-16 17:06:03,983 - root_module.sub1.sub2 - ERROR - ❌ ERROR msg 2025-04-16 17:06:03,983 - root_module.sub1.sub2 - CRITICAL - 💀 CRITICAL msg 🔥 caplog.messages: []  Pytest version is 8.3.5. I don't think it matters, but my setup.cfg: [tool:pytest] testpaths = tests  I am expecting caplog.records to contain all 5 logs, but it is empty list.

Answer: You're not supposed to configure logging during a test. Further, if any real code under test tries to configure loggers, you should probably patch that out. The way caplog fixture works is by adding a capturing handler, so if you've already configured the logging framework, or attempt to reconfigure it during test, then you're interfering with pytest's functionality. This is even explicitly mentioned in the caplog documentation:  ⚠️ Warning The caplog fixture adds a handler to the root logger to capture logs. If the root logger is modified during a test, for example with logging.config.dictConfig, this handler may be removed and cause no logs to be captured. To avoid this, ensure that any root logger configuration only adds to the existing handlers.  Remove the runtime logging configuration makes the test pass: import logging def test_logging(caplog): logger = logging.getLogger("root_module.sub1.sub2") with caplog.at_level(logging.DEBUG): logger.debug("🔥 DEBUG msg") logger.info("📘 INFO msg") logger.warning("⚠️ WARNING msg") logger.error("❌ ERROR msg") logger.critical("💀 CRITICAL msg") print("🔥 caplog.messages:", caplog.messages) # Final assertion assert any("CRITICAL" in r or "💀" in r for r in caplog.messages)
Question: Boiling down a bigger problem to its essentials, I would like to do this: import numpy as np import pandas as pd df = pd.DataFrame({'a': np.random.randint(0, 5, 1000), 'b': np.random.random(1000)}) for aval, subdf in df.groupby('a'): subdf.to_parquet(f'/tmp/{aval}.parquet')  in Polars using LazyFrame: import numpy as np import pandas as pd import polars as pl df = pd.DataFrame({'a': np.random.randint(0, 5, 1000), 'b': np.random.random(1000)}) lf = pl.LazyFrame(df) # ???  I would like to be able to control the name of the output files in a similar way. Thanks!

Answer: You could use a partitioning scheme e.g. PartitionByKey() lf.sink_parquet( pl.PartitionByKey("/tmp/output", by="a"), mkdir = True )  For your example this creates: /tmp/output /tmp/output/a=0 /tmp/output/a=0/0.parquet /tmp/output/a=1 /tmp/output/a=1/0.parquet /tmp/output/a=2 /tmp/output/a=2/0.parquet /tmp/output/a=3 /tmp/output/a=3/0.parquet /tmp/output/a=4 /tmp/output/a=4/0.parquet  The docs show an example of file_path= being used with a callback to customize the filename further if required.
Question: I want to make a pandas dataframe with three columns, such that the rows contain all permutations of three columns, each with its own range of values are included. In addition, I want to sort them asc by c1, c2, c3. For example, a = [0,1,2,3,4,5,6,7], b = [0,1,2], and c= [0,1]. The result I want looks like this: c1 c2 c3 0 0 0 0 0 1 0 1 0 0 1 1 0 2 0 0 2 1 1 0 0 1 0 1 1 1 0 ... 7 2 0 7 2 1  I keep trying to fill columns using numpy.arange, i.e., numpy.arange(0,7,1) for c1. But that doesn't easily create all the possible rows. In my example, I should end up with 8 * 3 * 2 = 48 unique rows of three values each. I need this to act as a mask of all possible value combinations from which I can merge a sparse matrix of experimental data. Does anyone know how to do this? Recursion?

Answer: You can use pandas MultiIndex (the doc) import pandas as pd a = range(8) b = range(3) c = range(2) multi_index = pd.MultiIndex.from_product([a, b, c], names=['c1', 'c2', 'c3']) df = multi_index.to_frame(index=False) print(df)  Output:  c1 c2 c3 0 0 0 0 1 0 0 1 2 0 1 0 3 0 1 1 4 0 2 0 5 0 2 1 6 1 0 0 7 1 0 1 8 1 1 0 9 1 1 1 10 1 2 0 11 1 2 1 12 2 0 0 13 2 0 1 14 2 1 0 15 2 1 1 16 2 2 0 17 2 2 1 18 3 0 0 19 3 0 1 20 3 1 0 21 3 1 1 22 3 2 0 23 3 2 1 24 4 0 0 25 4 0 1 26 4 1 0 27 4 1 1 28 4 2 0 29 4 2 1 30 5 0 0 31 5 0 1 32 5 1 0 33 5 1 1 34 5 2 0 35 5 2 1 36 6 0 0 37 6 0 1 38 6 1 0 39 6 1 1 40 6 2 0 41 6 2 1 42 7 0 0 43 7 0 1 44 7 1 0 45 7 1 1 46 7 2 0 47 7 2 1
Question: Consider this example where I have 3 worker tasks that push results in a queue and a tasks that deals with the pushed data.  async def worker1(queue: asyncio.Queue): while True: res = await do_some_work(param=1) await queue.put(res) async def worker2(queue: asyncio.Queue): while True: res = await do_some_work(param=2) await queue.put(res) async def worker3(queue: asyncio.Queue): while True: res = await do_some_work(param=3) await queue.put(res) async def handle_results(queue: asyncio.Queue): while True: res = await queue.get() await handle_result(res) queue.task_done() async def main(): queue = asyncio.Queue() t1 = asyncio.create_task(worker1(queue)) t2 = asyncio.create_task(worker2(queue)) t3 = asyncio.create_task(worker3(queue)) handler = asyncio.create_task(handle_result(queue)) while True: # do some other stuff .... asyncio.run(main())  The documentation says that asyncio.Queue is not thread-safe, but this should not apply here because all tasks are running in the same thread. But do I need an asyncio.Lock to protect the queue when I have 3 tasks that push into the same queue? Looking at the implementation in Python 3.12 (which creates a putter future and awaits on it before pushing into the queue) I would say no, but I'm not sure and the documentation does not mention what would happen in this case. So, is the asyncio.Lock in this case necessary?

Answer: No - there is no need to locks to put or read items from asyncio Queues. Keep in mind that Python multithreaded code will already require much less locks than most code in other languages, as the data-structures themselves are thread-safe - so, even with a free-threading build (without the GIL) if you have several threads appending values to a list, for example, the list will always be in a consistent state. Of course, code which would modify or create new keys in a shared dictionary will need proper locks, even though the dictionary itself won't ever "break". When we step down do async programming, other concurrent tasks will only ever run when our code hit an await expression (or async for or async with statement) - so the need for locks is reduced even further. In other words, if there is no code running in other threads, even with a lot of concurrent tasks, things like this: value = global_list[0] new_value = (complicated expression using value) global_list[0] = new_value  are concurrency-safe in async code. And on top of that, asyncio queues are built for consistency in async contexts. They'd never break and get into an inconsistent state if two concurrent tasks try to put, get or use the no_wait variants of those in code running in the same thread. (Although if you need to put data in another thread to be consumed in an asynchronous task, that is another thing and will require a carefully developed pattern to work) Just go for it - and keep in mind that unless you want to yield to the async loop at the point you are doing a put, or you are really concerned about constraining the queue size, you can simply to put_nowait (without the await) - that will even prevent other tasks of running "near" your put.
Question: I have a data set like the following and want to scale the data using any of the scalers in sklearn.preprocessing. Is there an easy way to fit this scaler not over the whole data set, but per group? My current solution can't be included in a Pipeline: import pandas as pd import numpy as np from sklearn.preprocessing import StandardScaler df = pd.DataFrame({'group': [1, 1, 1, 2, 2, 2], 'x': [1,2,3,10,20,30]}) def scale(x): # see https://stackoverflow.com/a/72408669/3104974 scaler = StandardScaler() return scaler.fit_transform(x.values[:,np.newaxis]).ravel() df['x_scaled'] = df.groupby('group').transform(scale)   group x x_scaled 0 1 1 -1.224745 1 1 2 0.000000 2 1 3 1.224745 3 2 10 -1.224745 4 2 20 0.000000 5 2 30 1.224745 

Answer: You can create custom transformer, using BaseEstimator and TransformerMixin for example: import pandas as pd import numpy as np from sklearn.base import BaseEstimator, TransformerMixin from sklearn.preprocessing import StandardScaler class GroupScaler(BaseEstimator, TransformerMixin): def __init__(self, group_column, scaler=None): self.group_column = group_column self.scaler = scaler or StandardScaler() self.scalers_ = {} def fit(self, X, y=None): self.scalers_ = {} for group, group_data in X.groupby(self.group_column): scaler = clone(self.scaler) scaler.fit(group_data.drop(columns=[self.group_column])) self.scalers_[group] = scaler return self def transform(self, X): X_scaled = [] for group, group_data in X.groupby(self.group_column): scaler = self.scalers_[group] scaled = scaler.transform(group_data.drop(columns=[self.group_column])) group_df = pd.DataFrame(scaled, index=group_data.index, columns=group_data.columns.drop(self.group_column)) group_df[self.group_column] = group X_scaled.append(group_df) return pd.concat(X_scaled).sort_index() from sklearn.base import clone df = pd.DataFrame({'group': [1, 1, 1, 2, 2, 2], 'x': [1, 2, 3, 10, 20, 30]}) scaler = GroupScaler(group_column='group') scaled_df = scaler.fit_transform(df) print(scaled_df)  Output:  x group 0 -1.224745 1 1 0.000000 1 2 1.224745 1 3 -1.224745 2 4 0.000000 2 5 1.224745 2
Question: I am trying to make a hidden file in Windows using Python, but after I make it hidden it becomes impossible to write. However, reading this file is possible. import ctypes with open("test.txt", "r") as f: print('test') ctypes.windll.kernel32.SetFileAttributesW("test.txt", 0x02) # adding h (hidden) attribute to a file with open("test.txt", "r") as f: print('test') # works with open("test.txt", "w") as f: print('test') # error  Error: Traceback (most recent call last): File "C:\Users\user\Documents\client\test.py", line 8, in <module> with open("test.txt", "w") as f: print('test') # error PermissionError: [Errno 13] Permission denied: 'test.txt'  Using 0x22 (which is a attribute and h attribute) instead of 0x02 doesnt't help. The same error occurs. After making the file visible again via attrib -H test.txt it becomes possible to open it in write mode. However, as far as I know, hidden files should be writable in Windows. OS: Windows 11 Python version: Python 3.10.2

Answer: I haven't seen specific Python documentation on this behavior, but believe it is because at the implementation level, 'w' eventually is calling The Windows API CreateFileW which has this note:  If CREATE_ALWAYS and FILE_ATTRIBUTE_NORMAL are specified, CreateFile fails and sets the last error to ERROR_ACCESS_DENIED if the file exists and has the FILE_ATTRIBUTE_HIDDEN or FILE_ATTRIBUTE_SYSTEM attribute. To avoid the error, specify the same attributes as the existing file.  Opening for writing mode='w' would eventually use CREATE_ALWAYS with FILE_ATTRIBUTE_NORMAL and would return access denied; whereas opening for reading and updating mode='r+' or appending mode='a' would work because OPEN_EXISTING would be used. Demonstrated below with Python: import os import ctypes FILE_ATTRIBUTE_HIDDEN = 0x2 FILE_ATTRIBUTE_NORMAL = 0x80 # Ensure file is deleted first for script repeatability. try: ctypes.windll.kernel32.SetFileAttributesW("test.txt", FILE_ATTRIBUTE_NORMAL) os.remove('text.txt') except Exception as e: pass # Create the file and hide it with open('test.txt', 'w') as f: print('test', file=f) ctypes.windll.kernel32.SetFileAttributesW("test.txt", FILE_ATTRIBUTE_HIDDEN) with open('test.txt', 'r') as f: # read it print('read:', f.read()) with open('test.txt', 'r+') as f: # update it f.truncate() print('written', file=f) with open('test.txt', 'r') as f: # read it back print('write:', f.read())  Output: read: test write: written  The command prompt works the same way: C:>attrib -h test.txt C:>del test.txt C:>echo line1 >test.txt C:>attrib +h test.txt C:>type test.txt line1 C:>echo line2 >test.txt # can't be directly written Access is denied. C:>echo line2 >>test.txt # CAN be appended (updated). C:>type test.txt line1 line2
Question: I have written this code just to read a csv file: import pandas as pd df = pd.read_table('C:\\XXXXX\\Python_Learn\\pandas.csv') print(df.to_string())  and output to this is coming as below:  Date|Invoice ID|Customer Name|Product|Category|Quantity|Unit Price|Total Amount|Region|Salesperson 0 2025-04-01|INV1001|John Smith|Apple iPhone 14|Electronics|1|999.00|999.00|North|Alice Johnson 1 2025-04-01|INV1002|Jane Doe|Samsung TV 55|Electronics|2|600.00|1200.00|NULL|Bob Williams 2 2025-04-02|INV1003|Michael Lee|Nike Sneakers|Apparel|3|120.00|360.00|West|Carol Chen 3 2025-04-02|INV1004|Emma Brown|Office Chair|Furniture|1|150.00|150.00|South|David Patel 4 2025-04-03|INV1005|Olivia Green|HP Laptop|Electronics|1|850.00|850.00|North|Alice Johnson 5 2025-04-03|INV1006|Noah White|Dining Table|Furniture|1|450.00|450.00|NULL|Bob Williams 6 2025-04-04|INV1007|Ava Scott|Levis Jeans|Apparel|2|80.00|160.00|West|Carol Chen 7 2025-04-04|INV1008|Liam Davis|AirPods Pro|Electronics|2|250.00|500.00|NULL|David Patel  Why is my output getting aligned to the right? I added .strip() but it still gives same output. Update: I updated my code with .read_csv() instead of .read_table() & it improved the output in terms of alignment, but now it doesn't show characters on the right side of the file. import pandas as pd df = pd.read_csv('C:\\XXXXX\\Python_Learn\\pandas.csv') print(df)  Output as below,  Date|Invoice ID|Customer Name|Product|Category|Quantity|Unit Price|Total Amount|Region|Salesperson 0 2025-04-01|INV1001|John Smith|Apple iPhone 14|... 1 2025-04-01|INV1002|Jane Doe|Samsung TV 55|Elec... 2 2025-04-02|INV1003|Michael Lee|Nike Sneakers|A... 3 2025-04-02|INV1004|Emma Brown|Office Chair|Fur... 4 2025-04-03|INV1005|Olivia Green|HP Laptop|Elec... 5 2025-04-03|INV1006|Noah White|Dining Table|Fur... 6 2025-04-04|INV1007|Ava Scott|Levis Jeans|Appar... 7 2025-04-04|INV1008|Liam Davis|AirPods Pro|Elec...  Why does it show ... instead of reading content on right side of the file?

Answer: You need to use pd.read_csv() with the correct delimiter. You can read more about it here import pandas as pd df = pd.read_csv('C:\\XXXXX\\Python_Learn\\pandas.csv', delimiter='|') print(df.to_string())