eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have the following code: def my_zip(*iterables): iterators = tuple(map(iter, iterables)) while True: yield tuple(map(next, iterators))  When my_zip is called, it just creates an infinite loop and never terminates. If I insert a print statement, it is revealed that my_zip is infinitely yielding empty tuples! My expectation was that something inside my_zip would eventually raise StopIteration. However, the (supposedly behaviorally) equivalent code with a generator expression instead works fine: def my_genexp_zip(*iterables): iterators = tuple(iter(it) for it in iterables) while True: try: yield tuple(next(it) for it in iterators) except: print("exception caught!") return  Why is the function with map not behaving as expected? (Or, if it is expected behavior, how could I modify its behavior to match that of the function using the generator expression?) I am testing with the following code: print(list(my_genexp_zip(range(5), range(0, 10, 2)))) print(list(my_zip(range(5), range(0, 10, 2)))) 

Answer: The two pieces of code you provided are not actually "equivalent", with the function using generator expressions notably having a catch-all exception handler around the generator expression producing items for tuple output. And if you actually make the two functions "equivalent" by removing the exception handler: def my_listcomp_zip(*iterables): iterators = tuple(iter(it) for it in iterables) while True: yield tuple(next(it) for it in iterators) print(list(my_listcomp_zip(range(5), range(0, 10, 2))))  you'll get a traceback of: Traceback (most recent call last): File "test.py", line 4, in <genexpr> yield tuple(next(it) for it in iterators) ~~~~^^^^ StopIteration The above exception was the direct cause of the following exception: Traceback (most recent call last): File "test.py", line 6, in <module> print(list(my_listcomp_zip(range(5), range(0, 10, 2)))) ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "test.py", line 4, in my_listcomp_zip yield tuple(next(it) for it in iterators) ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: generator raised StopIteration  So it is clear by now that the reason why your infinite loop with while True: can end at all with your generator expression version of the function is because a RuntimeError is caught by your catch-all exception handler, which returns from the function. And this is because since Python 3.7, with the implementation of PEP-479, StopIteration raised inside a generator gets automatically turned into a RuntimeError in order not to be confused with the StopIteration raised by an exhausted generator itself. If you try your code in an earlier Python version (such as 2.7), you'll find the generator expression version of the function gets stuck in the infinite loop just as well, where the StopIteration exception raised by next bubbles out from the generator and gets handled by the tuple constructor to produce an empty tuple, just like the map version of your function. And addressing this exception masking effect is exactly why PEP-479 was proposed and implemented.
Question: In typestubs for the Python standard library I noticed a peculiar type called MaybeNone pop up, usually in the form of NormalType | MaybeNone. For example, in the sqlite3-Cursor class I find this: class Cursor: # May be None, but using `| MaybeNone` (`| Any`) instead to avoid slightly annoying false positives. @property def description(self) -> tuple[tuple[str, None, None, None, None, None, None], ...] | MaybeNone: ...  The definition of this MaybeNone is given as: # Marker for return types that include None, but where forcing the user to # check for None can be detrimental. Sometimes called "the Any trick". See # CONTRIBUTING.md for more information. MaybeNone: TypeAlias = Any # stable  (I could not find additional information in the CONTRIBUTING.md, which I assume to be this one.) I understand the intention of marking a return type in such a way that a user is not forced to null check in cases where the null is more of a theoretical problem for most users. But how does this achieve the goal?  SomeType | Any seems to imply that the return type could be anything, when what I want to say is that it can be SomeType or in weird cases None, so this doesn't seem to express the intent.  MyPy already allows superfluous null-checks on variables that can be proven not to be None even with --strict (at least with my configuration?) so what does the special typing even accomplish as compared to simply doing nothing?  

Answer: A nice summary can be found in this comment explaining the "Any Trick" of typeshed.  We tend to use it whenever something can be None, but requiring users to check for None would be more painful than helpful.  As background they talk about xml.etree.ElementTree.getroot which in some case returns None (Happens when the tree is initialized without a root). To reflect this, getroot was updated to def getroot(self) -> Element | Any: ... with the possible return types(Element) and additionally | Any. The different possibilities and effects are summarized as:  -> Any means "please do not complain" to type checkers. If root has type Any, you will no error for this.   -> Element means "will always be an Element", which is wrong, and would cause type checkers to emit errors for code like if root is None.   -> Element | None means "you must check for None", which is correct but can get annoying. [..., it could be possible used] to do things like ET.parse("file.xml").getroot().iter("whatever").   -> Element | Any means "must be prepared to handle an Element". You will get an error for root.tagg, because it is not valid when root is an Element. But type checkers are happy with if root is None checks, because we're saying it can also be something else than an Element.  I did slightly modify the quotes by adding italics and -> type
Question: I have code that uses Twisted so I've written a test function for it and decorated it with @pytest_twisted.ensureDeferred. The function awaits on some Deferreds. Then, I need to run some aiohttp website in it so I've written a fixture that uses the pytest_aiohttp.plugin.aiohttp_client fixture, decorated it with @pytest_asyncio.fixture and used it in my test function. The result doesn't work (probably because I need to make Twisted and aiohttp use the same event loop or something like that?). Specifically, it prints "twisted/internet/asyncioreactor.py:50: DeprecationWarning: There is no current event loop" and then crashes with the following exception: .env/lib/python3.13/site-packages/pytest_twisted/__init__.py:343: in _run_inline_callbacks _instances.reactor.callLater(0.0, in_reactor, d, f, *args) .env/lib/python3.13/site-packages/twisted/internet/asyncioreactor.py:289: in callLater self._reschedule() .env/lib/python3.13/site-packages/twisted/internet/asyncioreactor.py:279: in _reschedule self._timerHandle = self._asyncioEventloop.call_at(abs_time, self._onTimer) /usr/lib/python3.13/asyncio/base_events.py:812: in call_at self._check_closed() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <_UnixSelectorEventLoop running=False closed=True debug=False> def _check_closed(self): if self._closed: > raise RuntimeError('Event loop is closed') E RuntimeError: Event loop is closed /usr/lib/python3.13/asyncio/base_events.py:556: RuntimeError  As the actual test function is not even executed its content doesn't seem to matter, and custom fixtures also aren't needed for this to fail so here is a minimal one that shows the problem: @ensureDeferred async def test_minimal(aiohttp_client): app = web.Application() await aiohttp_client(app)  My settings: [tool.pytest.ini_options] addopts = [ "--reactor=asyncio", ] asyncio_mode = "strict" asyncio_default_fixture_loop_scope = "function"  (not 100% sure about these, but I think moving to asyncio_mode = "auto" and replacing @pytest_asyncio.fixture with @pytest.fixture can only make it worse, and changing the fixture loop scope to "module" makes the runner hang before doing anything). What is the correct way to write such test functions, assuming it exists? Edit: it seems to me now (but I may be very wrong) that the "correct" loop scope for everything is in fact "session", because the asyncio reactor runs once and uses the same loop for the entire run, but the main problem is making all the pieces use the same loop, and I don't know if that's possible without changing either of the plugins.

Answer: According to the discussion in https://github.com/pytest-dev/pytest-twisted/issues/188 it doesn't seem possible without changing at least one of the plugins. However, if you control the async fixtures you need to use, you can use pytest-twisted for everything, decorating the fixtures with @pytest_twisted.async_yield_fixture, and don't use pytest-asyncio.
Question: I have a local Django 5.1/Celery 5.4 project that is using SQLite. I am the only user. Certain model saves trigger a Celery task that queries (SELECT) for the updated record (using the Django ORM), then runs an API call to update a remote record based on the local data, and then runs another UPDATE locally. The task wraps all this inside of with transaction.atomic():. (The Celery worker is configured to run tasks in serial.) While this task is running, any attempts to write to the database result in a "database is locked" OperationalError. I have configured Django/SQLite with the latest "production-ready" settings: DATABASES = { 'default': { 'ENGINE': 'django.db.backends.sqlite3', 'NAME': DB_DIR / 'db.sqlite3', 'OPTIONS': { 'init_command': """ PRAGMA foreign_keys=ON; PRAGMA journal_mode=WAL; PRAGMA synchronous=NORMAL; PRAGMA busy_timeout = 5000; PRAGMA temp_store = MEMORY; PRAGMA mmap_size=134217728; PRAGMA journal_size_limit=67108864; PRAGMA cache_size=2000; """, 'transaction_mode': 'IMMEDIATE', 'timeout': 20, }, }, }  I was under the impression that with these settings, concurrent access was possible. "SQLite in Production" is the latest hotness, and these settings, especially the new-to-Django 5.1 'transaction_mode': 'IMMEDIATE' in OPTIONS, would allow writes to queue. What am I missing?

Answer: The solution in this particular case was to shorten my transaction times, i.e. don't hold on to a transaction while making an external API call. This means I have to be more careful about not letting the views and tasks step on each others toes. I'm still flummoxed that so-called "production ready" settings don't allow concurrent access, with not so much as a queue + timeout!
Question: I’ve been using a Python script to perform multiple find/replace actions at once. It that has 3 inputs right inside the code, in a form like this: def main(): text_passage = """ (Here be huge blocks of text) “"" to_replace_input = “"" (Here be large strings to look for, each on a new line) “”” replacements_input = """ (Here be the corresponding replacements, each on a new line) “””  I’ve just been copy/pasting the 3 of these things right into the code of the script, saving, and running the script. It works fine, but I would like make the experience feel a little bit cleaner/faster with some simple UI elements. And since I already use macOS Shortcuts for many quick tools, I was hoping I could make a Shortcut for this. But I can’t figure out how to pass three different Shortcut inputs to the script. I’m using 3 of the ASK FOR INPUT actions and one RUN SHELL SCRIPT action. I tried using ChatGPT to modify the Python script for me to use in this way, but I am getting constant errors relating to input not being passed to the script.  Is there some special way I need to get all 3 inputs communicating to the RUN SHELL SCRIPT action? I should note that I am not a programmer or anything so it’s also possible I’ve just screwed up the Python script and don’t know what I’m doing. The original method I mentioned still works so this isn’t life or death but it would be nice to understand better and get this working as a Shortcut. Thanks!

Answer: I found How to pass variables into shell script : r/shortcuts and there is screenshot which suggests that you can put variables in script and it will copy/paste values automatically: Image from answer on Reddit (author: Infamous_Pea6200):
Question: Pandas astype() appears to unexpectedly switch to performing in-place operations after loading data from a pickle file. Concretly, for astype(str), the data type of the input dataframe values is modified. What is causing this behavior? Pandas version: 2.0.3 Minimal example: import pandas as pd import numpy as np # create a test dataframe df = pd.DataFrame({'col1': ['hi']*10 + [False]*20 + [np.nan]*30}) # print the data types of the cells, before and after casting to string print(pd.unique([type(elem) for elem in df['col1'].values])) _ = df.astype(str) print(pd.unique([type(elem) for elem in df['col1'].values])) # store the dataframe as pkl and directly load it again outpath = 'C:/Dokumente/my_test_df.pkl' df.to_pickle(outpath) df2 = pd.read_pickle(outpath) # print the data types of the cells, before and after casting to string print(pd.unique([type(elem) for elem in df2['col1'].values])) _ = df2.astype(str) print(pd.unique([type(elem) for elem in df2['col1'].values]))  Output: 

Answer: This is a bug that has been fixed in pandas 2.2.0:  Bug in DataFrame.astype() when called with str on unpickled array - the array might change in-place (GH 54654)  As noted by Itayazolay in the PR, regarding the pickle MRE used there:  The problem is not exactly with pickle, it's just a quick way to reproduce the problem. The problem is that the code here attempts to check if two arrays have the same memory (or share memory) and it does so incorrectly - result is arr See numpy/numpy#24478 for more technical details.  If you're using a version < 2.2 and cannot upgrade, you could try manually applying the fix mentioned in the PR and recompiling ".../pandas/_libs/lib.pyx". At #L759:  if copy and result is arr: result = result.copy()  Required change:  if copy and (result is arr or np.may_share_memory(arr, result)): result = result.copy()  There are now some extra comments in ".../pandas/_libs/lib.pyx", version 2.3.x, together with adjusted logic. See #L777-L785:  if result is arr or np.may_share_memory(arr, result): # if np.asarray(..) did not make a copy of the input arr, we still need # to do that to avoid mutating the input array # GH#54654: share_memory check is needed for rare cases where np.asarray # returns a new object without making a copy of the actual data if copy: result = result.copy() else: already_copied = False
Question: My Synology DS418play recently updated to the latest version of DSM7 that is available. I noticed that a lot of the python scripts that I had have started returning weirdly encoded data. Here is an example of the code: requests.get("https://www.23andmedatasettlement.com/").content  returns  b'(\xb5/\xfd\x04X\x1c&\x00\xb6\xbd\xba50\x8b&\x0f\xc0\xc0@\xc7\xb000P\x15\x7fkd\xe1\x8eRJ\x1d\xa4MC\x8bw;\xacv/Ln\x804E\xe7i\xf2\xff\x00U\x11Y\xd9n\x98C\xbe\xcc\xa0\x8ce\x15\xb1\x00\xab\x00\xa5\x00\xd5\xbf\xda\xd8Kl\xa7\x8ds(\x8aK\xb06|\x97\x9a{Tk\x154T\xa7d+\xed?\x15<\xa7?\xdfy\x12z\xe4\x9c\xb5\x1e\xae\xbb\xfb\xad\xf5p\x0f\x82\x05\xc6#\x12\x99\x98\xe8~kA\xd8\x98\xb2\xfa\x83\x87\xeb\xa7\xa8\xf4\x91\xa6E"\x11\x08%WiZI\xf8T\x94\x9c!\x8dM\xa5\x8f\xdc\x83 \xd1\x16\x18\xbd1\x1f\xac\xf5p\xceS\xf2%\xf3l-m\x10T\xfa\xa8%\xb84\x08[\xf60\xb1i\x9aZ\x93\xdc\xffH\xb5:`\xd1\x1a\x85\xd5\xce\x9f\xb9B|i\xc8\xc3 ......'  and it looks like this is because the request headers is 'Accept-Encoding': 'gzip, deflate, br, **zstd**'. Running  requests.get("https://www.23andmedatasettlement.com/", headers={'Accept-Encoding': 'deflate'}).content returns the proper data. I am trying to avoid changing each of my python requests to explicitly set Accept-Encoding. Is there a way to prevent requests from using zstd compression? The output breaks a lot of my scripts.

Answer: I was able to find a commit that basically says if zstandard module is installed, add zstd to the Accept-Encoding. Running pip uninstall zstandard fixed my issue.
Question: I hope I am explaining this correctly. I have a dataframe in which i need to identify pairs of rows based on the string value of two columns. Each row in the pair must have a different string value in another column. I then need to add a new value to a new column based on TRUE or FALSE condition of that third column AND the condition of the pairing. For example. A simplified version of the df would be:  The end result would look like this:  Any help would be greatly appreciated.

Answer: Hoping this helps, using data similar to the examples you shared. data = { 'name': ['John', 'John', 'Jane', 'Jane', 'Doe', 'Doe'], 'city': ['LA', 'LA', 'SF', 'SF', 'SD', 'SD'], 'item': ['Peanut Butter', 'Jelly', 'Peanut Butter', 'Peanut Butter', 'Jelly', 'Jelly'] } df = pd.DataFrame(data)  We can then create a dictionary of the string pairs from name and city with the values being a list of the items associated with the pairs. unique_dict = df.groupby(['name', 'city'])['item'].apply(list).to_dict()  Once we have this we define a function to handle the logic of what each pair needs and can apply it to the dataframe. def determine_needs(items): if 'Peanut Butter' in items and 'Jelly' in items: return None elif 'Peanut Butter' in items: return 'Jelly' elif 'Jelly' in items: return 'Peanut Butter' else: return None df['needs'] = df.apply(lambda row: determine_needs(unique_dict[(row['name'], row['city'])]), axis=1)
Question: In Python-Polars, I am trying to get the shrinked data type of a column using an expression, to be able to run validations against it. For example, I would like to build an expression that allows me to do the following: df = pl.DataFrame({"list_column": [[1, 2], [3, 4], [5, 6]]}) shape: (3, 1) ┌─────────────┐ │ list_column │ │ --- │ │ list[i64] │ ╞═════════════╡ │ [1, 2] │ │ [3, 4] │ │ [5, 6] │ └─────────────┘ df.select(type_check = pl.lit((pl.col("list_column").shrink_dtype() == pl.List))) shape: (3, 2) ┌─────────────┬────────────┐ │ list_column ┆ type_check │ │ --- ┆ --- │ │ list[i64] ┆ bool │ ╞═════════════╪════════════╡ │ [1, 2] ┆ true │ │ [3, 4] ┆ true │ │ [5, 6] ┆ true │ └─────────────┴────────────┘  Is this something feasible?

Answer: No. In first place, the data type for the list_column in your example is pl.List(pl.Int64()), so it would not be equal to pl.List - polars has a strong distinction between different nested types, and shrink_dtype does not currently works for that case at all. Secondly, the data type is always the same for all rows within a given column, so it does not makes much sense to do the same operation for every single row. You can use df.collect_schema() to get a Schema object instead, which contains the data type for each column. Alternatively, you might want to consider using dtype selectors if you wanted to perform different operations for each type.
Question: I found sample code for interrogating NTP servers on https://www.mattcrampton.com/blog/query_an_ntp_server_from_python/. The code is brief and well-written, but I don't understand the use of struct.unpack. This is the core code: client = socket.socket(AF_INET,SOCK_DGRAM) client.sendto(msg.encode('utf-8'),address) msg,address = client.recvfrom(buf) t = struct.unpack("!12I",msg)[10]  It returns an integer value (seconds from 1900-01-01) but I have two questions:  How does the unpack work in this instance? How did he arrive at "!12I" to do the decoding? Is it possible to get a floating point value from the remote server? 

Answer: Following the documentation of struct, you are unpacking the first twelve (12) big-endian (!) unsigned 4-byte integers (I) of the NTP header into a tuple, of which you then extract the value at index 10 ([10]), i.e. the penultimate integer value. Following the definition of the NTP header format, this value is the first part of the Transmit Timestamp (Offset Octet 40 in the linked Wikipedia article). The first part of each timestamp, that is, in your case, the part that you extracted, corresponds to the seconds passed since 1900-01-01 (midnight). Because UNIX timestamps are represented in seconds passed since 1970-01-01 (midnight) instead, the code in the blog post then continues to correct this difference by subtracting the number of seconds passed between 1900-01-01 and 1970-01-01: TIME1970 = 2208988800 ... t = struct.unpack("!12I", msg)[10] t -= TIME1970  As to the second part of your question (floating point value): The last integer of the message that you unpacked ([11]) contains the fractional part of a second for your timestamp, namely a numerator to be divided by the denominator 2³². So, to also get the floating point value that you are after, you could extend your code as follows: client = socket.socket(AF_INET, SOCK_DGRAM) client.sendto(msg.encode('utf-8'), address) msg, address = client.recvfrom(buf) t, f = struct.unpack("!12I", msg)[10:12] t += f / (2 ** 32)  All in all, reproducing the code from the quoted blog post, but with fractional seconds, you could do: from datetime import datetime from socket import AF_INET, SOCK_DGRAM import socket, struct def getNTPTime(host = "pool.ntp.org"): port = 123 buf = 1024 address = (host, port) msg = '\x1b' + 47 * '\0' TIME1970 = 2208988800 client = socket.socket(AF_INET, SOCK_DGRAM) client.sendto(msg.encode('utf-8'), address) msg, address = client.recvfrom(buf) t, f = struct.unpack("!12I", msg)[10:12] t -= TIME1970 t += f / (2 ** 32) return datetime.fromtimestamp(t).strftime("%a %b %d %H:%M:%S.%f %Y") if __name__ == "__main__": print(getNTPTime())  Update: Not sure if this makes a huge difference in practice, but a bit more robust approach would probably be to add the fractional seconds only after creation of the datetime object that I used for producing the format string, thus: from datetime import datetime, timedelta ... t, f = struct.unpack("!12I", msg)[10:12] t -= TIME1970 t = datetime.fromtimestamp(t) + timedelta(seconds=f / (2 ** 32))
Question: I have a list made by strings, correctly cleaned (split(',') can be safely used), and correctly sorted depending on numbers. As a small example: l = ['C1', 'C1,C2', 'C2,C3', 'C3,C4', 'C4', 'C5', 'C5,C6', 'C6,C7', 'C7,C8', 'C8', 'C10', 'C10,C11', 'C11,C12', 'C12,C13', 'C13']  What I'm trying to achieve is to create as many sublists that start and end with single strings, that is: [ ['C1', 'C1,C2', 'C2,C3', 'C3,C4', 'C4'], ['C5', 'C5,C6', 'C6,C7', 'C7,C8', 'C8'], ['C10', 'C10,C11', 'C11,C12', 'C12,C13', 'C13'] ]  I thought to add some logic like the following code, but I'm not sure if I'm on the correct way: tl = [] for i in l: # just get the variable val = i tl.append(val) # split by , val_split = len(i.split(',')) # check if the value is the first element of the list (C1) if val == l[0]: print(1, val) # check if the split of the character is longer than 2 (C1,C2) elif val_split > 1: print(2, val) # check is the split of the character siis equalt to 1 (C4) elif val_split == 1: # here the code should compare if the character is equal to the last value of the nested list. If yes go with teh next value (C5) if val != tl[-1]: print(3, val) else: print(4, val) 

Answer: If the input list is guaranteed to start and end with a single string and if there will always be at least one adjacent pair of single strings then: lst = ['C1', 'C1,C2', 'C2,C3', 'C3,C4', 'C4', 'C5', 'C5,C6', 'C6,C7', 'C7,C8', 'C8', 'C10', 'C10,C11', 'C11,C12', 'C12,C13', 'C13'] result = [[]] for e in lst: result[-1].append(e) if not "," in e: if len(result[-1]) > 1: result.append([]) result.pop() print(result)  Output: [['C1', 'C1,C2', 'C2,C3', 'C3,C4', 'C4'], ['C5', 'C5,C6', 'C6,C7', 'C7,C8', 'C8'], ['C10', 'C10,C11', 'C11,C12', 'C12,C13', 'C13']]