eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: The documentation for sympy.perfect_power says:  Return (b, e) such that n == b**e if n is a unique perfect power with e > 1, else False (e.g. 1 is not a perfect power). A ValueError is raised if n is not Rational.  Yet evaluating sympy.perfect_power(-64) results in False. However, -64 == (-4)**3, so sympy.perfect_power(-64) should return (-4, 3) (also because there is no other integer base with integer exponent > 1). Is this a bug? Or am I missing something here?

Answer: At the documentation, click on [source] and you'll find this:  if n < 0: pp = perfect_power(-n) if pp: b, e = pp if e % 2: return -b, e return False  Given -64, that first computes that 64 is 2^6 and then gives up because 6 isn't odd. I do think it's a bug and it should try to remove the factor 2 from the exponent. Maybe like this:  if n < 0: pp = perfect_power(-n) if pp: b, e = pp e2 = e & -e if e2 != e: return -(b**e2), e//e2 return False
Question: I am having trouble identifying whether a ZoneInfo is built with an alias: > a = ZoneInfo('Atlantic/Faeroe') > b = ZoneInfo('Atlantic/Faroe') > a == b False  It seems like these ZoneInfos are identical in practice. How do I identify that they are the same, as opposed to e.g. EST and UTC which are different?

Answer: The tzdata package publishes the data for the IANA time zone database. If installed, one could compare the data files for those zones located in: <PYTHON_DIR>\Lib\site-packages\tzdata\zoneinfo\Atlantic  The data files for Faeroe and Faroe compare as binary same. Programmatically, the binary data can be directly read and compared: from importlib import resources with resources.open_binary('tzdata.zoneinfo.Atlantic', 'Faroe') as f: data1 = f.read() with resources.open_binary('tzdata.zoneinfo.Atlantic', 'Faeroe') as f: data2 = f.read() print(data1 == data2) # True  Ref: https://tzdata.readthedocs.io/en/latest/#examples There is also the file <PYTHON_DIR>\Lib\site-packages\tzdata\zoneinfo\tzdata.zi which is a text representation of the IANA database, and contains the line: L Atlantic/Faroe Atlantic/Faeroe  An L line means Link LINK_FROM LINK_TO to define an alias. This indicates that Atlantic/Faeroe is an alias for the Atlantic/Faroe time zone defined earlier in the file.
Question: When working with pandas dataframes, I like to use method chains, because it makes the workflow similar to the tidyverse approach in R, where you use a string of pipes. Consider the example in this answer: N = 10 df = ( pd.DataFrame({"x": np.random.random(N)}) .assign(y=lambda d: d['x']*0.5) .assign(z=lambda d: d.y * 2) .assign(w=lambda d: d.z*0.5) )  I think I've heard that manipulating dataframes using lambda is inefficient, because it is not a vectorized operation, but some looping goes on under the hood. Is this an issue with examples like the one above? Are there alternatives to using lambda in a method chain that retain the tidyverse-like approach?

Answer: Your operations are vectorized, the lambda is not operating as the level of the values but rather for the column names. The running time of the function will be negligible for large enough datasets. However, each assign call is generating a new DataFrame. You could use a single assign call, this would avoid generating an intermediate for each step: df = (pd.DataFrame({'x': np.random.random(N)}) .assign(y=lambda d: d['x'] * 0.5, z=lambda d: d.y * 2, w=lambda d: d.z * 0.5, ) )  There is a significant gain in performance:   NB. I'm only timing .assign(x,y,z) vs .assign(x).assign(y).assign(z), the DataFrame is pre-generated.
Question: I have a multiindex dataframe, which comes from groupby. Here is a demo: In [54]: df = pd.DataFrame({'color': ['blue', 'grey', 'blue', 'grey', 'black'], 'name': ['pen', 'pen', 'pencil', 'pencil', 'box'],'price':[2.5, 2.3, 1.5, 1.3, 5.2],'bprice':[2.2, 2, 1.3, 1.2, 5.0]}) In [55]: df Out[55]: color name price bprice 0 blue pen 2.5 2.2 1 grey pen 2.3 2.0 2 blue pencil 1.5 1.3 3 grey pencil 1.3 1.2 4 black box 5.2 5.0 In [56]: a = df.groupby(['color', 'name'])[['price', 'bprice']].sum() In [57]: a Out[57]: price bprice color name black box 5.2 5.0 blue pen 2.5 2.2 pencil 1.5 1.3 grey pen 2.3 2.0 pencil 1.3 1.2  I want to add a row in every color index, the ideal output is:  price bprice color name black * 5.2 5.0 box 5.2 5.0 blue * 4.0 3.5 pen 2.5 2.2 pencil 1.5 1.3 grey * 3.6 3.2 pen 2.3 2.0 pencil 1.3 1.2  There are two requirements:  The new * row should be in the first row of each group expect for the * row, the other row should be sorted by price  I tried a lot of methods, but not find a elegant method. Insert a row into a multiindex dataframe with specified position seems be hard. could you help on this?

Answer: Compute a groupby.sum on a, then append a level with * and concat, finally sort_index based on color: # compute the sum per color/name # sort by descending price a = (df.groupby(['color', 'name'])[['price', 'bprice']].sum() .sort_values(by='price', ascending=False) ) # compute the sum per color # concatenate, sort_index in a stable way out = (pd.concat([a.groupby('color').sum() .assign(name='*') .set_index('name', append=True), a]) .sort_index(level='color', kind='stable', sort_remaining=False) )  Output:  price bprice color name black * 5.2 5.0 box 5.2 5.0 blue * 4.0 3.5 pen 2.5 2.2 pencil 1.5 1.3 grey * 3.6 3.2 pen 2.3 2.0 pencil 1.3 1.2
Question: I dont understand why in python when i write: from collections.abc import * print(map.__mro__) print(issubclass(map,Iterator))  The output I get is: (<class 'map'>, <class 'object'>) True but the iterator class is not displayed in the return of map.___mro___, why does that happen? It is related on how mro works? I expected that "class Iterator" appeared in the console when i wrote print(map.___mro___)

Answer: This is because collections.abc.Iterator implements a __subclasshook__ method that considers a given class to be its subclass as long as it has the __iter__ and __next__ methods defined: class Iterator(Iterable): __slots__ = () @abstractmethod def __next__(self): 'Return the next item from the iterator. When exhausted, raise StopIteration' raise StopIteration def __iter__(self): return self @classmethod def __subclasshook__(cls, C): if cls is Iterator: return _check_methods(C, '__iter__', '__next__') return NotImplemented
Question: I'm learning how Python f-strings handle formatting and came across this syntax: a = 5.123 b = 2.456 width = 10 result = f"The result is {(a + b):<{width}.2f}end" print(result)  This works as expected, however I don't understand why {width} needs its own curly braces within the format specification. Why can't I just use width directly just as "a" and "b"? Isn't width already inside the outer curly braces?

Answer: The braces are needed because the part starting with : is the format specification mini-language. That is parsed as its own f-string, where all is literal, except if put in braces. That format specification language would bump into ambiguities if those braces were not required: that language gives meaning to certain letters, and it would not be clear whether such letter(s) would need to be taken literally or as an expression to be evaluated dynamically. For instance, in Python 3.11+ we have the z option that can occur at the position where you specified the width: z = 5 f"The result is {(a + b):<z.2f}end"  So what would this mean if braces were not needed? Would it be the z option or what the z variable represents? There are many more examples that could be constructed to bring the same conclusion home: the braces are needed for anything that needs to be evaluated inside the format specification part.
Question: If I'm encoding a string using utf-16be and decoding the encoded string using utf-8, I'm not getting any error and the output seems to be correctly getting printed on the screen as well but still I'm not able to convert the decoded string into Python representation using json module. import json str = '{"foo": "bar"}' encoded_str = str.encode("utf-16be") decoded_str = encoded_str.decode('utf-8') print(decoded_str) print(json.JSONDecoder().decode(decoded_str))  I know that encoded string should be decoded using the same encoding, but why this behaviour is what I'm trying to understand? I want to know:  Why encoding str with utf-16be and decoding encoded_str with utf-8 doesn't result in an error?  As encoding and decoding is not resulting in an error and the decoded_str is a valid JSON (as can be seen through the print statement), why decode(decoded_str) result in an error?  Why writing the output to a file and viewing the file through less command show it as binary file? file = open("data.txt", 'w') file.write(decoded_str)  When using less command to view the data.txt: "data.txt" may be a binary file. See it anyway?   If the decoded_str is an invalid JSON or something else, how can I view it in its original form (print() is printing it as a valid JSON )   I'm using Python 3.10.12 on Ubuntu 22.04.4 LTS

Answer:   Why encoding str with utf-16be and decoding encoded_str with utf-8 doesn't result in an error?   Because in this case, the resulting bytes of str.encode("utf-16be") are also valid UTF-8. This is in fact always the case with ASCII characters, you really need to go above U+007F to trigger possible errors here (eg. use the string str = '{"foo": "！"}' which uses a full-width exclamation mark, U+FF01).   As encoding and decoding is not resulting in an error and the decoded_str is a valid JSON (as can be seen through the print statement), why decode(decoded_str) result in an error?   Just because you can print a string does not make it valid JSON. In particular because of the encoding to UTF-16, a bunch of null bytes got added. For example, f in UTF-16BE is 0x0066. Those bytes when re-encoded in UTF-8 actually constitute two characters, f and the null character 0x00. Based on my reading of the JSON spec, null characters are not allowed and that is why decode(decoded_str) fails.   Why writing the output to a file and viewing the file through less command show it as binary file?   Probably those null bytes again. With a lot of null bytes, less is probably flagging it might be a binary file as this is relatively uncommon in UTF-8 (and Linux much prefers UTF-8 over UTF-16)   If the decoded_str is an invalid JSON or something else, how can I view it in its original form (print() is printing it as a valid JSON )   Too many possible answers here, it really depends on what the real use case is here. The quickest one is just don't encode/decode with different encodings. The next quickest is reverse the encode/decode process, though this is not lossless with all strings or encoding possibilities, in particular the surrogate range when dealing with a UTF-16 + UTF-8 mix-up.
Question: I want to increment a column value based on a certain condition within a polars dataframe, while considering how many times that condition was met. Example data. import polars as pl df = pl.DataFrame({ "before": [0, 0, 0, 0, 0, 0, 0, 0, 0], "cdl_type": ["REC", "REC", "GEC", None, None, "GEC", None, "REC", "GEC"], })  Current approach. df = df.with_columns( a=( pl.when(pl.col("cdl_type").is_in(["GEC", "REC"])).then( pl.int_ranges( pl.col("cdl_type") .is_in(["REC", "GEC"]) .rle() .struct.field("len") ).flatten() ) .when(pl.col('cdl_type').is_null().and_(pl.col('cdl_type').shift(1).is_not_null())) .then(pl.lit(1)) .otherwise(0) ) )  Expected output. ┌────────┬──────────┬───────┐ │ before ┆ cdl_type ┆ after │ │ --- ┆ --- ┆ --- │ │ i64 ┆ str ┆ i64 │ ╞════════╪══════════╪═══════╡ │ 0 ┆ REC ┆ 0 │ │ 0 ┆ REC ┆ 1 │ │ 0 ┆ GEC ┆ 2 │ │ 0 ┆ null ┆ 3 │ │ 0 ┆ null ┆ 0 │ │ 0 ┆ GEC ┆ 0 │ │ 0 ┆ null ┆ 1 │ │ 0 ┆ REC ┆ 0 │ │ 0 ┆ GEC ┆ 1 │ └────────┴──────────┴───────┘ 

Answer: Based on the current approach and the expected result, I take that the condition is that cdcl_type equals either "REC" or "GEC". The expected output can the be obtained as follows.  For each contiguous block of rows satisfying the condition, we obtain a corresponding id using pl.Expr.rle_id on expression for the condition. We use the id to create an increasing integer sequence for each such block using pl.int_range. Finally, we shift the sequence add 1 and fill any missing value with 0.  df.with_columns( pl.when( pl.col("cdl_type").is_not_null() ).then( pl.int_range( pl.len() ).over( pl.col("cdl_type").is_in(["REC", "GEC"]).rle_id() ) ).add(1).shift().fill_null(0) )  shape: (9, 3) ┌────────┬──────────┬─────────┐ │ before ┆ cdl_type ┆ literal │ │ --- ┆ --- ┆ --- │ │ i64 ┆ str ┆ i64 │ ╞════════╪══════════╪═════════╡ │ 0 ┆ REC ┆ 0 │ │ 0 ┆ REC ┆ 1 │ │ 0 ┆ GEC ┆ 2 │ │ 0 ┆ null ┆ 3 │ │ 0 ┆ null ┆ 0 │ │ 0 ┆ GEC ┆ 0 │ │ 0 ┆ null ┆ 1 │ │ 0 ┆ REC ┆ 0 │ │ 0 ┆ GEC ┆ 1 │ └────────┴──────────┴─────────┘
Question: Is there a consistent way of reading a c struct from a file back into a Python Dataclass? E.g. I have this c struct struct boh { uint32_t a; int8_t b; boolean c; };  And I want to read it's data into its own python Dataclass @dataclass class Boh: a: int b: int c: bool  Is there a way to decorate this class to make this somehow possible? Note: for I've been reading bytes and offsetting them manually into the class when I'm creating it, I would like to know if there is a better option.

Answer: Since you already have the data read as bytes, you can use struct.unpack to unpack the bytes into a tuple, which can then be unpacked as arguments to your data class contructor. The formatting characters for struct.unpack can be found here, where L denotes an unsigned long, b denotes a signed char, and ? denotes a boolean: import struct from dataclasses import dataclass @dataclass class Boh: a: int b: int c: bool data = bytes((0xff, 0xff, 0xff, 0xff, 0x7f, 1)) print(Boh(*struct.unpack('Lb?', data)))  This outputs: Boh(a=4294967295, b=127, c=True)
Question: I have a pl.DataFrame with a column that is a list of struct entries. The lengths of the lists might differ: pl.DataFrame( { "id": [1, 2, 3], "s": [ [ {"a": 1, "b": 1}, {"a": 2, "b": 2}, {"a": 3, "b": 3}, ], [ {"a": 10, "b": 10}, {"a": 20, "b": 20}, {"a": 30, "b": 30}, {"a": 40, "b": 40}, ], [ {"a": 100, "b": 100}, {"a": 200, "b": 200}, {"a": 300, "b": 300}, {"a": 400, "b": 400}, {"a": 500, "b": 500}, ], ], } )  This looks like this: shape: (3, 2) ┌─────┬─────────────────────────────────┐ │ id ┆ s │ │ --- ┆ --- │ │ i64 ┆ list[struct[2]] │ ╞═════╪═════════════════════════════════╡ │ 1 ┆ [{1,1}, {2,2}, {3,3}] │ │ 2 ┆ [{10,10}, {20,20}, … {40,40}] │ │ 3 ┆ [{100,100}, {200,200}, … {500,… │ └─────┴─────────────────────────────────┘  I've tried various versions of unnest and explode, but I am failing to turn this into a long pl.DataFrame where the list is turned into rows and the struct entries into columns. This is what I want to see: pl.DataFrame( { "id": [1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 3], "a": [1, 2, 3, 10, 20, 30, 40, 100, 200, 300, 400, 500], "b": [1, 2, 3, 10, 20, 30, 40, 100, 200, 300, 400, 500], } )  Which looks like this: shape: (12, 3) ┌─────┬─────┬─────┐ │ id ┆ a ┆ b │ │ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ i64 │ ╞═════╪═════╪═════╡ │ 1 ┆ 1 ┆ 1 │ │ 1 ┆ 2 ┆ 2 │ │ 1 ┆ 3 ┆ 3 │ │ 2 ┆ 10 ┆ 10 │ │ 2 ┆ 20 ┆ 20 │ │ … ┆ … ┆ … │ │ 3 ┆ 100 ┆ 100 │ │ 3 ┆ 200 ┆ 200 │ │ 3 ┆ 300 ┆ 300 │ │ 3 ┆ 400 ┆ 400 │ │ 3 ┆ 500 ┆ 500 │ └─────┴─────┴─────┘  Is there a way to manipulate the first pl.DataFrame into the second pl.DataFrame?

Answer: First explode, then unnest: df.explode('s').unnest('s')  Output: ┌─────┬─────┬─────┐ │ id ┆ a ┆ b │ │ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ i64 │ ╞═════╪═════╪═════╡ │ 1 ┆ 1 ┆ 1 │ │ 1 ┆ 2 ┆ 2 │ │ 1 ┆ 3 ┆ 3 │ │ 2 ┆ 10 ┆ 10 │ │ 2 ┆ 20 ┆ 20 │ │ … ┆ … ┆ … │ │ 3 ┆ 100 ┆ 100 │ │ 3 ┆ 200 ┆ 200 │ │ 3 ┆ 300 ┆ 300 │ │ 3 ┆ 400 ┆ 400 │ │ 3 ┆ 500 ┆ 500 │ └─────┴─────┴─────┘
Question: Given a taskgroup and number of running tasks, per taskgroup docs if any of the tasks raises an error, rest of the tasks in group will be cancelled. If some of these tasks need to perform cleanup upon cancellation, then how would one go about detecting within the task it's being cancelled? Was hoping some exception is raised in the task, but that's not the case: script.py: import asyncio class TerminateTaskGroup(Exception): """Exception raised to terminate a task group.""" async def task_that_needs_to_cleanup_on_cancellation(): try: await asyncio.sleep(10) except Exception: print('exception caught, performing cleanup...') async def err_producing_task(): await asyncio.sleep(1) raise TerminateTaskGroup() async def main(): try: async with asyncio.TaskGroup() as tg: tg.create_task(task_that_needs_to_cleanup_on_cancellation()) tg.create_task(err_producing_task()) except* TerminateTaskGroup: print('main() termination handled') asyncio.run(main())  Executing, we can see no exception is raised in task_that_needs_to_cleanup_on_cancellation(): $ python3 script.py main() termination handled 

Answer: Casually, I might avoid a design where a task group is intentionally cancelled in this way  order and timing of Exception may be difficult to predict potentially-awkward cleanup new work created during cancellation  However, you can except asyncio.CancelledError or use a finally block async def task_that_needs_to_cleanup_on_cancellation(): try: await asyncio.sleep(10) # placeholder for work except asyncio.CancelledError: print('task cancelled, performing cleanup...') raise # end without running later logic  In order to avoid this cleanup-during-cancellation, you could pack potential cleanup work into a collection for the caller to handle in its Exception handler or later Both designs feel a little ugly, but also are very specific to the individual cleanup case as I believe the Exception could occur during any await within them async def task_1(task_id, cleanups): cleanups[task_id] = "args for later work" await asyncio.sleep(10) # placeholder for work del cleanups[task_id] async def task_2(name, cleanups): try: await asyncio.sleep(10) # placeholder for work except asyncio.CancelledError: # consider list instead cleanups[task_id] = "args for later work" raise ... cleanups = {} try: # task group logic except* TerminateTaskGroup: print('main() termination handled') if cleanups: # probably create and asyncio.gather()  Note that the docs also mention that CancelledError should be re-raised, but it may not matter if there is no further logic! Additionally, when asyncio was added in Python 3.7, asyncio.CancelledError inherited from Exception - this was changed to BaseException in 3.8 (and is the case in all later versions) https://docs.python.org/3/library/asyncio-exceptions.html
Question: For example, if I visit http://localhost:8000/detail/PayPal I get a Page not found error 404 with the following message: Using the URLconf ... Django tried these URL patterns, in this order: ... detail/<slug:slug> [name='processor_detail'] The current path, detail/PayPal, matched the last one.  Here is my code: views.py: class ProcessorDetailView(DetailView): model = Processor template_name = 'finder/processor_detail.html' slug_field = 'slug' # Tell DetailView to use the `slug` model field as the DetailView slug slug_url_kwarg = 'slug' # Match the URL parameter name  models.py: class Processor(models.Model): #the newly created database model and below are the fields name = models.CharField(max_length=250, blank=True, null=True) #textField used for larger strings, CharField, smaller slug = models.SlugField(max_length=250, blank=True) ... def __str__(self): #displays some of the template information instead of 'Processot object' if self.name: return self.name[0:20] else: return '--no processor name listed--' def get_absolute_url(self): # new return reverse("processor_detail", args=[str(self.slug)]) def save(self, *args, **kwargs): #`save` model a certain way(detailed in rest of function below) if not self.slug: #if there is no value in `slug` field then... self.slug = slugify(self.name) #...save a slugified `name` field value as the value in `slug` field super().save(*args, **kwargs)  urls.py: path("detail/<slug:slug>", views.ProcessorDetailView.as_view(), name='processor_detail') I want that if I follow a link it either 1. doesn't matter what case I use or 2. the case in the browser url window changes to all lowercase.

Answer: You're getting 404, not because urls.py couldn't find a match, but because ProcessorDetailView couldn't find a slug named "PayPal", even if "paypal" was in the database. So the problem isn't with urls.py, it's with the view trying to look for the slug you've specified. After some research, it turned out that you could make the model lookup case insensitive by using __iexact Here's the modified views.py: from django.views.generic.detail import DetailView from .models import Processor from django.http import Http404 class ProcessorDetailView(DetailView): def get_object(self, queryset=None): if queryset is None: queryset = self.get_queryset() slug = self.kwargs.get(self.slug_url_kwarg) if slug: slug_field = self.get_slug_field() queryset = queryset.filter(**{slug_field + '__iexact': slug}) try: obj = queryset.get() except queryset.model.DoesNotExist: raise Http404("No %(verbose_name)s found matching the query" % {'verbose_name': queryset.model._meta.verbose_name}) return obj model = Processor template_name = 'finder/processor_detail.html' slug_field = 'slug' # Tell DetailView to use the `slug` model field as the DetailView slug slug_url_kwarg = 'slug' # Match the URL parameter name  Here we just overridden the get_object function, which results in modifying the queryset to be case insensitive. Check this out for more info on overriding get_object:  Django docs manually setting PK for get_object
Question: I have integer arrays of the type: import numpy as np seed_idx = np.asarray([[0, 1], [1, 2], [2, 3], [3, 4]], dtype=np.int_) target_idx = np.asarray([[2,9,4,1,8], [9,7,6,2,4], [1,0,0,4,9], [7,1,2,3,8]], dtype=np.int_)  For each row of target_idx, I want to select the elements whose indices are not the ones in seed_idx. The resulting array should thus be: [[4,1,8], [9,2,4], [1,0,9], [7,1,2]]  In other words, I want to do something similar to np.take_along_axis(target_idx, seed_idx, axis=1), but excluding the indices instead of keeping them. What is the most elegant way to do this? I find it surprisingly annoying to find something neat.

Answer: You can mask out the values you don't want with np.put_along_axis and then index the others: >>> np.put_along_axis(target_idx, seed_idx, -1, axis=1) >>> target_idx[np.where(target_idx != -1)].reshape(len(target_idx), -1) array([[4, 1, 8], [9, 2, 4], [1, 0, 9], [7, 1, 2]])  If -1 is a valid value, use target_idx.min() - 1.
Question: How can I obtain the (first occurence) indices of unique elements for a column of type list in polars dataframe? I am looking for something similar to arg_unique, but that only exists for pl.Series, such as to be performed over a whole column. I need this to work one level below that, so on every list that is inside the column. Given the dataframe df = pl.DataFrame({ "fruits": [["apple", "banana", "apple", "orange"], ["grape", "apple", "grape"], ["kiwi", "mango", "kiwi"]] })  I expect the output to be df = pl.DataFrame({ "fruits": [[0, 1, 3], [0, 1], [0, 1]] }) 

Answer: .list.eval() can be used as a fallback when there is no specific .list.* method currently implemented. df.with_columns( pl.col("fruits").list.eval(pl.element().arg_unique()).alias("idxs") )  shape: (3, 2) ┌────────────────────────────────────────┬───────────┐ │ fruits ┆ idxs │ │ --- ┆ --- │ │ list[str] ┆ list[u32] │ ╞════════════════════════════════════════╪═══════════╡ │ ["apple", "banana", "apple", "orange"] ┆ [0, 1, 3] │ │ ["grape", "apple", "grape"] ┆ [0, 1] │ │ ["kiwi", "mango", "kiwi"] ┆ [0, 1] │ └────────────────────────────────────────┴───────────┘