eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am trying to create a Python class that does operations mod n. For example using mod 100: 11*11==21 66+39==5 1+2+3-30==73 2**10==24  This is how I am doing it: class ModInteger: def __init__(self, value, mod): self.mod = mod self.value = value % mod def __add__(self, other): return ModInteger((self.value + other.value) % self.mod, self.mod) def __sub__(self, other): return ModInteger((self.value - other.value) % self.mod, self.mod) def __mul__(self, other): return ModInteger((self.value * other.value) % self.mod, self.mod) def __pow__(self, other): return ModInteger((self.value ** other.value) % self.mod, self.mod) #...  It works, but is there a less tedious and repetitive way of doing it?

Answer: A more generic approach would be to build ModInteger with a metaclass that creates wrapper methods around relevant int methods. To allow compatibility with both int and ModInteger objects the wrapper methods should convert arguments of ModInteger objects into int objects before passing them to int methods, and convert returning values back to ModInteger objects if they are int. The demo below shows how wrapper methods work for both unary and binary operations, as well as methods such as __repr__, which doesn't take an argument and doesn't return an int object: class ModIntegerMeta(type): def __new__(metacls, name, bases, members): cls = super().__new__(metacls, name, bases, members) for name in '__add__', '__sub__', '__mul__', '__neg__', '__repr__': setattr(cls, name, cls.create_method(name)) return cls def create_method(cls, name): def method(self, *args): value = getattr(self.value, name)(*( arg.value if isinstance(arg, cls) else arg for arg in args )) if type(value) is int: return cls(value, self.mod) return value return method class ModInteger(metaclass=ModIntegerMeta): def __init__(self, value, mod): self.mod = mod self.value = value % mod print((-ModInteger(66, 100) + ModInteger(39, 100) * 5))  This outputs 29 because (-66 + (39 * 5) % 100) % 100 == 29 Demo: https://ideone.com/m5upVv
Question: Say I have a dataframe such as df = pd.DataFrame(data = {'col1': [1,np.nan, '>3', 'NA'], 'col2':["3.","<5.0",np.nan, 'NA']})  out: col1 col2 0 1 3. 1 NaN <5.0 2 >3 NaN 3 NA NA  What I would like is to strip stuff like "<" or ">", and get to floats only out: col1 col2 0 1 3. 1 NaN 5.0 2 3 NaN 3 NaN NaN  I thought about something like df['col2'].replace({'.*' : r"[-+]?(?:\d*\.*\d+)"}, regex=True, inplace=True)  the idea being, replace anything with the regex for float, (I think), but this fails error: bad escape \d at position 8  I tried along the lines of df['col2'].replace({r"[-+]?(?:\d*\.*\d+)": r"\1"}, regex=True, inplace=True)  assuming ">"or "<" come before the float number, but then this fails (does not catch anything) if the field is a string or np.nan. Any suggestions please?

Answer: This should work. Logic: First match, but do not include in the capture group \1, anything we do not want to keep at the beginning of the number with the negated character class [^...]. Python (re module flavor regex): pattern = "^[^\w+-\.]*([+-]?\d*\.?\d*)" replacement = \1  Regex Demo: https://regex101.com/r/Halz4X/2 NOTES:  ^ Matches the beginning of the string. [^\w+-\.]* *Negated class [^...] that matches anything that is not an alphanumeric or underscore, _, character (\w), or a literal +, - or ., 0 or more times (*). ( Begin first (and only) capture group, referred to in the replacement string with \1. [+-]?\d*.?\d*  [+-]? Match 0 or 1 (?) + or - \d* Match 0 or more (*) digits \d. \. Match a literal dot, .. \d* Match 0 or more (*) digits \d.   ) End capture (group 1 \1). Replacement string: \1 replaces the matched string with only the characters in the capture group 1.
Question: In the NumPy module, when you call np.polynomial.Polynomial.fit with full=True and you look at the residuals value that's returned you get an object of type array. If this value is always a single number, why is it returned as an array?

Answer: Because, what Polynomial.fit basically does, is calling lstsq. See an artificial example import numpy as np x=np.linspace(-1,1,11,1) # I choose this range because I am lazy: then I know that there is no conversion needed to get the returned coefficient y=12+x+3*x**2+0.5*x**3 np.polynomial.polynomial.Polynomial.fit(x,y,3,full=True) # (Polynomial([12. , 1. , 3. , 0.5], domain=[-1., 1.], window=[-1., 1.], symbol='x'), [array([7.69471606e-30]), 4, array([1.38623557, 1.32269864, 0.50046809, 0.27991237]), 2.4424906541753444e-15])  And now, trying to redo it myself with lstsq (Find coefficient such as linear combination of columns of M with those coefficients result into y) # Columns of M are 1, x, x², x³. M=np.array([np.ones_like(x), x, x**2, x**3]).T np.linalg.lstsq(M, y) # So returned coefficient are such as y ≈ cf[0]+cf[1]*x+cf[2]*x²+cf[3]*x³ # So, the coefficient of the deg 3 polynomial # (array([12. , 1. , 3. , 0.5]), array([1.7292925e-30]), 4, array([3.60116183, 2.60171483, 1.07908918, 0.50695163]))  So, same coefficients, of course. I didn't add any noise, so fitting is perfect. And other returned values are the one returned by Polynomial.fit: sum of squared residuals, rank of matrix, singular values (Focus on the form of the answer, rather than on the values: it is not exactly that least square that Polynomial.fit does) So, Polynomial.fit returns a singleton array of sum of squared residuals, because it returns what lstsq returns, and that is what lstsq returns. Now, of course, your next question has to be "but then, why lstsq does that". Because, contrarily to Polynomial.fit, lstsq could be called with an array of y vectors (so a 2D array) For example: y = np.array([12+x+3*x**2+0.5*x**3, 1+x]).T np.linalg.lstsq(M, y) #(array([[1.20000000e+01, 1.00000000e+00], # [1.00000000e+00, 1.00000000e+00], # [3.00000000e+00, 7.77156117e-16], # [5.00000000e-01, 0.00000000e+00]]), # array([1.72929250e-30, 4.38386909e-31]), 4, array([3.60116183, 2.60171483, 1.07908918, 0.50695163]))  As you can see, 2 questions (2 polynomials to guess), so 2 answers: 2 sets of coefficients, 2 sum of squared residuals (but only one rank and one singular values, since those are only specific to M, and there is only one M) So, that is why lstsq returns an array: it is the sum of squared residuals for all y we fit. If we fit only one y, then, there is only one sum in that array. Since Polynomial.fit cannot be called with a 2D-array as y, it is always in this case where these is only one sum in that array.
Question: I'm running an experiment on copy-on-write mechanism on Python Multiprocessing. I created a large file of 10GB and load the file into large_object in main. file_path = 'dummy_large_file.bin' try: large_object = load_large_object(file_path) print("File loaded successfully") except ValueError as e: print(e) except Exception as e: print(f"An error occurred: {e}") # Create a dummy file of 10GB for testing dummy_file_path = 'dummy_large_file.bin' with open(dummy_file_path, 'wb') as dummy_file: dummy_file.seek(10 * 1024 * 1024 * 1024 - 1) dummy_file.write(b"A") print(f"Dummy file created at {dummy_file_path}")  I intentionally pass the large_object to the multiprocessing workers to see the workers copy the large object. And to make sure multiprocessing copy, I modify the content of the object. # Multi-processing num_workers = 4 chunk_size = 10 processes = [] for i in range(num_workers): start = i * chunk_size p = multiprocessing.Process(target=process_chunk, args=(large_object, dummy_file_path, start, chunk_size)) processes.append(p) p.start() for p in processes: p.join() # Multi-processing def process_chunk(object, file_path, start, size): random_chunk = bytes([random.randint(0, 255) for _ in range(size)]) object[start:start + size] = random_chunk # Modify the chunk to a random binary number print(f"large object [{start}:{start + size}]: {object[start:start + size]}") # Process the chunk  The expectation here is the program should hold large_object of 10GB, and it copies the large_object to 4 other workers - which sums to 40GB. So the total memory should be 50GB. However, I'm only seeing the total memory of 20GB. Why there's only 20GB memory consumption? Does python implement some other lazy loading or granular copy for large objects?

Answer: linux knows nothing about python or how big its objects are. the computer divides memory into pages, a memory page is a contiguous block of memory, linux uses 4KB memory pages on most systems, that's the "unit" of memory that could get duplicated with COW when you write to it, if you modify a single byte then the entire 4KB page gets duplicated the address space is virtualized at the page level, just because you mapped 10GB worth of pages doesn't mean the computer will allocate 10 GB contiguous physical memory, you only get 10 GB of contiguous virtual address space, those addresses may not even be mapped to physical memory at all until you write to it. each of your 4 workers only writes to 2.5 GB, therefore only 2.5 GB worth of pages are duplicated into each worker, equating to the 10 GB you see. I think the only optimization that python does is that it doesn't attempt to serialize/deserialize the objects when launched with multiprocessing.Process, as it will be COWed by linux anyway. this is not true if you use other multiprocessing methods like Pool which needs to serialize/deserialize its arguments and you'd have 50GB used.
Question: Python PIL Image not working properly with overlay text image. I am trying to use FPDF image to convert an overlayed text png image to pdf file. However, the overlay text is not in expected colour (looks transparent) on a white background image. However, the same logic works in a zebra pattern background image. Source background image:   Text overlayed image:   Code used for overlayed text: Python Image AttributeError: 'ImageDraw' object has no attribute '__array_interface__' code: fontuse = "D:/Ubuntusharefolder/CustomFonts/EnglishFonts/NotoSans-Medium.ttf" font_color_bgra = (0,0,0,1)#black #(0,255,0,0)#green font = ImageFont.truetype(fontuse, 32 * int(2), layout_engine=ImageFont.LAYOUT_RAQM) src_img_use = np.array(Image.open(os.path.join(input_subfolder_path, filename) ) ) #(511, 898, 4) print('src_img_use gen size: ',os.path.join(input_subfolder_path, filename), src_img_use.shape) src_img_pil = Image.fromarray(src_img_use) print('src_img_pil gen size: ', src_img_use.size) img_pil_4dtxt = ImageDraw.Draw(src_img_pil) img_pil_4dtxt.text(textpos, subjectuse, font = font, fill = font_color_bgra) #fill = "black" ) src_img_pil.save(output_folder_final_path + '/updtxtimg/' + str(imgidx) + '_' + filename) print('label_img_withtxt drawtext saved subjectuse idx', imgidx, subjectuse)  Note sure whether there is a problem with input image or overlay code logic. The same image has to be used in FPDF.image but same behavior noticed. Updated Console logs: //zebra pattern image > & C:/ProgramData/Anaconda3/python.exe ./code/txtoverimgv1.py src_base imguse gen size: (1920, 1080, 3) input img file path: D:/testinput/order_image_1.png src_imguse gen size: (540, 360) saved img in path: D:/testinput/updtxtimg/upd_order_image_1.png <PIL.Image.Image image mode=RGB size=540x360 at 0x1293F0228E0> saved_img1 done //white background > & C:/ProgramData/Anaconda3/python.exe ./code/txtoverimgv1.py src_base imguse gen size: (1920, 1080, 3) input img file path: D:/testinput/order_image_1.png src_imguse gen size: (1920, 1080) saved img in path: D:/testinput/updtxtimg/upd_order_image_1.png <PIL.Image.Image image mode=RGBA size=1920x1080 at 0x2636D84B280> saved_img1 done  mode= RGB and RGBA differs for each image but both are png images Code update: Converting the image to RGB while opening solves the problem, but would like to know best way to work on RGBA images too. src_img_use = np.array(Image.open( input_subfolder_path + '/' + filename ).convert("RGB") ) 

Answer: The reason you are not seeing the text correctly on the RGBA is that you did not define the colors correctly. The text() method's parameter fill expects an integer-valued tuple, so (0,0,0,1) would result in a nearly transparent color (0=fully transparent, 255=fully opaque). Try again with font_color_bgra = (0,0,0,255) # black  This should resolve your issue. On the pure RGB image the alpha-component is not considered, thus you do not see the same error on the zebra background image.
Question: Suppose I have an RGB888 image (= 8 bit per color) in which the color of each pixel evenly encodes a (raw) data byte. Scheme: Channel red encodes Bit 7:6 (R: 0-63 = 0b00, 64-127 = 0b01, 128-191 = 0b10, 192-255 = 0b11) Channel green encodes Bit 5:3 (G: 0-31 = 0b000, 32-63 = 0b001, 64-95 = 0b010, 96-127 = 0b011, ...) Channel blue encodes Bit 2:0 (B: 0-31 = 0b000, 32-63 = 0b001, 64-95 = 0b010, 96-127 = 0b011, ...) Example: RGB = [150, 40, 94] represents (raw) data byte 0b10001010. Question: I would like to use the Python Imaging Library (PIL) to decode my image. If I got it right, then I would need to define a palette and apply it to my image using the quantize() method (as described here). However, I don't understand how the mapping of the colors does work. How does the palette translate the original 24 Bits of color information into 8 Bits? Finally: How would I need to set up my palette so that the decoding works according to my scheme? Edit: My image does not contain a subject. It contains data. Below is an example ("Data" is what I like to decode using PIL). 

Answer: I think your question maybe means something different altogether. I think you want to convert ranges of numbers into small integers, i.e. 0-31 becomes 0, 32-63 becomes one. So you just need to shift your values to the right and you will lose the less significant bits. If your image is this: import numpy as np array([102, 220, 225, 95, 179, 61, 234, 203, 92, 3], dtype=uint8)  Then, because 2^6 is 64, you can do get integers in range 0..3, with: partA = image >> 6  which will give you: array([1, 3, 3, 1, 2, 0, 3, 3, 1, 0], dtype=uint8)  Or , if image is three channels and you want to extract from the Red channel, use: partA = image[...,0] >> 6  Similarly, if image is three channels and you want to extract from the Blue channel, use: partA = image[...,2] >> 6   And, because 2^5 is 32, you can get integers in range 0..7, with: partB = image >> 5  which will give you: array([3, 6, 7, 2, 5, 1, 7, 6, 2, 0], dtype=uint8)   Now you can reconstruct your (hidden?) byte by shifting and ORing together the parts: reconstructed = (partA << 6) | (partB << 3) | partC  Once you have your data in a Numpy array called reconstructed toy can make it into a PIL Image with: from PIL import Image pImage = Image.fromarray(reconstructed)
Question: I need to calculate the floor of a localized timestamp with daily resolution, but I get an exception when the daylight saving time starts. >>> pd.Timestamp('2024-09-08 12:00:00-0300', tz='America/Santiago').floor("D") NonExistentTimeError: 2024-09-08 00:00:00  I understand that midnight does not exist on that day, clocks are moved to 1am after 11:59pm. Still I would have expected floor to return pd.Timestamp('2024-09-08 01:00:00-0300', tz='America/Santiago'). The same happens with ceil applied to the previous day: >>> pd.Timestamp('2024-09-07 12:00:00-0300', tz='America/Santiago').ceil("D") NonExistentTimeError: 2024-09-08 00:00:00  I made two attempts at solving this but I either get the same exception or the wrong answer: >>> pd.Timestamp('2024-09-08 12:00:00').floor("D").tz_localize('America/Santiago') NonExistentTimeError: 2024-09-08 00:00:00 >>> pd.Timestamp('2024-09-08 12:00:00-0300').floor("D").tz_convert('America/Santiago') Timestamp('2024-09-07 23:00:00-0400', tz='America/Santiago') # Wrong answer 

Answer: By default a non-existent time will raise an error. There is an nonexistent option in Timestamp.floor to shift the time forward/backward: (pd.Timestamp('2024-09-08 12:00:00-0300', tz='America/Santiago') .floor('D', nonexistent='shift_backward') ) # Timestamp('2024-09-07 23:59:59-0400', tz='America/Santiago') (pd.Timestamp('2024-09-08 12:00:00-0300', tz='America/Santiago') .floor('D', nonexistent='shift_forward') ) # Timestamp('2024-09-08 01:00:00-0300', tz='America/Santiago')  Or, to get 23:00 passing pd.Timedelta('-1h') (not sure of the relevance to do this): (pd.Timestamp('2024-09-08 12:00:00-0300', tz='America/Santiago') .floor('D', nonexistent=pd.Timedelta('-1h')) ) # Timestamp('2024-09-07 23:00:00-0400', tz='America/Santiago')
Question: I have a Python source file with some dummy code: a = 3 if a == 1: print("a = 1") elif a == 2: print("a = 2") else: print("Other")  When I submit the code to terminal with shift+enter, I get the following error. It looks like VS Code changed the indentation of my code. The same code ran just fine with shift+enter on my other computer. The error message is as below: PS C:\Users\win32> & C:/Users/win32/AppData/Local/Programs/Python/Python313/python.exe Python 3.13.0 (tags/v3.13.0:60403a5, Oct 7 2024, 09:38:07) [MSC v.1941 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> a = 3 >>> if a == 1: ... print("a = 1") ... elif a == 2: ... print("a = 2") ... else: ... print("Other") ... File "<python-input-1>", line 3 elif a == 2: ^^^^ SyntaxError: invalid syntax  Any insights?

Answer: bug: Python3.13 All functions fail to send: IndentationError: unexpected indent Use python 3.12
Question: How can I place the icon after the text of a QPushButton in PyQt5? The icon should not come before the text - it should come after or below the text. Currently it's looking like this:  but it should be like this: Track ⨁ from PyQt5.QtWidgets import QApplication, QPushButton, QWidget, QVBoxLayout from PyQt5.QtGui import QIcon from PyQt5.QtCore import Qt import sys app = QApplication(sys.argv) window = QWidget() layout = QVBoxLayout(window) btn_add_track = QPushButton("Track") btn_add_track.setIcon(QIcon("Icons/plus.png")) layout.addWidget(btn_add_track) window.show() sys.exit(app.exec_()) 

Answer: The simplest option is to change the direction of the button layout: btn_add_track.setLayoutDirection(Qt.LayoutDirection.RightToLeft)  Unlike the other solution, this will maintain the appearance of the original button such as centering the icon text and text relative to the button (this can also be done with the other solution but requires much more code)
Question: This is my dataframe: df = pd.DataFrame({ 'a': [0, 0, 1, -1, -1, 0, 0, 0, 0, 0, -1, 0, 0, 1, 0] })  Expected output is creating column b:  a b 0 0 0 1 0 0 2 1 0 3 -1 1 4 -1 -1 5 0 -1 6 0 -1 7 0 -1 8 0 0 9 0 0 10 -1 0 11 0 -1 12 0 -1 13 1 -1 14 0 1  Logic: I explain the logic by some examples: I want to create column b to df I want to have a window of three rows for example for row number 3 I want to look at three previous rows and capture the last non 0 value. if all of the values are 0 then 'b' is 0. in this case the last non zero value is 1. so column b is 1 for example for row number 4 . The last non zero value is -1 so column b is -1 I want to do the same for all rows. This is what I have tried so far. I think there must be a better way. import pandas as pd df = pd.DataFrame({ 'a': [0, 0, 1, -1, -1, 0, 0, 0, 0, 0, -1, 0, 0, 1, 0] }) def last_nonzero(x): # x is a pandas Series representing a window nonzero = x[x != 0] if not nonzero.empty: # Return the last non-zero value in the window (i.e. the one closest to the current row) return nonzero.iloc[-1] return 0 # Shift by 1 so that the rolling window looks only at previous rows. # Use a window size of 3 and min_periods=1 to allow early rows. df['b'] = df['a'].shift(1).rolling(window=3, min_periods=1).apply(last_nonzero, raw=False).astype(int) 

Answer: I don't think there is a much more straightforward approach. There is currently no rolling.last method. You could however simplify a bit your code: def last_nonzero(s): return 0 if (x:=s[s != 0]).empty else x.iloc[-1] df['b'] = (df['a'].shift(1, fill_value=0) .rolling(window=3, min_periods=1).apply(last_nonzero) .convert_dtypes() )  With a lambda: df['b'] = (df['a'].shift(1, fill_value=0) .rolling(window=3, min_periods=1) .apply(lambda s: 0 if (x:=s[s != 0]).empty else x.iloc[-1]) .convert_dtypes() )  Actually, if you have a range index, you could also use a merge_asof on the indices: window = 3 out = pd.merge_asof( df, df['a'].shift(1, fill_value=0).loc[lambda x: x != 0].rename('b'), left_index=True, right_index=True, tolerance=window-1, direction='backward', ).fillna({'b': 0})  Output:  a b 0 0 0 1 0 0 2 1 0 3 -1 1 4 -1 -1 5 0 -1 6 0 -1 7 0 -1 8 0 0 9 0 0 10 -1 0 11 0 -1 12 0 -1 13 1 -1 14 0 1
Question: I want to match and substitute for strings as shown in the example below, but not for some strings which start with test or !!. I have used negative lookahead to skip matching unwanted strings but (Street|,)(?=\d) matching for Street & comma replacing group 1 with UK/ is not working as expected. import re input = [ 'Street1-2,4,6,8-10', '!! Street4/31/2', 'test Street4' ] pattern = r'(^(?!test\s|!!\s).*(Street|,)(?=\d))' output = [re.sub(pattern, r'\g<1>UK/', line) for line in input ]  Actual output: ['Street1-2,4,6,UK/8-10', '!! Street4/31/2', 'test Street4']  Expected output: ['StreetUK/1-2,UK/4,UK/6,UK/8-10', '!! Street4/31/2', 'test Street4'] 

Answer: You could change the pattern to use 2 capture groups, and then use a callback with re.sub. The callback checks if there is a group 1 value. If there is, use it in the replacement, else use group 2 followed by UK/ ^((?:!!|test)\s.*)|(Street|,)(?=\d)  The regex matches  ^((?:!!|test)\s.*) Capture either !! or test at the start of the string followed by a whitespace char and then the rest of the line in group 1 | Or (Street|,)(?=\d) Capture either Street or , in group 2 while asserting a digit to the right  See a regex101 demo import re lst = ['Street1-2,4,6,8-10', '!! Street4/31/2', 'test Street4'] pattern = r'^((?:!!|test)\s.*)|(Street|,)(?=\d)' output = [re.sub(pattern, lambda m: m.group(1) or m.group(2) + 'UK/', line) for line in lst] print(output)  Output ['StreetUK/1-2,UK/4,UK/6,UK/8-10', '!! Street4/31/2', 'test Street4']
Question: I have a pandas Series s, and when I call s.std(skipna=True) and s.std(skipna=False) I get different results even when there are no NaN/null values in s, why? Did I misunderstand the skipna parameter? I'm using pandas 1.3.4 import pandas as pd s = pd.Series([10.0]*4800000, index=range(4800000), dtype="float32") # No NaN/null in the Series print(s.isnull().any()) # False print(s.isna().any()) # False # Why the code below prints different results? print(s.std(skipna=False)) # 0.0 print(s.std(skipna=True)) # 0.61053276 

Answer: This is an issue with the Bottleneck optional dependency, used to accelerate some NaN-related routines. I think the wrong result happens due to loss of precision while calculating the mean, since Bottleneck uses naive summation, while NumPy uses more accurate pairwise summation. You can disable Bottleneck with pd.set_option('compute.use_bottleneck', False)  to fall back to the NumPy handling.