eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I want to do something along the lines of... import numpy as np arr = np.linspace(0, 10, 100) s = slice(1, 10) print(arr[s]) print(arr[~s])  How could I apply the "not" operator to a slice, so that in this case arr[~s] would be the concatenation of arr[0] and arr[10:]?

Answer: np.delete will do what you want: np.delete(arr, s)  For something more complex than a slice, you may want to store the index. To do that, you can invert a mask built from the slice: mask = np.ones(arr.shape, dtype=bool) mask[s] = False arr[mask]  OR mask = np.zeros(arr.shape, dtype=bool) mask[s] = True arr[~mask]  You can also use np.delete on an index array: arr[np.delete(np.arange(arr.size), s)]  This gets a bit more complex if you have a multidimensional array. For example, you would probably use np.indices instead of np.arange.
Question: I am practicing on using the super function and dataclass inheritance in general. I have enabled the kw_only attribute for cases when the parent class has default values. I completely understand that super doesn't need to be used in a dataclass if you're just passing variables and I can avoid using super here. My goal is to understand the super feature better through this example. I can't understand the error message I'm getting though. @dataclass(kw_only=True) class ZooAnimals(): food_daily_kg: int price_food: float area_required: float name: str c = ZooAnimals(food_daily_kg=565, price_food=40, area_required=10, name='Monkey' ) print(c) @dataclass(kw_only=True) class Cats(ZooAnimals): meowing: str def __init__(self, food_daily_kg, price_food, area_required, meowing, name): self.meowing = meowing super().__init__(food_daily_kg, price_food, area_required, name) z = Cats(food_daily_kg=465, price_food=30, area_required=10, meowing='Little Bit', name='Leopard' ) print(z)  Output: ZooAnimals(food_daily_kg=565, price_food=40, area_required=10, name='Monkey') TypeError: ZooAnimals.__init__() takes 1 positional argument but 5 were given 

Answer: You shouldn't define an __init__ method in a data class if you don't have any custom initialization logics. And if you do have custom initialization logics, you should define them in a __post_init__ method instead. Your code produces the error because the __init__ method of the subclass calls the __init__ method of the base class with positional arguments when the method is configured to accept keyword arguments only with your kw_only=True option. You can fix it by passing keyword arguments instead, or by simply removing the __init__ method from the subclass entirely since one would be generated for the data class with inheritance in mind already. For example, this subclass definition would work just fine: @dataclass(kw_only=True) class Cats(ZooAnimals): meowing: str  Demo here Or if you would still like to define a custom __init__ method for some reason: @dataclass(kw_only=True) class Cats(ZooAnimals): meowing: str def __init__(self, food_daily_kg, price_food, area_required, meowing, name): self.meowing = meowing super().__init__( food_daily_kg=food_daily_kg, price_food=price_food, area_required=area_required, name=name)  Demo here
Question: Trying to figure out how to transform a k-v string that is inside a column where the k-v string is separated by commas, and could contain different keys. The different keys would then be transformed into their own columns, where missing values would contain nulls. For example, pl.DataFrame({ "apple": [1, 2, 3], "data": ["a=b, b=c", "a=y, y=z", "k1=v1, k2=v2"] })  would look like: pl.DataFrame({ "apple": [1, 2, 3], "a": ["b", "y", None], "b": ["c", None, None], "y": [None, "z", None], "k1": [None, None, "v1"], "k2": [None, None, "v2"], "data": ["a=b, b=c", "a=y, y=z", "k1=v1, k2=v2"] })  once transformed. Does anyone know what is the most efficient way to do this (perhaps without pre-processing of the data, if possible?) 

Answer: You could attempt to reformat it as JSON objects. df.with_columns(pl.format('{"{}"}', pl.col("data").str.replace_many({"=": '":"', ", ": '","'}) ))  shape: (3, 3) ┌───────┬──────────────┬───────────────────────┐ │ apple ┆ data ┆ literal │ │ --- ┆ --- ┆ --- │ │ i64 ┆ str ┆ str │ ╞═══════╪══════════════╪═══════════════════════╡ │ 1 ┆ a=b, b=c ┆ {"a":"b","b":"c"} │ │ 2 ┆ a=y, y=z ┆ {"a":"y","y":"z"} │ │ 3 ┆ k1=v1, k2=v2 ┆ {"k1":"v1","k2":"v2"} │ └───────┴──────────────┴───────────────────────┘  Which would allow you to .str.json_decode() into a struct. And then .unnest() into columns. df.with_columns( pl.format('{"{}"}', pl.col("data").str.replace_many({"=": '":"', ", ": '","'}) ) .str.json_decode() .alias("json") ).unnest("json")  shape: (3, 7) ┌───────┬──────────────┬──────┬──────┬──────┬──────┬──────┐ │ apple ┆ data ┆ a ┆ b ┆ y ┆ k1 ┆ k2 │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ i64 ┆ str ┆ str ┆ str ┆ str ┆ str ┆ str │ ╞═══════╪══════════════╪══════╪══════╪══════╪══════╪══════╡ │ 1 ┆ a=b, b=c ┆ b ┆ c ┆ null ┆ null ┆ null │ │ 2 ┆ a=y, y=z ┆ y ┆ null ┆ z ┆ null ┆ null │ │ 3 ┆ k1=v1, k2=v2 ┆ null ┆ null ┆ null ┆ v1 ┆ v2 │ └───────┴──────────────┴──────┴──────┴──────┴──────┴──────┘
Question: I want to find the daytime and night-time occurrence duration of an event from its' start time to end time. The event duration is volatile and can take place for a long time. I can't figure out a formula. I am seeking to figure out the duration with a formula or VBA code or Python code. Link to the sample file with manually calculated duration for each event. The event occurs in multiple scenarios to be considered. 

Answer: If you find it hard to calculate NightTime, I suggest you calculate DayTime first, and use TotalHourse - DayTime for night time, then you wouldn't need to worry about "crossing midnight". So, for day time duration, here is the formula (in K10) =(INT(E10)-INT(D10))*(F10-G10)-MEDIAN(MOD(D10,1),G10,F10)+MEDIAN(MOD(E10,1),G10,F10) For night time duration: =E10-D10-K10 (in L10)  Finally, remember to format your cell as [h]:mm:ss:  Explaining the formula:  think of INT(E10)-INT(D10) as number of days between start date and end date. (including start date, but excluding end date).  INT(E10)-INT(D10) * (18 -6) can be interpreted as number of dayTime hours from start date to end date. (start date inclusive, end date exclusive)  To get number of hours between 12/09/2024 7:00 and 13/09/2024 16:00, we need to deduct extra hours from start date, and add extra hours to the end date. i.e we need to deduct (7:00-6:00) from 12/09, and add (16:00 -6:00) for 13/09. that is, (13/09/2024 - 12/09/2024) * (18 -6) -(7-6) + (16-6) is the number of dayTime hours between 12/09/2024 7:00 to 13/09/2024 16:00. It can be simplified to (13/09/2024 - 12/09/2024) * (18 -6) + 16 -7  If start time on start date is outside DayTime Hours, e.g for 12/09/2024 4:00 to 13/09/2024 16:00, we just treat it as ifstart date starts at 6:00. i.e (13/09/2024 - 12/09/2024) * (18 -6) - (6-6) + (16 - 6) = (13/09/2024 - 12/09/2024) * (18 -6) + 16 -6  If end time on end date is outside DayTime hours, e. for  12/09/2024 4:00 to 13/09/2024 19:00, we just treat it as if end date ends at 18:00. i.e (13/09/2024 - 12/09/2024) * (18 -6) - (6-6) + (18 - 6) = (13/09/2024 - 12/09/2024) * (18 -6) + 18 -6  Finally, 4 & 5 can be simplified with -MEDIAN(start hour,6,18)+MEDIAN(end hour, 6, 18)
Question: I compiled a C binary on a linux machine and executed it, in that binary I have a struct called Location defined as follows typedef struct { size_t x; size_t y; } Location;  and here is my main function int main(void) { srand(0); Location loc; while (1) { loc.x = rand()%10; loc.y = rand()%10; sleep(2); } return 0; }  How do I monitor the values of x and y? There are some limitations to consider  I can't modify the binary code monitoring should be done with python ASLR always enabled  Things I tried  Reading /proc/pid/maps location stack then reading /proc/pid/mem didn't find anything I used gdb to find the address of loc but it is outside the range of stack found in maps (most probably ASLR) 

Answer: We know the location is located in the stack somewhere, open maps file calculated stack size, start and end address, then open memory file in bytes mode and read stack bytes, loop over the bytes until you find a bytes sequence that maps to a given struct. PS I have to add a third attribute to struct for sake of making the struct easier to find, will remove it in my actual code. import sys import ctypes class Location(ctypes.Structure): _fields_ = [ ("x", ctypes.c_size_t), ("y", ctypes.c_size_t), ("z", ctypes.c_size_t) ] pid = sys.argv[1] with open(f"/proc/{pid}/maps", "r") as f: lines = f.readlines() for line in lines: if "[stack]" in line: start, end = line.split()[0].split("-") start = int(start, 16) end = int(end, 16) loc_size = ctypes.sizeof(Location) with open (f"/proc/{pid}/mem", "rb") as f: f.seek(start) data = f.read(end-start) print (len(data)) for i in range(0, len(data), loc_size): chunk = data[i:i+loc_size] if len (chunk) < loc_size: continue location = Location.from_buffer_copy(chunk) if location.z == 1337: print (f"\tx: {location.x}") print (f"\ty: {location.y}") print (f"\tz: {location.z}")
Question: I have built OpenCV4 on Windows 11. When I attempt to use the superres module by Python, I encountered an error: module 'cv2' has no attribute 'superres' import cv2 model = cv2.superres.createSuperResolution_BTVL1()  I had included the superres module when building OpenCV. Cmake info:  Then I found using C++ is ok. So can this module only be used in C++? #include <iostream> #include <cstring> #include "opencv2/imgproc.hpp" #include "opencv2/imgcodecs.hpp" #include "opencv2/superres.hpp" int main(int argc, char* argv[]) { // no error. cv::superres::createSuperResolution_BTVL1(); return 0; } 

Answer: The Python wrappers for OpenCV are generally automatically generated (there are some hand-written ones, but that's a tiny fraction). In order for this to happen, annotations in form of macros have to present in the header file(s) of the respective module. For free-standing functions and classes this is usually CV_EXPORTS_W and friends, for class member functions some variation of CV_WRAP, and there are some more that may need to be used for function arguments. I've described a few in an answer to this question, and the OpenCV documentation contains some useful bits as well. If we examine the header of superres module, we can se no such annotations (CV_EXPORTS is just a generic export/import doohickey, no wrappers are generated there). The superres module was moved out from main to contrib during transition to 4.0, apparently due to lack of maintenance. When I look at the 3.x branch, I can't find any annotations either. I didn't dig any deeper, so who knows, it might have had something available in 2.x, but that's not really relevant today. Generally, when Python wrappers are generated, so is a function signature for the documentation. None can be seen in the documentation, so that's some more evidence to support our conclusion.  Based on the above, this module's functionality is not available in the standard Python bindings for OpenCV. That does not necessarily mean it's available only from C++, since some bindings for other languages are implemented differently, but since this is an unmaintained contrib module, it's likely the case. Possible solutions:  As Christoph mentioned, have a look at dnn_superres -- that has Python bindings available. Adding the annotations to superres likely wouldn't be too difficult, if you took inspiration from some other modules that have them on classes. I have no idea how easy it would be to get such a pull request accepted there, but might be worth a try, unless you want to ship your own custom build with your program. I suppose you could try to use it via ctypes, but given it's C++, it's like opening another can of worms, and portability goes outta the window. You could write a thin wrapper around the needed functionality using Boost.Python or pybind11. Both have support for numpy arrays, so it shouldn't be too hard to make it play nice with regular OpenCV Python code.
Question: Is it possible to have static type checking or IDE intelligence support for a numpy array/matrix shape? For example, if I imagine something like this: A_MxN: NDArray(3,2) = ... B_NxM: NDArray(2,3) = ...  Even better would be: N = 3 M = 2 A_MxN: NDArray(M,N) = ... B_NxM: NDArray(N,M) = ...  And if I assign A to B, I would like to have an IDE hint during development time (not runtime), that the shapes are different. Something like: A_MxN = B_NxM Hint/Error: Declared shape 3,2 is not compatible with assigned shape 2,3  As mentioned by @simon, this seems to be possible: M = Literal[3] N = Literal[2] A_MxN: np.ndarray[tuple[M,N], np.dtype[np.int32]]  But if I assign an array which does not fulfill the shape requirement, the linter does not throw an error. Does someone know if there is a typechecker like mypy or pyright which supports the feature?

Answer: It's possible to type the shape of an array, like was mentioned before. But at the moment (numpy 2.1.1), the shape-type of the ndarray is lost in most of numpys own functions. But the shape-typing support is gradually improving, and I'm actually personally involved in this. But that doesn't mean that you can't use shape-typing yet. If you write a function yourself, you can add shape-typing support to it without too much hassle. For instance, in Python 3.12 (with PEP 695 syntax) you can e.g. do: from typing import Any import numpy as np def get_shape[ShapeT: tuple[int, ...]](a: np.ndarray[ShapeT, Any]) -> ShapeT: return a.shape  This will be valid in all static type-checkers with numpy>=2.1. If this current syntax is too verbose for you, you could use the lightweight optype (of which I'm the author) to make it more readable: from optype.numpy import Array, AtLeast0D def get_shape[ShapeT: AtLeast0D](a: Array[ShapeT]) -> ShapeT: return a.shape  Array is a handy alias for ndarray, that uses (PEP 696) type parameter defaults. See the docs if you want to know the details.
Question: I am using the following code to generate a plot with a sine curve marked with 24 'hours' over 360 degrees. Each 'hour' is annotated, however the arrow lengths decrease (shrivel?) with use and even their direction is incorrect. The X axis spans 360 degrees whereas the Y axis spans 70 degrees. The print statement verifies that the arrows on 6 and 18 hours have the same length and are vertical, according to the offsets specified. This is not so as seen in the resulting plot:  Matplotlib version = 3.9.2; Numpy version = 2.1.1 Here is the python code: #!/usr/bin/env python3 # -*- coding: utf-8 -*- import numpy as np import matplotlib.pyplot as plt arrow_style = {'arrowstyle': '-|>'} def make_plot(): fig, ax = plt.subplots() ax.axis([0,+360,-35,+35]) ax.set(xlabel = 'X (degrees)', ylabel = 'Y (degrees)', title='Vanishing arrow length example') degree = np.pi / 180.0 # degrees to radians arrow_angle = 72.0 # degrees arrow_length = 27.0 eps = 23.43927945 eps_x = np.linspace(0, 360, 200) eps_y = -eps * np.sin(2 * np.pi * eps_x / 360) ax.plot(eps_x, eps_y, 'r') hr_x = np.linspace(0, 360, 25) hr_y = -eps * np.sin(2 * np.pi * hr_x / 360) i = 0 for x in hr_x: ax.plot(hr_x[i], hr_y[i],'bo') if hr_y[i] > 0 and arrow_angle > 0: arrow_angle = -75 # degrees arrow_x = np.cos(arrow_angle*degree) * arrow_length arrow_y = np.sin(arrow_angle*degree) * arrow_length ax.annotate(str(i), xy=(hr_x[i], hr_y[i]), xytext=(arrow_x, arrow_y), \ xycoords='data', textcoords='offset points', arrowprops=arrow_style) print("arrow_x {:.2f} arrow_y {:.2f} arrow_angle {:.1f} at {} hours" \ .format(arrow_x, arrow_y, arrow_angle, i)) if hr_y[i] <= 0: arrow_angle += 3.0 if hr_y[i] > 0: arrow_angle -= 3.0 i += 1 ax.grid() ax.axhline(0.0) ax.axvline(180.0) plt.show() return fig make_plot().savefig('vanishing_arrow_length.png')  I have only a few days experience with matplotlib so I guess this is a really simple user error. However the documentation is no help to me in this case.

Answer: By default the text is aligned at its bottom left corner. You might want to change this to align at its centre. ax.annotate(str(i), xy=(hr_x[i], hr_y[i]), xytext=(arrow_x, arrow_y), xycoords='data', textcoords='offset points', arrowprops=arrow_style, verticalalignment='center', horizontalalignment='center')
Question: I would like to merge two dataframes which have only one column say df1 and df2 as below. Expected output dataframe as df3. import pandas as pd data1 = [ 'A', 'B', 'C'] df1 = pd.DataFrame(data1,columns=['name']) data2 = [ 'B', 'C', 'D'] df2 = pd.DataFrame(data2,columns=['name']) data3 = [ ['A',], ['B','B'], ['C','C'], [None,'D']] df3 = pd.DataFrame(data3,columns=['name_x','name_y']) print(df1) print(df2) print(df3)  Output:  name 0 A 1 B 2 C name 0 B 1 C 2 D name_x name_y 0 A None 1 B B 2 C C 3 None D  Should I use merge to do it or any other way?

Answer: Quick hack, you could merge passing one of the keys as Series: df1.merge(df2, left_on='name', right_on=df2['name'], how='outer').drop(columns='name')  Output:  name_x name_y 0 A NaN 1 B B 2 C C 3 NaN D  If you don't drop you'll also get a column with the merged values: df1.merge(df2, left_on='name', right_on=df2['name'], how='outer') name name_x name_y 0 A A NaN 1 B B B 2 C C C 3 D NaN D  If you had extra columns in the input, this would give you: df1.merge(df2, left_on='name', right_on=df2['name'], how='outer') name name_x col1 name_y col2 0 A A x NaN NaN 1 B B x B x 2 C C x C x 3 D NaN NaN D x  generic method If you have more than two inputs to combine, you can generalize with concat: dfs = [df1, df2, df1, df2] out = pd.concat( [ d.set_axis(d['name']).add_suffix(f'_{i}') for i, d in enumerate(dfs, start=1) ], axis=1, ).reset_index(drop=True)  Output:  name_1 name_2 name_3 name_4 0 A NaN A NaN 1 B B B B 2 C C C C 3 NaN D NaN D
Question: [edited] It appears there is a new bug in opencv that introduces an issue causing fillPoly's boundaries to exceed polylines's. Here is humble code to draw a red filled polygon with a blue outline. import cv2 import numpy as np def draw_polygon(points, resolution=50): # create a blank black canvas img = np.zeros((resolution, resolution, 3), dtype=np.uint8) pts = np.array(points, np.int32) pts = pts.reshape((-1, 1, 2)) # draw a filled polygon in blue cv2.fillPoly(img, [pts], (0, 0, 255)) # draw an outline in red cv2.polylines(img, [pts], True, (255, 0, 0), 1) # show the image cv2.imshow("Polygon", img) cv2.waitKey(0) cv2.destroyAllWindows() # why is the infill outside the line? if __name__ == "__main__": # 4 vertices of the quad (clockwise) quad = np.array([[[44, 27], [7, 37], [7, 19], [38, 19]]]) draw_polygon(quad)  QUESTION The polygon's infill appears to bleed outside of the outline (two highlighted pixels). I'm looking for a temporary solution until this bug is addressed so the infill stays completely inside the outline. Solution has to work with concave polygons. 

Answer: Where there's a will there's a way. fillPoly appears to be able to draw as a line when given only two vertices. And that line matches the edges of the previously drawn polygon. \o/ I modified my code to draw the edges as a single fillPoly call and it seems to work decently. I would still prefer if fillPoly and polyLines would match, but for now I am unblocked. import cv2 import numpy as np def draw_polygon(points, resolution=50): # create a blank black canvas img = np.zeros((resolution, resolution, 3), dtype=np.uint8) poly_pts = np.array(points, np.int32) edge_pts = np.vstack([poly_pts, poly_pts[0]]) # draw a filled polygon in blue cv2.fillPoly(img, [poly_pts], (0, 0, 255)) # draw the outlines as individual edges, # drawn in a single fillPoly call, in red e0 = edge_pts[:-1] e1 = edge_pts[1:] edge_polygons = np.hstack((e0[:,None], e1[:,None])) cv2.fillPoly(img, edge_polygons, (255, 0, 0)) # show the image cv2.imshow("Polygon", img) cv2.waitKey(0) cv2.destroyAllWindows() # better outline if __name__ == "__main__": # 4 vertices of the quad (clockwise) quad = np.array([[44, 27], [7, 37], [7, 19], [38, 19]]) draw_polygon(quad)
Question: I have a tuple, RELAY_PINS that holds GPIO pin numbers in the order that the relays are installed. RELAY_PINS is immutable, and its ordering does not change, while the order that the devices are defined in changes frequently. The MRE: from random import shuffle, randint class Device: def __init__(self, pin_number): self.pin_number = pin_number def __str__(self): return str(self.pin_number) RELAY_PINS = ( 14, 15, 18, 23, 24, 25, 1, 12, 16, 20, 21, 26, 19, 13, 6, 5 ) def MRE(): devices = [ Device(pin) for pin in RELAY_PINS ] # the ordering for the list of devices should be considered random for the sake of this question shuffle(devices) return devices  My solution "works", but frankly, it's embarrassing: def main(): devices = MRE() pin_map = { pin_number : index for index, pin_number in enumerate(RELAY_PINS) } ordered_devices = [ None for _ in range(len(RELAY_PINS)) ] for device in devices: index = pin_map[device.pin_number] ordered_devices[index] = device return [ dev for dev in ordered_devices if dev is not None ]  I know there is a better solution, but I can't quite wrap my head around it. What is the pythonic solution to this problem?

Answer: You can use sorted with a key function: from random import randint, shuffle class Device: def __init__(self, pin_number: int): self.pin_number = pin_number def __str__(self) -> str: return str(self.pin_number) def __repr__(self) -> str: return f'Device(pin_number={self.pin_number})' RELAY_PINS: tuple[int, ...] = (14, 15, 18, 23, 24, 25, 1, 12, 16, 20, 21, 26, 19, 13, 6, 5) def MRE() -> None: devices = [Device(pin) for pin in RELAY_PINS] devices.pop(randint(0, len(RELAY_PINS) - 1)) shuffle(devices) return devices def main() -> None: devices = MRE() ordered_devices = sorted(devices, key=lambda d: RELAY_PINS.index(d.pin_number)) print(ordered_devices) if __name__ == '__main__': main()  Example Output (device with pin_number=12 randomly popped): [Device(pin_number=14), Device(pin_number=15), Device(pin_number=18), Device(pin_number=23), Device(pin_number=24), Device(pin_number=25), Device(pin_number=1), Device(pin_number=16), Device(pin_number=20), Device(pin_number=21), Device(pin_number=26), Device(pin_number=19), Device(pin_number=13), Device(pin_number=6), Device(pin_number=5)]
Question: Here is some random code. # create df import pandas as pd df2 = pd.DataFrame({'var1':['1_0','1_0','1_0','1_0','1_0'], 'var2':['X','y','a','a','a']}) df2.to_json('df2.json') # import df df2 = pd.read_json('df2.json') df2  This is the expected output:  var1 var2 0 1_0 X 1 1_0 y 2 1_0 a 3 1_0 a 4 1_0 a  However it generates:  var1 var2 0 10 X 1 10 y 2 10 a 3 10 a 4 10 a  If I modify an entry inside ['var1'] to a string, then the code it generates when df is imported is correct. Here is an example to illustrate it. I replaced one of the entries with 'hello' df2 = pd.DataFrame({'var1':['1_0','hello','1_0','1_0','1_0'], 'var2':['X','y','a','a','a']}) df2.to_json('df2.json') # import df df2 = pd.read_json('df2.json') df2  Generates this  var1 var2 0 1_0 X 1 hello y 2 1_0 a 3 1_0 a 4 1_0 a  Same problem is observed if file is saved in csv format and then imported. Has anyone encountered the same issue?

Answer: This is due to the fact that underscores are valid separators in python (often used as thousand separator: 1_000 is 1000). You could force the dtype upon import (or use dtype=False): df2 = pd.read_json('df2.json', dtype='str')  If you want to keep dtype detection for the other columns: df2 = pd.read_json('df2.json', dtype={'var1': 'str'})  Output:  var1 var2 0 1_0 X 1 1_0 y 2 1_0 a 3 1_0 a 4 1_0 a  When you have a string in the json, there is no ambiguity that the values are not numbers and the conversion is not done.