eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a piece of code that was working fine until last week, but now it's failing with the following error: AttributeError: module 'fiona' has no attribute 'path' I’ve ensured that all the necessary libraries are installed and imported. Does anyone have any ideas on what might be going wrong or how I can resolve this issue? Thanks!  pip install geopandas pip install fiona import geopandas as gpd import fiona countries = gpd.read_file(gpd.datasets.get_path("naturalearth_lowres")) 

Answer: TL;DR update to geopandas==0.14.4 OR pin fiona to version 1.9.6 -- It seems fiona recently upgraded to 1.10.0 (as of 2024-09-04 01:14 UTC) and that may have broken some older versions of geopandas, which only depend on fiona being higher than some version, not lower than. Upon closer look, geopandas up to version 0.14.3 still calls fiona.path, but in version 0.14.4 it no longer does. So upgrading geopandas to 0.14.4 should fix it. Alternatively, forcing fiona to stay on version 1.9.6 should also work. NOTE: upgrading geopandas to >=1.0 seems to remove fiona as a dependency altogether, so it will also solve this issue. But it opens up a whole new can of worms by removing geopandas.dataset. For details on that one, see How to get maps to geopandas after datasets are removed?
Question: I am attempting to create a 'counter' of consecutive binary values = 1, resetting when the binary value = 0, for each group. Example of data: data = {'city_id': [1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 5, 5, 5, 5, 5, 5, 5, 6, 6, 6, 6, 6, 6, 6], 'week': [1, 2, 3, 4, 5, 6, 7, 1, 2, 3, 4, 5, 6, 7, 1, 2, 3, 4, 5, 6, 7, 1, 2, 3, 4, 5, 6, 7, 1, 2, 3, 4, 5, 6, 7, 1, 2, 3, 4, 5, 6, 7], 'binary': [0, 1, 1, 1, 0, 0, 0, 0, 1, 1, 0, 1, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 1]} df = pd.DataFrame(data)  For each id, the first binary = 1 should begin with a consecutive_count = 1 rather than 0. And this should reset each time binary = 0, along with each time we move on to a new id. I have already created a solution that does this. It looks like this: df['consecutive'] = 0 for city in df['city_id'].unique(): city_df = df[df['city_id'] == city] consecutive_count = 0 for i in range(len(city_df)): if city_df['binary'].iloc[i] == 1: consecutive_count += 1 else: consecutive_count = 0 df.loc[(df['city_id'] == city) & (df['week'] == city_df['week'].iloc[i]), 'consecutive'] = consecutive_count  The main issue is that my solution is extremely inefficient for large data. I have a large set of ids, ~2.5M, and this solution either times out or runs for hours and hours, so I am struggling in making this more efficient. TIA.

Answer: The issue with your approach is that you're repeatedly slicing. You should use the builtin groupby functions for efficiency. You can form a custom group with groupby.cumsum to reset the groups on 0s, then use this to compute the consecutive counts: df['consecutive'] = df.groupby( ['city_id', df['binary'].eq(0).groupby(df['city_id']).cumsum()] )['binary'].cumsum()  Output:  city_id week binary consecutive 0 1 1 0 0 1 1 2 1 1 2 1 3 1 2 3 1 4 1 3 4 1 5 0 0 5 1 6 0 0 6 1 7 0 0 7 2 1 0 0 8 2 2 1 1 9 2 3 1 2 10 2 4 0 0 11 2 5 1 1 12 2 6 0 0 13 2 7 1 1 14 3 1 1 1 15 3 2 1 2 16 3 3 1 3 17 3 4 0 0 18 3 5 0 0 19 3 6 0 0 20 3 7 0 0 21 4 1 0 0 22 4 2 0 0 23 4 3 1 1 24 4 4 1 2 25 4 5 1 3 26 4 6 1 4 27 4 7 1 5 28 5 1 1 1 29 5 2 1 2 30 5 3 1 3 31 5 4 0 0 32 5 5 0 0 33 5 6 0 0 34 5 7 0 0 35 6 1 1 1 36 6 2 0 0 37 6 3 1 1 38 6 4 0 0 39 6 5 1 1 40 6 6 0 0 41 6 7 1 1
Question: While trying to reduce the size of a Docker image, I noticed pip install torch adds a few GB. A big chunk of this comes from [...]/site-packages/nvidia. Since I'm not using a GPU, I'd like to not install the nvidia things. Here is a minimal example: FROM python:3.12.5 RUN pip install torch  (Ignoring -slim base images, since this is not the point here.) Resulting size:  FROM python:3.12.5 -> 1.02GB After RUN pip install torch -> 8.98GB With RUN pip install torch && pip freeze | grep nvidia | xargs pip uninstall -y instead -> 6.19GB.  While the last point reduces the final size, all the nvidia stuff is still downloaded and installed, which costs time and bandwidth. So, how can I install torch without nvidia directly? Using --no-deps is not a convenient solution, because of the other transitive dependencies, that I would like to install. Of course, I could explicitly list every single one, but looking at this list of packages installed with torch mpmath typing-extensions sympy nvidia-nvtx-cu12 nvidia-nvjitlink-cu12 nvidia-nccl-cu12 nvidia-curand-cu12 nvidia-cufft-cu12 nvidia-cuda-runtime-cu12 nvidia-cuda-nvrtc-cu12 nvidia-cuda-cupti-cu12 nvidia-cublas-cu12 networkx MarkupSafe fsspec filelock triton nvidia-cusparse-cu12 nvidia-cudnn-cu12 jinja2 nvidia-cusolver-cu12 torch  I'd like to avoid manually maintaining this list since it would change with future versions of torch.

Answer: As (roundaboutly) documented on pytorch.org's getting started page, Torch on PyPI is Nvidia enabled; use the download.pytorch.org index for CPU-only wheels: RUN pip install torch --index-url https://download.pytorch.org/whl/cpu  Also please remember to specify a somewhat locked version of Torch, e.g. RUN pip install torch~=2.4.0 --index-url https://download.pytorch.org/whl/cpu
Question: In the Stone Game problem, Alice and Bob take turns picking a pile of stones from the start or the end. The goal is to maximize Alice's total def play(turn, left, right): if left > right: return 0 end = piles[right] + play(1 - turn, left, right - 1) start = piles[left] + play(1 - turn, left + 1, right) return max(start, end) if turn == 0 else min(start, end) alice = play(0, 0, n - 1)  This follows the classic minimax algorithm. Let's now take a look at Stone Game II. In this problem, Alice and Bob can pick the next 1 <= x <= 2m piles of stones, where m is the maximum x somebody has used. To my surprise, classic minimax would return the same number of stones whether it is Alice or Bob's turn, giving us an incorrect final answer # DOESN'T WORK def play(left, m, turn): if left == n-1: return 0 total = 0 ans = inf if turn else -inf for pos in range(left+1, min(n, left+2*m+1)): total += piles[pos] value = total + play(pos, max(m, pos - left), 1 - turn) if turn == 0: ans = max(ans, value) else: ans = min(ans, value) return ans alice = play(-1, 1, 0)  However, if we only include total in Alice's calculation, it suddenly works: # WORKS def play(left, m, turn): if left == n-1: return 0 total = 0 ans = inf if turn else -inf for pos in range(left+1, min(n, left+2*m+1)): total += piles[pos] value = play(pos, max(m, pos - left), 1 - turn) if turn == 0: ans = max(ans, total + value) else: ans = min(ans, value) return ans alice = play(-1, 1, 0)  Could someone explain why we're not supposed to add the local total for the minimizer in the second example? Here's a discrepancy I noticed that may have something to do with the answer: value is always the same recursive call when we take min/max in the second problem, but in the first problem, end and start are different recursive calls.

Answer: The function we're trying to maximize is f := amount of stones alice gets. That's why we only add stones for Alice; the function we're maximizing doesn't include the amount of stones Bob gets. So then why does the first algorithm work? Turns out it's not generally correct, and only works because this specific problem constrains len(piles) as even, making it so Alice always wins. If we actually look at the values from the first function, it returns the same answer for Bob and Alice too (this was just obfuscated because alice > sum(piles) - alice is always true, so the bug didn't produce wrong answers) An actually correct minimax implementation for game 1 would look like this: def play(turn, left): right_used = turn - left right = n - 1 - right_used if left > right: return 0 end = play(turn + 1, left) start = play(turn + 1, left + 1) return min(start, end) if turn % 2 else max(piles[left] + start, piles[right] + end)
Question: I have a Pandas dataframe that has user information and also has a column for their permissions: UserName Permissions John Doe 02 John Doe 11 Example 09 Example 08 User3 11  I am trying to create a new column called User Class that is based on their Permissions (looking at all of the users permissions). If a user has all permissions <10, they are considered Admin. If a user has all permission >=10, they are considered User. However if they have permissions that are both <10 and >=10, then they will be coded as Admin/User. So my resulting output would be: UserName Permissions User Class John Doe 02 Admin/User John Doe 11 Admin/User Example 09 Admin Example 08 Admin User3 11 User  What would be the best way to do this? My original idea was to do: for UserName, User_df in df.groupby(by='UserName'): LT10 = (User_df['Permissions'] < 10).any() GTE10 = (User_df['Permissions'] >= 10).any() if (LT10 & GTE10): UserClass = 'Admin/User' elif LT10: UserClass = 'Admin' elif GTE10: UserClass = 'User' df.at[User_df.index, 'User Class'] = UserClass  However these seems very inefficient because df has ~800K records

Answer: Another possible solution: df['User Class'] = ( df.groupby('UserName')['Permissions'] .transform(lambda x: 'Admin' if (x < 10).all() else 'User' if (x >= 10).all() else 'Admin/User'))  Output:  UserName Permissions User Class 0 John Doe 2 Admin/User 1 John Doe 11 Admin/User 2 Example 9 Admin 3 Example 8 Admin 4 User3 11 User
Question: I have a custom enum, MyEnum, with some elements that have different names but the same value. from enum import Enum class MyEnum(Enum): A = 1 B = 2 C = 3 D = 1 # Same value as A  Consequently, list(MyEnum) returns only the names of some of the members (the first name for each value): >>>list(MyEnum) [<MyEnum.A: 1>, <MyEnum.B: 2>, <MyEnum.C: 3>]  Apparently, list(MyEnum.__members__) returns all the names: >>>list(MyEnum.__members__) ['A', 'B', 'C', 'D']  However, if I try to override the __iter__() method for my enum, the override seems to fail: class MyEnum(Enum): A = 1 B = 2 C = 3 D = 1 # Same value as A @classmethod # an attempt to override list(MyEnum) that doesn't change anything def __iter__(cls): return iter(list(cls.__members__))  Apparently list(MyEnum) doesn't ever hit the custom __iter__() (as indicated by, say, adding a print() before returning in our custom __iter__()). Why is that? How can I override the default behavior of list(MyEnum) so that I get all the distinct names?

Answer: A class method is not the same as an instance method on a metaclass, which is what __iter__ for Enum is. You need to define a new metaclass, which you can use to define a new subclass of Enum that does what you are looking for. A caveat: I make no claim that replacing the current behavior of EnumType.__iter__ with your suggestion will be compatible with Enum's current semantics, only that this will make your definition available. from enum import EnumType, Enum class MyEnumType(EnumType): def __iter__(self): # Must return an Iterator, something with a __next__ method, # not an Iterable. return iter(list(self.__members__)) class MyEnumBase(Enum, metaclass=MyEnumType): pass class MyEnum(MyEnumBase): A = 1 B = 2 C = 3 D = 1 assert list(MyEnum) == ['A', 'B', 'C', 'D']
Question: I received a sympy equation from a library. It makes extensive use of log2, but the output was converted to log(x)/log(2). This makes reading the results messy. I would like to have sympy simplify this equation again with a focus on using log2 directly where possible. How could this be done? Example: (log(A) / log(2)) * (log(B + C) / log(2)) 

Answer: Basically, you can just reverse the process: replace log(w) with log(2) * log2(w) and let sympy cancel all the factors of log(2) / log(2). The only wrinkle is that you don't want to substitute when you find log(2) itself, but that's easy enough. You just use a wildcard, and check if the matched value is literally 2 before doing the substitution: import sympy from sympy import log from sympy.codegen.cfunctions import log2 A, B, C = sympy.symbols("A, B, C") w = sympy.Wild("w") expr = (log(A) / log(2)) * (log(B + C) / log(2)) expr.replace(log(w), lambda w: log(2) if w==2 else log(2)*log2(w))  The result is log2(A)*log2(B + C)  which is correct. [H/T @ThomasWeller for finding log2 buried in codegen.]
Question: I wish to display a barchart over a time series canvas, where the bars have width that match the duration and where the edges connect the first value with the last value. In other words, how could I have slanted bars at the top to match the data? I know how to make barcharts using either the last value (example 1) or the first value (example 2), but what I'm looking for are polygons that would follow the black line shown. Example 1  Example 2  Code:  import pandas as pd from pandas import Timestamp import datetime import matplotlib.pyplot as plt import numpy as np # np.nan dd = {'Name': {0: 'A', 1: 'B', 2: 'C'}, 'Start': {0: Timestamp('1800-01-01 00:00:00'), 1: Timestamp('1850-01-01 00:00:00'), 2: Timestamp('1950-01-01 00:00:00')}, 'End': {0: Timestamp('1849-12-31 00:00:00'), 1: Timestamp('1949-12-31 00:00:00'), 2: Timestamp('1979-12-31 00:00:00')}, 'Team': {0: 'Red', 1: 'Blue', 2: 'Red'}, 'Duration': {0: 50*365-1, 1: 100*365-1, 2: 30*365-1}, 'First': {0: 5, 1: 10, 2: 8}, 'Last': {0: 10, 1: 8, 2: 12}} d = pd.DataFrame.from_dict(dd) d.dtypes d # set up colors for team colors = {'Red': '#E81B23', 'Blue': '#00AEF3'} # reshape data to get a single Date | is there a better way? def reshape(data): d1 = data[['Start', 'Name', 'Team', 'Duration', 'First']].rename(columns={'Start': 'Date', 'First': 'value'}) d2 = data[['End', 'Name', 'Team', 'Duration', 'Last']].rename(columns={'End': 'Date', 'Last': 'value'}) return pd.concat([d1, d2]).sort_values(by='Date').reset_index(drop=True) df = reshape(d) df.dtypes df plt.plot(df['Date'], df['value'], color='black') plt.bar(d['Start'], height=d['Last'], align='edge', width=list(+d['Duration']), edgecolor='white', linewidth=2, color=[colors[key] for key in d['Team']]) plt.show() plt.plot(df['Date'], df['value'], color='black') plt.bar(d['End'], height=d['First'], align='edge', width=list(-d['Duration']), edgecolor='white', linewidth=2, color=[colors[key] for key in d['Team']]) plt.show() 

Answer: You can use Matplotlibs Axes.fill_between to generate these types of charts. Importantly this will accurately represent the gap between your rows where they exist, whereas the approach with the bars will make that gap appear to be wider than they truly are unless you set the edgewidth of the bars to 0. Additionally, for your data transformation this is a pandas.lreshape which is similar to performing multiple melts operations at the same time. import pandas as pd from pandas import Timestamp import matplotlib.pyplot as plt dd = pd.DataFrame({ 'Name': ['A', 'B', 'C'], 'Start': pd.to_datetime(['1800-01-01', '1850-01-01', '1950-01-01']), 'End': pd.to_datetime(['1849-12-31', '1949-12-31', '1979-12-31']), 'Team': ['Red', 'Blue', 'Red'], 'Duration': [50*365-1, 100*365-1, 30*365-1], 'First': [5, 10, 8], 'Last': [10, 8, 12] }) df = ( pd.lreshape(dd, groups={'Date': ['Start', 'End'], 'Value': ['First', 'Last']}) .sort_values('Date') ) colors = {'Red': '#E81B23', 'Blue': '#00AEF3'} fig, ax = plt.subplots() for team in df['Team'].unique(): ax.fill_between( df['Date'], df['Value'], where=(df['Team'] == team), color=colors[team], linewidth=0, ) ax.set_ylim(bottom=0) plt.show()
Question: The input image is here : the input image I try to use cv2.HoughCircles in opencv-python to find the expected circle, but the result is noise as in this picture : result in param2=0.2 the code is: import cv2 import numpy as np img = cv2.imread('image.png') # apply GaussianBlur kernel_size = (15, 15) sigma = 0 blurred_image = cv2.GaussianBlur(img, kernel_size, sigma) gray = cv2.cvtColor(blurred_image, cv2.COLOR_BGR2GRAY) circles = cv2.HoughCircles(gray, cv2.HOUGH_GRADIENT_ALT, dp=1.5, minDist=20, param1=50, param2=0.2, minRadius=100, maxRadius=400) # draw the result if circles is not None: circles = np.uint16(np.around(circles)) for i in circles[0, :]: cv2.circle(img, (i[0], i[1]), i[2], (0, 255, 0), 2) cv2.circle(img, (i[0], i[1]), 1, (0, 0, 255), 3) # show the result cv2.imshow('used image', gray) cv2.imshow('detected circles', img) cv2.waitKey(0) cv2.destroyAllWindows()  If i set param2=0.8, then no any circle can be found! What should I do to get a better result that fit the further perfect circle in the example picture above? The "perfect circle" can be defined as follows: Given an angular resolution of 0.1 degrees, the area will be divided into 360/0.1 = 3600 segments. A perfect circle includes as many sectors as possible, where these sectors contain pixels with grayscale values below a certain threshold (e.g., 15). In other words, the goal is to have the proportion of sectors that meet the criteria as close to 1 as possible. If there are multiple resulted circles with same proportion, any one of them can be considered a perfect circle. An example "perfect circle" : perfect circle

Answer: This is sort of a static solution for now, but if you always have such nice contrast and you can accuretly get the coordinates, then fitting a circle with least square might not be such a bad idea: def fit_circle(x, y): A = np.c_[x, y, np.ones(len(x))] # design matrix A with columns for x, y, and a constant term f = x**2 + y**2 # get the function as x²+y² C, _, _, _ = np.linalg.lstsq(A, f, rcond=None) # optimize cx = C[0] / 2 # get the centre x coordinate cy = C[1] / 2 # get the centre y coordinate radius = np.sqrt(C[2] + cx**2 + cy**2) # calculate the radius return round(cx), round(cy), round(radius) # return everythin as int  To use this function, I did the following: im = cv2.imread("circle.png") # read as BGR imGray = cv2.imread("circle.png", 0) # read as gray y, x = np.where(imGray==0) # get x,y coords cx, cy, r = fit_circle(x,y) # get the circle properties, center and radius im = cv2.circle(im, (cx,cy), r, (255, 0, 0), 5) im = cv2.line(im, (cx-r,cy), (cx+r,cy), (0, 255,0), 2) im = cv2.line(im, (cx,cy-r), (cx,cy+r), (0, 255,0), 2) im = cv2.circle(im, (cx,cy), 10, (0, 0, 255), -1) cv2.imwrite("WithCircle.png", im) # save im for stack  The result:  As I said, very important that you can get those nice pixels that define your circle, this method can work with some added noise as well I presume, but definitely not good if you have any one-sided deviations.
Question: I have a signal like the orange one in the following plot that can only have integer values:  As you can see, the orange signal in a bit noisy and "waffles" between levels sometimes when its about to change to a new steady state. I'd like to "smooth" this effect and achieve the blue signal. The blue signal is the orange one filtered such that the transitions don't occur until 3 samples in a row have made the jump to the next step. This is pretty easy if I loop through each sample manually and use a couple state variables to track how many times in a row I've jumped to a new step, but its also slow. I'd like to find a way to vectorize this in numpy. Any ideas? Here's an example of the non-vectorized way that seems to do what I want: up_count = 0 dn_count = 0 out = x.copy() for i in range(len(out)-1): if out[i+1] > out[i]: up_count += 1 dn_count = 0 if up_count == 3: up_count = 0 out[i+1] = out[i+1] else: out[i+1] = out[i] elif out[i+1] < out[i]: up_count = 0 dn_count += 1 if dn_count == 3: dn_count = 0 out[i+1] = out[i+1] else: out[i+1] = out[i] else: dn_count = 0 up_count = 0  EDIT: Thanks to @Bogdan Shevchenko for this solution. I already have numpy and scipy available, so rather than get pandas involved here's my numpy/scipy version of his answer: def ffill(arr, mask): idx = np.where(~mask, np.arange(mask.shape[0])[:, None], 0) np.maximum.accumulate(idx, axis=0, out=idx) return arr[idx, np.arange(idx.shape[1])] x_max = scipy.ndimage.maximum_filter1d(x, 3, axis=0, origin=1, mode="nearest") x_min = scipy.ndimage.minimum_filter1d(x, 3, axis=0, origin=1, mode="nearest") x_smooth = ffill(x, x_max!=x_min) 

Answer: There could be a lot of different strategies, based on DataFrame.rolling processing. In example: t = pd.Series([ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 1, 2, 1, 2, 1, 1, 2, 2, 3, 1, 2, 2, 1, 1, 2, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2]) min_at_row = 3 t_smoothed = t.copy() t_smoothed[ t.rolling(min_at_row, min_periods=1).max() != t.rolling(min_at_row, min_periods=1).min() ] = None t_smoothed = t_smoothed.ffill()  Result looks like what you want to obtain:  If you want to stay with numpy only, without pandas, use np.lib.stride_tricks.sliding_window_view(t, min_at_row).max(axis=1) instead of t.rolling(min_at_row).max() (and analogously with min), but be aware that it will return array without first (min_at_row - 1) values You can also use t_smoothed.shift(-min_at_row + 1) to remove delay of smoothened signal.
Question: I encountered an issue while trying out a simple example of creating and detecting aruco-images. In the following code-snippet, I generate aruco images, save them to a file and then load one of these files for detection: import cv2 aruco_dict= cv2.aruco.getPredefinedDictionary(cv2.aruco.DICT_ARUCO_ORIGINAL) params = cv2.aruco.DetectorParameters() ##create aruco marker_size = 500 #mm for i in range(6): marker_image = cv2.aruco.generateImageMarker(aruco_dict, i, marker_size) cv2.imwrite(f"marker_{i}.png", marker_image) ##load aruco image img = cv2.imread("marker_5.png") #detect markers aruco_detector = cv2.aruco.ArucoDetector(aruco_dict,params) corners, ids, _ = aruco_detector.detectMarkers(img) print(corners) print(ids)  This code results in an empty list of corners (e.g. the detector wasn't able to find the aruco). I assumed it should be able to detect it easily if I simply re-use the image created by OpenCV. Does someone have an idea what I did wrong or where the issue lies? Any tips are welcome. Kind regards!

Answer: These markers need a "quiet zone" around them so the edge of the marker is detectable. That quiet zone should be a white border. You may have heard of this in relation to QR codes. generateImageMarker() does not produce a quiet zone around the marker. This is as it should be. The image contains just the imagery that goes inside the marker's bounds. Just the marker:  Use copyMakeBorder() to add a quiet zone. The recommended width is one "module". Half a module may work marginally but I would not recommend operating near that limit. The width of the quiet zone also matters when you print and cut out markers. That quiet zone must be visible on the paper. You can't just cut the marker right up to its edge. Maintain one module of quiet zone. Your marker is sized as 7x7 modules (5x5 of data, and 1 of border all around), resolved at 500 pixels width. One module would figure to be 500/7 ~= 71 pixels wide. The size of a marker (counted in modules) depends on the dictionary you chose (which was "original"). 4x4 and 6x6 flavors are popular. This specifies the data area, so those would be sized 6x6 and 8x8 respectively. border_width = int(500 / 7) marker_image = cv.copyMakeBorder( marker_image, border_width, border_width, border_width, border_width, borderType=cv.BORDER_CONSTANT, value=255)  The marker, padded to about one module of quiet zone, will now be detectable: