eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: All, I use ESRI World Countries Generalized shape file, that is available here using GeoPandas shp_file =gpd.read_file('World_Countries/World_Countries_Generalized.shp') print(shp_file.crs)  The CRS I got is EPSG:3857, yet once I add the CRS to the gpd.read_file as the following shp_file1 =gpd.read_file('../../Downloads/World_Countries/World_Countries_Generalized.shp',crs='EPSG:3857')  I got the following error /opt/anaconda3/envs/geo_env/lib/python3.12/site-packages/pyogrio/raw.py:198: RuntimeWarning: driver ESRI Shapefile does not support open option CRS return ogr_read( Do you know why I get this error, and does it mean the file is not read correctly? Thanks

Answer: Since geopandas 1.0, another, faster, underlying library is used by default to read files in geopandas: pyogrio. Some more info can be found here: fiona vs pyogrio. When this new library is used, the crs parameter is not supported. The easiest solution for this specific case is to just remove the crs='EPSG:3857' parameter as is is useless anyway as the crs is already read correctly? shp_file1 = gpd.read_file('../../Downloads/World_Countries/World_Countries_Generalized.shp')  If you want to read a shapefile that doesn't have a .prj file, or a wrong one, you can e.g. use the set_crs function of the GeoDataFrame to set or overrule the crs after reading the file: shp_file2 = gpd.read_file('wrong_crs.shp').set_crs(crs='EPSG:3857', allow_override=True)
Question: I use OpenCV and Python and I want to remove the "+" sign that is repeated on my image. The following is an example of an image in question.  The goal is to produce the same image, but with the "+" signs removed. How can I achieve this? I've tried using the below code to achieve this. img = cv2.imread(image_path) gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY) img_inverted = cv2.bitwise_not(gray) thresh = cv2.adaptiveThreshold(img_inverted, 255, cv2.ADAPTIVE_THRESH_MEAN_C, cv2.THRESH_BINARY, 15, -2) cross_structure = cv2.getStructuringElement(cv2.MORPH_CROSS, (3, 3)) detected_cross = cv2.morphologyEx(thresh, cv2.MORPH_OPEN, cross_structure) mask = detected_cross kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (3, 3)) dilated_mask = cv2.dilate(mask, kernel, iterations=1) output_telea = cv2.inpaint(img, dilated_mask, 3, cv2.INPAINT_TELEA)  But it seems like the detected_cross isn't really detecting the expected objects, and as a result, it returns the same threshold image. This is the result I end up with. 

Answer: This is basically implementing the comment of fmw42; that is, create the mask by thresholding for white regions (also I increased your dilation to 2 iterations for a visually slightly better result): import cv2 import numpy as np image_path = ... # TODO: adjust as necessary img = cv2.imread(image_path) mask = (cv2.cvtColor(img, cv2.COLOR_BGR2GRAY) > 250).astype(np.uint8) kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (3, 3)) dilated_mask = cv2.dilate(mask, kernel, iterations=2) output_telea = cv2.inpaint(img, dilated_mask, 3, cv2.INPAINT_TELEA) cv2.imshow("result", output_telea) cv2.waitKey(0) cv2.destroyAllWindows()  Result:  In general, I agree with Cris Luengo's comments though:  If you have multiple such images where the crosses are always in the same place, create a mask once (possibly manually) and reuse it. Even better, try to remove the crosses at image acquisition time (i.e. don't acquire them in the first place) rather than as a post-processing step. That is to say, if you have control over the acquisition process, of course.
Question: When invoking a Snowpark-registered SPROC, I get the following error: SnowparkSQLException: (1304): <uuid>: 100357 (P0000): <uuid>: Python Interpreter Error: snowflake.connector.errors.ProgrammingError: 251005: User is empty in function MY_FUNCTION with handler compute  for the following python code and invocation: def my_function(session: Session, input_table: str, limit: int) -> None: # Even doing nothing doesn't work! return sproc_my_function = my_session.sproc.register(func=my_function, name='my_function', is_permanent=True, replace=True, stage_location='@STAGE_LOC', execute_as="owner", input_table = 'x.y.MY_INPUT_TABLE' sproc_process_row(my_session, input_table, 100, )  I can't find a reference to this exception and "User is empty in function" anywhere on the internet - which makes me wonder if its a drop-through of some sort. I also can't find a way to pass a user to the register method (this is already done successfully when my_session is set up). Please help!

Answer: Using Snowflake Notebooks: from snowflake.snowpark.session import Session my_session = snowflake.snowpark.context.get_active_session() def my_function(session: Session, input_table: str, limit: int) -> None: return None sproc_my_function = my_session.sproc.register(func=my_function, name='my_function', is_permanent=True, replace=True, stage_location='STAGE_LOC', execute_as="owner", packages=["snowflake-snowpark-python"])  Output:
Question: I have a python script that receives incoming messages from web and process the data: async with asyncio.TaskGroup() as task_group: processor_task = task_group.create_task(processor.start(message), name=f"process_message_{message.sender_id}_task"  I added naming for tasks with f-string interpolation so in the debugging i know which task is doing what and for who. Each iteration of message processing may take up to an hour. I want to only process one message from a sender at a time, however, senders sometimes send multiple messages. How can i drop further messages from the same sender_id until there is no task running for them? How can i check all running tasks for a same name before creating a task in TaskGroup? I Tried reading the documentation of TaskGroup but i didn't found anything related.

Answer: you can keep track of the working tasks like this background_tasks = set() async with asyncio.TaskGroup() as task_group: if sender_id not in background_tasks: # check if the sender has a task running processor_task = task_group.create_task(processor.start(message), name=f"process_message_{message.sender_id}_task") background_tasks.add(sender_id) # add to background tasks processor_task.add_done_callback(lambda x: background_tasks.discard(sender_id)) # remove from background tasks after the task is finished else: print(f'user {sender_id} already has a task running')
Question: I am trying to use the GridSearchCV from sklearn.model_selection. My data is a set of classification that is indexed by time. As a result, when doing cross validation, I want the training set to be exclusively the data with time all before the data in the test set. So my training set X_train, y_train looks like Time feature_1 feature_2 result 2020-01-30 3 6 1 2020-02-01 4 2 0 2021-03-02 7 1 0  and the test set X_test, y_test looks like Time feature_1 feature_2 result 2023-01-30 3 6 1 2023-02-01 4 2 0 2024-03-02 7 1 0  Suppose I am using a model such as xgboost, then to optimise the hyperparameters, I used GridSearchCV and the code looks like param_grid = { 'max_depth': [1,2,3,4,5], 'min_child_weight': [0,1,2,3,4,5], 'gamma': [0.5, 1, 1.5, 2, 5], 'colsample_bytree': [0.6, 0.8, 1.0], } clf = XGBClassifier(learning_rate=0.02, n_estimators=600, objective='binary:logistic', silent=True, nthread=1) grid_search = GridSearchCV( estimator=clf, param_grid=param_grid, scoring='accuracy', n_jobs= -1) grid_search.fit(X_train, y_train)  However, how should i set the cv in grid_search? Thank you so much in advance. Edit: So I tried to set cv=0 since I want my training data to be strictly "earlier" then test data and I got the following errors: InvalidParameterError: The 'cv' parameter of GridSearchCV must be an int in the range [2, inf), an object implementing 'split' and 'get_n_splits', an iterable or None. Got 0 instead.

Answer: the default cross-validation in GridSearchCV does not consider temporal dependency when splitting. You can use TimeSeriesSplit instead of the default CV from model selection. TimeSeriesSplit is built for this exact use case of yours.
Question: I'm using spacy for some downstream tasks, mainly noun phrase extraction. My texts contain a lot of parentheses, and while applying the lemma, I noticed all the punctuation that doesn't end sentences becomes --: import spacy nlp = spacy.load("de_core_news_sm") doc = nlp("(Das ist ein Test!)") for token in doc: print(f"Text: '{token.text}', Lemma: '{token.lemma_}'")  Output: Text: '(', Lemma: '--' Text: 'Das', Lemma: 'der' Text: 'ist', Lemma: 'sein' Text: 'ein', Lemma: 'ein' Text: 'Test', Lemma: 'Test' Text: '!', Lemma: '--' Text: ')', Lemma: '--'  Is that normal, and if yes, why, and what can I do to keep the parentheses? I'm on 3.7.4 with Python 3.11

Answer: I can confirm the issue with German, but when I try the equivalent sentence in Dutch the ( and ) are kept as lemma instead of --. So this is something particular in the German model. You can override the default lemmata if you want: import spacy nlp = spacy.load("de_core_news_sm") nlp.get_pipe("attribute_ruler").add([[{"TEXT": "("}]], {"LEMMA": "("}) nlp.get_pipe("attribute_ruler").add([[{"TEXT": ")"}]], {"LEMMA": ")"}) doc = nlp("(Das ist ein Test!)") print(doc.text) for token in doc: print(token.text, token.lemma_, token.pos_, token.dep_)  Result: (Das ist ein Test!) ( ( PUNCT punct Das der PRON sb ist sein AUX ROOT ein ein DET nk Test Test NOUN pd ! -- PUNCT punct ) ) PUNCT punct
Question: How can I silence the following warning while running GridSearchCV(model, params, cv=10, scoring='precision', verbose=1, n_jobs=20, refit=True)? /opt/dev/myenv/lib/python3.9/site-packages/sklearn/metrics/_classification.py:1531: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.  I have tried without success: import os, warnings warnings.simplefilter("ignore") warnings.filterwarnings("ignore") with warnings.catch_warnings(): warnings.simplefilter("ignore") os.environ["PYTHONWARNINGS"] = "ignore" 

Answer: Try this code import warnings from sklearn.exceptions import UndefinedMetricWarning with warnings.catch_warnings(): # Ignore only UndefinedMetricWarning warnings.filterwarnings("ignore", category=UndefinedMetricWarning)
Question: I wanted to delete a user. After a bit of struggling I ended up with: views.py the_user = get_user_model() @login_required def del_user(request): email = request.user.email the_user.objects.filter(email=email).delete() messages.warning(request, "bruker slettet.") return redirect("index")  But I really do not understand the following line: email = request.user.email.  And why not? email = request.the_user.email  Is this because the user is referring to the AbstractBaseUser?

Answer: request is an instance of an HttpRequest object. It represents the current received request. The AuthenticationMiddleware authenticates the user that made the request (based on cookies or however you have configured it) and adds the request.user attribute to it, so your code can get who the current user is that made the request. What you call the_user is the User model class. That's an arbitrary name you gave to a variable in your code. FYI, the request.user object is already a full fledged User instance, which already has a delete method. You don't need to find the same user again via the the_user model class, you can just do: @login_required def del_user(request): request.user.delete() messages.warning(request, "bruker slettet.") return redirect("index")
Question: I have been working this leet code questions https://leetcode.com/problems/maximum-average-subarray-i/description/ I have been able to create a solution after understanding the sliding window algorithm. I was wondering with my code where my logic is going wrong, I do think my my issue seems to be in this section section of the code, but I am unable to pinpoint why.  while temp > k: temp -= nums[left] left += 1 ans = temp / (curr - left + 1)  While I do appreciate other solutions and ways solving this problem, I want to understand and get solution working first before I start looking at different ways of doing the problem, this way i get a better understanding of the algorithm. Full code reference def findMaxAverage(self, nums, k): """ :type nums: List[int] :type k: int :rtype: float """ left = 0 ans = 0 temp = 0 for curr in range(len(nums)): temp += nums[curr] curr += 1 while temp > k: temp -= nums[left] left += 1 ans = temp / (curr - left + 1) return ans 

Answer: temp is the sum of the subarray, while k is the number of elements in it - you shouldn't be comparing the two, they're two totally different things. To implement a sliding window, I'd sum the first k elements of nums, and then run over it where in each iteration I drop the first element and add the last and then check if the sum has increased or not. Once you find the maximum sum, you can find the average by dividing it by k. class Solution: def findMaxAverage(self, nums: List[int], k: int) -> float: currSum = sum(nums[:k]) maxSum = currSum for i in range(k, len(nums)): currSum = currSum - nums[i - k] + nums[i] maxSum = max(maxSum, currSum) return maxSum / k
Question: A LeetCode problem which is about linked lists that goes with this structure: # Definition for singly-linked list. # class ListNode: # def __init__(self, val=0, next=None): # self.val = val # self.next = next  Gave an error while attempting to print the val of the next node, but still worked when given a nullity check (it never even went to the else statement). Assuming l1 is an instance of the class ListNode with print(l1.nextNode) gives: ListNode{val: 4, next: ListNode{val: 3, next: None}} And: nextNode = l1.next Why does this Fail: print(nextNode.val)  AttributeError: 'NoneType' object has no attribute 'val'  While this Works: if nextNode is not None: print(nextNode.val) else: print("Node is None")  Extra: I wonder if the answer to the above is related to why this also Fails with try/catch: try: print("try block executed") print(nextNode.val) except: **print("except block executed1") print(nextNode.val)** if nextNode is not None: print("except block executed"2) print(nextNode.val) else: print("Node is None")  While this Works and prints try block executed: try: print("try block executed") print(nextNode.val) except: if nextNode is not None: print("except block executed") print(nextNode.val) else: print("Node is None")  EDIT: Found the cause, turned out that the code fails for certain test case where it has only 1 node, but when it succeeds, it shows another test case result Found this out while trying to create a copy-able code, rookie mistake... For more details check the LeetCode problem in the link provided at the start.

Answer: When you reach the end of the list, nextNode will be None. None doesn't have a val attribute, so nextNode.val raises an exception. If you check whether it's None first, you don't execute that erroneous expression, so there's no error. When you use try/except, it catches the exception. But then in the except: block you try to print the same thing. It raises the exception again, and this time there's no try/except to catch it, so the program stops with an error.
Question: models.py ''' class Person(models.Model): firstname = models.CharField(max_length=30) lastname = models.CharField(max_length=30) othernames = models.CharField(max_length=40) dateOfBirth = models.DateField() gender = models.CharField(max_length=20) birthGender = models.CharField(max_length=20) email = models.EmailField(max_length=100) class Student(Person): studentId = models.IntegerField() admissionDate = models.DateField() enrolmentStatus = models.BooleanField() studentExamCode = models.IntegerField() schoolYear = models.IntegerField()  ''' forms.py ''' class studentF(ModelForm): class Meta: model = Student fields = ['firstname', 'lastname', 'othernames', 'dateOfBirth', 'gender', 'birthGender', 'email', 'studentId', 'admissionDate', 'enrolmentStatus', 'studentExamCode', 'schoolYear']  ''' I am getting the error in the title, how do I fix it?

Answer: make sure you have done what @raphael said you should do. if issue persist, python manage.py makemigrations --merge python manage.py migrate --fake  reset db python manage.py migrate *yourapp* zero python manage.py migrate
Question: I have a scenario where a class contains an inner class, and I want to reference that inner class (or its attributes) within the outer class. Here’s a concrete example using Django: from django.db import models from django.utils.translation import gettext_lazy as _ class DummyModel(models.Model): class StatusChoices(models.TextChoices): ACTIVE = "active", _("Active") INACTIVE = "inactive", _("Inactive") status = models.CharField( max_length=15, choices=StatusChoices.choices, verbose_name=_("Status"), help_text=_("Current status of the model."), default=StatusChoices.ACTIVE, null=False, blank=False, ) class Meta: verbose_name = _("Dummy Model") verbose_name_plural = _("Dummy Models") constraints = [ models.CheckConstraint( name="%(app_label)s_%(class)s_status_valid", check=models.Q(status__in=[choice.value for choice in DummyModel.StatusChoices]), ) ]  In this case, the constraints list in the Meta class tries to reference DummyModel.StatusChoices. However, at the time this reference is evaluated, DummyModel is not fully defined, leading to an error (neither StatusChoices is accessible in that line). I would like to solve this without significantly altering the structure of the code—StatusChoices must remain defined inside DummyModel. How can I resolve this issue while keeping the inner class and its attributes accessible as intended?

Answer: You can probably do this by defining the choices outside the class first, because the Meta class is actually constructed even before the status is accessible: # 🖟 outside DummyModel class StatusChoices(models.TextChoices): ACTIVE = 'active', _('Active') INACTIVE = 'inactive', _('Inactive') class DummyModel(models.Model): status = models.CharField( max_length=15, choices=StatusChoices.choices, verbose_name=_('Status'), help_text=_('Current status of the model.'), default=StatusChoices.ACTIVE, null=False, blank=False, ) class Meta: verbose_name = _('Dummy Model') verbose_name_plural = _('Dummy Models') constraints = [ models.CheckConstraint( name='%(app_label)s_%(class)s_status_valid', check=models.Q( status__in=[choice.value for choice in StatusChoices] ), ) ] DummyModel.StatusChoices = StatusChoices For what it is worth, I made a small Django package named django-enforced-choices [GitHub] that can enforce choices at the database by just looking at the field with choices.
Question: I am having an issue obtaining a linestring from an intersection with a polygon using GeoPandas. The linestring is self-intersecting, which is what is causing my issues. A line intersecting a polygon:  Given the following code: import geopandas as gp from shapely.geometry import LineString, Polygon # Draw a polygon that is 100 x 100 units, starting at coordinates 0, 0 polygon = Polygon([(50, 0), (50, 100), (100, 100), (100, 0)]) # Convert the polygon to a geodataframe polygon = gp.GeoDataFrame(index=[0], crs='EPSG:4326', geometry=[polygon]) # Draw a horizontal line that starts at coordinates 50, 0 and is 200 units long line = LineString([(0, 50), (75, 50), (70, 35), (55, 40), (250, 50)]) # Convert the line to a geodataframe line = gp.GeoDataFrame(index=[0], crs='EPSG:4326', geometry=[line]) print(line) # Intersect the line with the polygon intersection = line.intersection(polygon) print(intersection)  I have the following results:  0 LINESTRING (0.00000 50.00000, 75.00000 50.0000... 0 MULTILINESTRING ((50.00000 50.00000, 75.00000 ...  After intersect, I am returned a multilinestring instead of a linestring. The line is being split by the polygon (desired) but is also being split into multiple lines where it self-intersects (not desired). I have tried to re-join the multiline with unary-union without success. The output remains a multilinestring. I'm unsure what else I can do to keep only the portion of the line contained within the polygon, as a single line. Any thoughts on how I might be able to accomplish this?

Answer: Because line_merge apparently doesn't reconstruct the single linestring when it is self-intersecting, the only option I see is to extract the coordinates and create a new linestring, which results in a single linestring. In the code sample I use some functions that are only available in shapely 2, so it needs a relatively up-to-date version of shapely. Code sample: import geopandas as gp import shapely from shapely.geometry import LineString, Polygon # Draw a polygon that is 100 x 100 units, starting at coordinates 0, 0 polygon = Polygon([(50, 0), (50, 100), (100, 100), (100, 0)]) # Convert the polygon to a geodataframe polygon = gp.GeoDataFrame(index=[0], crs='EPSG:4326', geometry=[polygon]) # Draw a horizontal line that starts at coordinates 50, 0 and is 200 units long line = LineString([(0, 50), (75, 50), (70, 35), (55, 40), (250, 50)]) # Convert the line to a geodataframe line = gp.GeoDataFrame(index=[0], crs='EPSG:4326', geometry=[line]) # print(line) # Intersect the line with the polygon intersection = line.intersection(polygon) print(f"{intersection=}") # Recreate the intersection line from its coordinates intersection_line = shapely.linestrings(shapely.get_coordinates(intersection)) print(f"{intersection_line=}")
Question: I have a pandas Series containing strictly non-negative integers like so: 1 2 3 4 5  I want to convert them into n-bits binary representation based on the largest value. For example, the largest value here is 5, so we would have 3 bits/3 columns, and the resulting series would be something like this 0 0 1 0 1 0 0 1 1 1 0 0 1 0 1  Thanks a lot in advance!

Answer: If your values are less than 255, you could unpackbits: s = pd.Series([1, 2, 3, 4, 5]) N = int(np.log2(s.max())) powers = 2**np.arange(N, -1, -1) out = pd.DataFrame(np.unpackbits(s.to_numpy(np.uint8)[:, None], axis=1)[:, -N-1:], index=s.index, columns=powers)  If your have larger numbers, compute a mask with & and an array of powers of 2: s = pd.Series([1, 2, 3, 4, 5]) powers = 2**np.arange(int(np.log2(s.max())), -1, -1) out = pd.DataFrame((s.to_numpy()[:, None] & powers).astype(bool).astype(int), index=s.index, columns=powers)  Output:  4 2 1 0 0 0 1 1 0 1 0 2 0 1 1 3 1 0 0 4 1 0 1
Question: I have an issue where blackboxprotobuf takes response of protobuf & returning the dictionary where i see few values where suppose to be negative instead coming as positive value. Calling an APi with lat (40.741895) & long(-73.989308). Using these lat & long, a key is genereated '81859706' that be used in the api. For Key Generation we are using paid framework. url = "https://gspe85-ssl.ls.apple.com/wifi_request_tile" response =requests.get(url, headers={ 'Accept': '*/*', 'Connection': 'keep-alive', 'X-tilekey': "81859706", 'User-Agent': 'geod/1 CFNetwork/1496.0.7 Darwin/23.5.0', 'Accept-Language': 'en-US,en-GB;q=0.9,en;q=0.8', 'X-os-version': '17.5.21F79' })  Which returns protobuf as response. For the same using blackboxprotobuf to convert protobuf_to_json snippet message, typedef = blackboxprotobuf.protobuf_to_json(response.content) json1_data = json.loads(message)  Response:  "2": [ { "4": { "2": { "1": 1, "2": 1 } }, "5": 124103876854927, "6": { "1": 407295068, "2": 3555038608 //This values should be negative } },  Any help how to debug this response & fix this issue. Thank you

Answer: The bit value that comes for the number is >>> bin(3555038608) '0b11010011111001011001010110010000'  If you take 2s complement of that number, you will get the negative that you want. >>> def twos_comp(val, bits): ... """compute the 2's complement of int value val""" ... if (val & (1 << (bits - 1))) != 0: # if sign bit is set e.g., 8bit: 128-255 ... val = val - (1 << bits) # compute negative value ... return val # return positive value as is ... >>> twos_comp(3555038608, 32) -739928688  If you know the bit length of coordinates, in this case 32. Since its an int. You can convert it back, using this.