eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Is there a way to safely type-check a python class which subclasses a protocol? If I define a protocol with a certain method function signature, then implicit subclasses must define a method with a compatible signature: # protocols.py from abc import abstractmethod from dataclasses import dataclass from typing import Protocol class SupportsPublish(Protocol): @abstractmethod def publish(self, topic: str): ... def publish(m: SupportsPublish): m.publish("topic") @dataclass class Publishable: foo: str = "bar" def publish(self): print(self) publish(Publishable()) # ✗ mypy protocols.py # protocols.py:24: error: Argument 1 to "publish" has incompatible type "Publishable"; expected "SupportsPublish" [arg-type] # protocols.py:24: note: Following member(s) of "Publishable" have conflicts: # protocols.py:24: note: Expected: # protocols.py:24: note: def publish(self, topic: str) -> Any # protocols.py:24: note: Got: # protocols.py:24: note: def publish(self) -> Any # Found 1 error in 1 file (checked 1 source file)  But, if I explicitly subtype SupportsPublish, mypy does not report a type error: ... @dataclass class Publishable(SupportsPublish): ... # ✗ mypy protocols.py # Success: no issues found in 1 source file  Based on this blurb from the PEP, I expected the type checker to find the function signature mismatch:  Note that there is little difference between explicit and implicit subtypes, the main benefit of explicit subclassing is to get some protocol methods “for free”. In addition, type checkers can statically verify that the class actually implements the protocol correctly:  This is my environment:  > mypy --version mypy 1.3.0 (compiled: yes) > python --version Python 3.9.17  I expected mypy to point out the function signature mismatch.

Answer: I just want to point out, as was established in the comments, it is not in fact true that if you explicitly subtype SupportsPublish, mypy does not report a type error. The problem is that you weren't type annotating your method, which essentially tells mypy "don't check this". If you do, for example: from dataclasses import dataclass from typing import Protocol class SupportsPublish(Protocol): def publish(self, topic: str) -> None: ... def publish(m: SupportsPublish): m.publish("topic") @dataclass class Publishable: foo: str = "bar" def publish(self) -> None: print(self)  Then mypy will complain: (py311) Juans-MBP:~ juan$ mypy foo.py foo.py:18: error: Argument 1 to "publish" has incompatible type "Publishable"; expected "SupportsPublish" [arg-type] foo.py:18: note: Following member(s) of "Publishable" have conflicts: foo.py:18: note: Expected: foo.py:18: note: def publish(self, topic: str) -> None foo.py:18: note: Got: foo.py:18: note: def publish(self) -> None Found 1 error in 1 file (checked 1 source file)  Because this is a requirement of just regular subclassing with method overriding. If you aren't going to run mypy with full --strict mode, at least somehow (through how it is invoked or by using a mypy.ini) make sure you have --disallow-untyped-defs or --disallow-untyped-calls
Question: NumPy provides the np.polynomial.legendre.leggauss() function to compute the sample points and weights for Gauss-Legendre quadrature. I'm trying to use this function to get the 2D points and weights for a quadrilateral. I am able to use the function as shown here: def gauss2D(n): x, w = np.polynomial.legendre.leggauss(n) weights = [] gauss_pts = [] for i in range(n): for j in range(n): wts = w[i] * w[j] weights.append(wts) g = [x[i], x[j]] gauss_pts.append(g) return np.array(weights), np.array(gauss_pts)  Which outputs the following for n=2 integration points: weights [1. 1. 1. 1.] points [[-0.57735027 -0.57735027] [-0.57735027 0.57735027] [ 0.57735027 -0.57735027] [ 0.57735027 0.57735027]]  If it's possible, I would like to get rid of the for-loops by using NumPy arrays but my attempt (see function below) does not generate the correct results. def gauss2Dnumpy(n): x, w = np.polynomial.legendre.leggauss(n) weights = np.concatenate((w, w)) gauss_pts = np.zeros((n*2, n)) for i in range(n*2): for j in range(n): gauss_pts[i] = x[j], x[j] return weights, gauss_pts  Is there a way to accomplish this without the for-loops?

Answer: This would be: x, w = np.polynomial.legendre.leggauss(n) gauss_pts = np.array(np.meshgrid(x,x,indexing='ij')).reshape(2,-1).T weights = (w*w[:,None]).ravel()
Question: Is it possible to create a construction like this, where a method foo of a class A is used to set the value of an attribute of another class B. Then this method should be called on an instance of A without calling A.foo() directly. class A: def foo(self): print("Hello") class B: def __init__(self): self.func = A.foo def bar(self): obj = A() obj.self.func()  My use case is that in the __init__() method of B different functions of A can be chosen for set the value of the attribute self.func by an if-else statement; this because calling the method of class A directly it is not possible, since it is not clear which one will be chosen.

Answer: In this case, you could just pass the instance explicitly: class A: def foo(self): print("Hallo") class B: def __init__(self): self.func = A.foo def bar(self): obj = A() self.func(obj)
Question: I have 2 numpy arrays of different sizes. One of them 'a' contains int values, while the other (larger) np array 'b' contains float values with (say) 3-4 values per element/value in 'a'. a = np.random.randint(low = 1, high = 100, size = (7)) a # array([35, 11, 48, 20, 13, 31, 49]) b = np.array([34.78, 34.8, 35.1, 34.99, 11.3, 10.7, 11.289, 18.78, 19.1, 20.05, 12.32, 12.87, 13.5, 31.03, 31.15, 29.87, 48.1, 48.5, 49.2]) a.shape, b.shape # ((7,), (19,))  The idea is to find the value in 'b' matching each unique value in 'a' in terms of closest distance, which I am computing using abs values. To do it using a single element of 'a', or using the first element of 'a': # Compare first element of a with all elements of b- np.abs(a[0] - b).argsort() ''' array([ 3, 2, 1, 0, 14, 13, 15, 16, 17, 18, 9, 8, 7, 12, 11, 10, 4, 6, 5]) ''' # b[3] # 34.99 # np.abs(a[0] - b).argsort()[0] # 3 b[np.abs(a[0] - b).argsort()[0]] # 34.99  So, the 4th element in 'b' (b[3]) is the closest match to a[0]. To compute this for all values in 'a', I use a loop as: for e in a: idx = np.abs(e - b).argsort() print(f"{e} has nearest match = {b[idx[0]]:.4f}") ''' 35 has nearest match = 34.9900 11 has nearest match = 11.2890 48 has nearest match = 48.1000 20 has nearest match = 20.0500 13 has nearest match = 12.8700 31 has nearest match = 31.0300 49 has nearest match = 49.2000 '''  How can I achieve this without the slow for loop? Note: a.shape = 1400 and b.shape = 1.5 million (approxmately)

Answer: If you have a lot of values to check you could also try using a kdTree. For few values the overhead of building the tree won't make this worth it, but for large n and especially for searching for nearest neighbors in a multidimensional space it's a lot quicker than computing the distance for all pairs: import numpy as np from scipy import spatial a = np.random.randint(low = 1, high = 100, size = (7)) b = np.array([34.78, 34.8, 35.1, 34.99, 11.3, 10.7, 11.289, 18.78, 19.1, 20.05, 12.32, 12.87, 13.5, 31.03, 31.15, 29.87, 48.1, 48.5, 49.2]) kd_tree = spatial.KDTree(np.expand_dims(b, 1)) # This returns distance to the nearest neighbor d # and position of the nearest neighbor i d,i = kd_tree.query(np.expand_dims(a, 1), k=[1])
Question: When running the code below I get the first item not the last. Why is that? async def example(): db: google.cloud.firestore.AsyncClient = AsyncClient() await db.document("1/1").set({"a": "1"}) await db.document("1/2").set({"a": "2"}) last = ( await ( db.collection("1") .order_by("a") .limit_to_last(1) # <= not working - getting the first not last .get() ) )[0].to_dict() first = ( await ( db.collection("1") .order_by("a") .limit(1) .get() ) )[0].to_dict() print(first, last) # {'a': '1'} {'a': '1'} asyncio.run(example())  The workaround is the use of descending ordering with limit(). But I don't understand - what is the purpose of the limit_to_last() function?

Answer: If your database schema looks like this: db | --- 1 (collection) | --- 1 (document) | | | --- a: "1" | --- 2 (document) | --- a: "2"  Then the following query: db.collection("1").order_by("a").limit_to_last(1)  Will indeed return: {'a': '2'}  Because the last element in the results is the second document. I tested and it worked. However, when you want to store numbers, it's not recommended to store them as strings but as numbers. Why? Because when you order strings the order is lexicographically.
Question: I have something like this: from unittest.mock import MagicMock class A: pass class B(A): pass mock_B = MagicMock(spec_set=B) assert issubclass(mock_B, A) # TypeError: issubclass() arg 1 must be a class  How can I get this to pass? isinstance and Mocking had a lot of stuff various answers, but from there I can't figure it out.

Answer: The problem here is that you want to treat mock_B, an instance of MagicMock, as a class, but an object in Python can only be considered as a class if it is an instance of type, which mock_B is simply not. So instead of trying to make a Mock object class-like, which is technically impossible since Python does not allow dynamically changing the type of an object, a workaround would be to make a class Mock-like. To do that, we can create a class with the same name and base classes as the class you want to mock, but then delegate all of its attribute lookups to a Mock object by making the __getattribute__ method of the class' metaclass perform such a delegation. This way, the new class passes the issubclass check easily because it is really a subclass of the intended base class A, and yet it also behaves like a Mock object because all attribute lookups are delegated to one. However, one problem with delegating all attribute lookups to a Mock object is that the resulting "class" would not behave like a class as the Mock object has no class-like attributes. We can solve this by defaulting attributes that aren't available in the Mock object to the class being mocked instead: from unittest.mock import MagicMock def mock_class(cls): class meta(type): def __getattribute__(self, name): try: return getattr(mock, name) except AttributeError: return getattr(cls, name) mock = MagicMock(spec_set=cls) return meta(cls.__name__, cls.__bases__, {})  so that: class A: pass class B(A): def foo(self): pass mock_B = mock_class(B) assert issubclass(mock_B, A) print(mock_B) print(mock_B.foo) print(mock_B.__name__) print(mock_B.__bases__)  passes the issubclass assertion and outputs: <class '__main__.B'> <MagicMock name='mock.foo' id='1970990059024'> B (<class '__main__.A'>,)  Demo: Try it online!
Question: I have a list word_list = ['cat', 'dog', 'rabbit']. I want to use list comprehension to print each individual character from the list but removes any duplicate character. This is my code: word_list = ['cat', 'dog', 'rabbit'] letter_list = [""] letter_list = [letter for word in word_list for letter in word if letter not in letter_list ] print(letter_list)  this returns ['c', 'a', 't', 'd', 'o', 'g', 'r', 'a', 'b', 'b', 'i', 't'] which is not the desired result ['c', 'a', 't', 'd', 'o', 'g', 'r', 'b', 'i'] and I can't figure out why.

Answer: It is technically possible to implement deduplication with a list comprehension if initialization of some variables are allowed. You can use a set seen to keep track of letters already encountered, and a set include to record whether the current letter was already seen before it is added to the set seen: seen = set() include = set() print([ letter for word in word_list for letter in word if ( include.clear() if letter in seen else include.add(1), seen.add(letter) ) and include ])  Since Python 3.8 you can also use an assignment expression to avoid having to rely on side effects of functions, which are generally discouraged in a list comprehension: seen = set() print([ letter for word in word_list for letter in word if ( include := letter not in seen, seen := seen | {letter} ) and include ])  But if you are not dead set on implementing the deduplication with a list comprehension, it would be cleaner to use the dict.fromkeys method instead since dict keys are always unique and follow insertion order since Python 3.7: from itertools import chain print([*{}.fromkeys(chain(*word_list))])  Demo: Try it online!
Question: I modeled a LGBM binary classification model, however I am not interested in the classification itself, I am interested in the probabilities of a given set of observations to output a positive classification. I wanted to see how my model works using SHAP values, but I am struggling to find how to find the SHAP values for the probability values, mainly because the output is a vector with two values. Here is how I tried to do it: import shap model = LGBMClassifier() model = model.fit(X_train,y_train) # Get SHAP values explainer = shap.TreeExplainer(model, data=X_train, feature_perturbation="interventional", model_output="probability") shap_values = explainer.shap_values(X_train)  The only plot that works with the SHAP values generated is the summary plot, which vale values that range from -0.2 to 0.4 . I just want to be safe that those values are really referring to probabilities and would like to see it in a force plot, waterfall plot or any other plot.

Answer: Try link="logit" for the force_plot: import pandas as pd import numpy as np import shap import lightgbm as lgbm from sklearn.model_selection import train_test_split from sklearn.datasets import load_breast_cancer from scipy.special import expit shap.initjs() data = load_breast_cancer() X = pd.DataFrame(data.data, columns=data.feature_names) y = data.target X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=42 ) model = lgbm.LGBMClassifier() model.fit(X_train, y_train) explainer_raw = shap.TreeExplainer(model) shap_values = explainer_raw(X_train) # force plot of first row for class 1 class_idx = 1 row_idx = 0 expected_value = explainer_raw.expected_value[class_idx] shap_value = shap_values[:, :, class_idx].values[row_idx] shap.force_plot( base_value=expected_value, shap_values=shap_value, features=X_train.iloc[row_idx, :], link="logit", # <-- here )
Question: I would like to create automated examples of valid data based on my pydantic models. How can I do this? Example: import pydantic from typing import Any class ExampleData(pydantic.BaseModel): a: int b: str = pydantic.Field(min_length=10, max_length=10) @staticmethod def example() -> dict[str, Any]: # some logic return {} a.example() """Returns { "a": 1, "b": "0123456789" } """  P.S. I suspect that pydantic provides this functionality because fastapi generates sample data, but I'm not sure if this is exactly its functionality and I couldn't find such a method. Can any one help me understand this?

Answer: Here's how you can do it using pydantic and Faker:  Installation  pip install Faker pip install pydantic   Script  import uuid from datetime import date, datetime, timedelta from typing import List, Union from pydantic import BaseModel, UUID4 from faker import Faker # your pydantic model class Person(BaseModel): id: UUID4 name: str hobbies: List[str] age: Union[float, int] birthday: Union[datetime, date] class PersonFactory: @classmethod def generate_id(cls): return str(uuid.uuid4()) @classmethod def generate_name(cls): # Implement your own logic to generate realistic names return Faker().name() @classmethod def generate_hobbies(cls): # Implement your own logic to generate hobbies return Faker().words(nb=1) @classmethod def generate_age(cls): # Implement your own logic to generate realistic ages return Faker().random_int(1940, 2023) @classmethod def generate_birthday(cls): # Implement your own logic to generate realistic birthdays return Faker().date_of_birth(tzinfo=None, minimum_age=18, maximum_age=80) @classmethod def build(cls): id = cls.generate_id() name = cls.generate_name() hobbies = cls.generate_hobbies() birthday = cls.generate_birthday() age = datetime.now().year - birthday.year return Person(id=id, name=name, hobbies=hobbies, age=age, birthday=birthday) result = PersonFactory.build() print(result)   Output:  id=UUID('4b7ffc04-48a1-4f4d-8c5b-d3167717dd69') name='Katherine Brown' hobbies=['stay'] age=30 birthday=datetime.date(1993, 2, 17)  This will generate a fake person data everytime you run PersonFactory.build()
Question: In a code that evaluates a web response, I would like to zip elements from several lists. However, the elements of the iterators are dict's. Therefore, I would like to fill up the missing values also with dict's, but each generated element should have it's own dict instance. The following code groups elements from each list by itertools.zip_longest. As long as there is a non mutable fillvalue specified, there is no problem. import collections import itertools l1 = [{"a": 100}, {"b": 200}, {"c": 300}] l2 = [{"d": 400}] ll = list(itertools.zip_longest(l1, l2, fillvalue=0)) print(ll)  -> [({'a': 100}, {'d': 400}), ({'b': 200}, 0), ({'c': 300}, 0)] Now, when a mutable fillvalue is specified, all the fillvalue's share the same instance and so changing one, changes all: import collections import itertools l1 = [{"a": 100}, {"b": 200}, {"c": 300}] l2 = [{"d": 400}] ll = list(itertools.zip_longest(l1, l2, fillvalue=dict())) ll[1][1]["x"] = 150 print(ll)  -> [({'a': 100}, {'d': 400}), ({'b': 200}, {'x': 150}), ({'c': 300}, {'x': 150})] To prevent that all the dicts share the same instance I used copy.deepcopy: import collections import copy import itertools l1 = [{"a": 100}, {"b": 200}, {"c": 300}] l2 = [{"d": 400}] ll = list(itertools.zip_longest(l1, l2, fillvalue=copy.deepcopy(dict()))) ll[1][1]["x"] = 150 print(ll)  -> [({'a': 100}, {'d': 400}), ({'b': 200}, {'x': 150}), ({'c': 300}, {'x': 150})] As a result, still all dict's from the fillvalue share the same instance. I would like to add that ll = [item or dict() for item in itertools.zip_longest(l1, l2)] works neither, assuming a fillvalue of None. So, how can I make each fillvalue unique?

Answer: You can use a sentinel value and wrap zip_longest in another pipeline. For example: sentinel = object() l = list(tuple({} if x is sentinel else x for x in items) for items in zip_longest(l1, l2, fillvalue=sentinel))  For this example, you can probably use None rather than a custom sentinel object.
Question: I am using pandas to perform data analysis, I have a datetime column and I am using describe() as below. s = pd.Series([np.datetime64("2000-01-01"), np.datetime64("2010-01-01"), np.datetime64("2010-01-01")]) print('--------------------') print(s) print('--------------------') print(s.dtype) print('--------------------') print(s.describe())  I am getting output as below: -------------------- 0 2000-01-01 1 2010-01-01 2 2010-01-01 dtype: datetime64[ns] -------------------- datetime64[ns] -------------------- count 3 mean 2006-09-01 08:00:00 min 2000-01-01 00:00:00 25% 2004-12-31 12:00:00 50% 2010-01-01 00:00:00 75% 2010-01-01 00:00:00 max 2010-01-01 00:00:00 dtype: object  I was going through documentation, The above output summary should be generated for numeric datatype as opposed to non-numeric datatypes like object, category, timestamp which have different summary. pandas version: pandas = "^2.0.0" documentation: https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.describe.html What I am missing, does datetime datatype is treated as numeric datatype inside describe()?

Answer: Yes, internally datetime objects are stored as integers (number of nanoseconds since epoch), making them inherently numeric data. For the reported data, all values (['count', 'mean', 'min', '25%', '50%', '75%', 'max']) are valid as they represent the count or a date in the min-max range. You can note that std is absent, since computing a standard deviation wouldn't make sense. The behavior on datetime data is actually demonstrated in the describe documentation:  Describing a timestamp Series. s = pd.Series([ np.datetime64("2000-01-01"), np.datetime64("2010-01-01"), np.datetime64("2010-01-01") ]) s.describe() count 3 mean 2006-09-01 08:00:00 min 2000-01-01 00:00:00 25% 2004-12-31 12:00:00 50% 2010-01-01 00:00:00 75% 2010-01-01 00:00:00 max 2010-01-01 00:00:00 dtype: object   Internally, describe is handled in pandas/core/methods/describe.py. The select_describe_func function is choosing how to handle the data based on the dtype and a specific approach (describe_timestamp_1d) is used for datetime data. Follow-up: "Using astype on the above example, if the datatype is set as datetime then summary is numeric and if the datatype is set as object then summary is non-numeric. So, In general while performing analysis on such column, do we set datatype as datetime or object, which is preferred?" When using s.astype('object').describe() you get the same behavior as you would have for strings: s.astype('object').describe() count 3 # number of values unique 2 # number of unique values top 2010-01-01 00:00:00 # most frequent value freq 2 # count of the most frequent value dtype: object  It's up to you to determine which behavior you need, if you use your dates as datetime information then keep the original type, if you use them as a Categorical (for instance if you have 2 or 3 different dates and their absolute value is not meaningful other than to define a group, then the non-numeric approach is probably fine)
Question: I am trying to setup a tls context in python. I want to force TLSv1.3 usng: context = ssl.SSLContext(ssl.PROTOCOL_TLSv1_3)  This does not work as I receive the following error: AttributeError: module 'ssl' has no attribute 'PROTOCOL_TLSv1_3'  I am on Ubuntu 20.04 and am using python version 3.8 and openssl version 1.1.1f. Why doesn't it support TLSv1.3?

Answer: TLS 1.3 protocol will be available with PROTOCOL_TLS in OpenSSL >= 1.1.1. There is no dedicated PROTOCOL constant for just TLS 1.3. https://docs.python.org/3.11/library/ssl.html#ssl.SSLContext Footnote 3. According to the protocol version description, this is how you set TLS 1.3 as the minimum version supported: client_context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT) client_context.minimum_version = ssl.TLSVersion.TLSv1_3 client_context.maximum_version = ssl.TLSVersion.TLSv1_3  https://docs.python.org/3.11/library/ssl.html#protocol-versions https://docs.python.org/3.11/library/ssl.html#tls-1-3