eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: First time using dataclass, also not really good at Python. The following behaviour conflicts with my understanding so far: from dataclasses import dataclass @dataclass class X: x: int = 1 y: int = 2 @dataclass class Y: c1: X = X(3, 4) c2: X = X(5, 6) n1 = Y() n2 = Y() print(id(n1.c1)) print(id(n2.c1)) n1.c1.x = 99999 print(n2)  This prints 140459664164272 140459664164272 Y(c1=X(x=99999, y=4), c2=X(x=5, y=6))  Why does c1 behave like a class variable? What can I do to keep n2.c1 != n1.c1, do I need to write an init function? I can get sensible results with this addition to Y:  def __init__(self): self.c1 = X(3, 4) self.c2 = X(5, 6)  prints: 140173334359840 140173335445072 Y(c1=X(x=3, y=4), c2=X(x=5, y=6)) 

Answer:  Why does c1 behave like a class variable?  Because you specified default value for them and they're now a class attribute. In the Mutable Default Values section, it's mentioned:  Python stores default member variable values in class attributes.  But look at this: @dataclass class X: x: int = 1 y: int = 2 @dataclass class Y: c1: X c2: X = X(5, 6) print("c1" in Y.__dict__) # False print("c2" in Y.__dict__) # True  c1 doesn't have default value so it's not in class's namespace. Indeed by doing so(defining default value), Python stores that c1 and c2 inside both instance's namespace (n1.__dict__) and class's namespace (Y.__dict__). Those are the same objects, only the reference is passed: @dataclass class X: x: int = 1 y: int = 2 @dataclass class Y: c1: X = X(3, 4) c2: X = X(5, 6) n1 = Y() n2 = Y() print("c1" in Y.__dict__) # True print("c1" in n1.__dict__) # True print(id(n1.c1)) # 140037361903232 print(id(n2.c1)) # 140037361903232 print(id(Y.c1)) # 140037361903232  So now, If you want them to be different you have several options:  Pass arguments while instantiating (Not a good one):  @dataclass class X: x: int = 1 y: int = 2 @dataclass class Y: c1: X = X(3, 4) c2: X = X(5, 6) n1 = Y(X(3, 4), X(5, 6)) n2 = Y(X(3, 4), X(5, 6)) print("c1" in Y.__dict__) # True print("c1" in n1.__dict__) # True print(id(n1.c1)) # 140058585069264 print(id(n2.c1)) # 140058584543104 print(id(Y.c1)) # 140058585065088   Use field and pass default_factory:  from dataclasses import dataclass, field @dataclass class X: x: int = 1 y: int = 2 @dataclass class Y: c1: X = field(default_factory=lambda: X(3, 4)) c2: X = field(default_factory=lambda: X(5, 6)) n1 = Y() n2 = Y() print("c1" in Y.__dict__) # False print("c1" in n1.__dict__) # True print(id(n1.c1)) # 140284815353136 print(id(n2.c1)) # 140284815353712  In the second option, because I didn't specify default parameter(you can't mix both), nothing is going to be stored in the class's namespace. field(default=SOMETHING) is another way of saying = SOMETHING.
Question: I want to download text files using python, how can I do so? I used requests module's urlopen(url).read() but it gives me the bytes representation of file.

Answer: When downloading text files with python I like to use the wget module import wget remote_url = 'https://www.google.com/test.txt' local_file = 'local_copy.txt' wget.download(remote_url, local_file)  If that doesn't work try using urllib from urllib import request remote_url = 'https://www.google.com/test.txt' file = 'copy.txt' request.urlretrieve(remote_url, file)  When you are using the request module you are reading the file directly from the internet and it is causing you to see the text in byte format. Try to write the text to a file then view it manually by opening it on your desktop import requests remote_url = 'test.com/test.txt' local_file = 'local_file.txt' data = requests.get(remote_url) with open(local_file, 'wb')as file: file.write(data.content)
Question: I have a standard dataframe like the one below :  Id Type Speed Efficiency Durability 0 Id001 A OK OK nonOK 1 Id002 A nonOK OK nonOK 2 Id003 B nonOK nonOK nonOK 3 Id004 B nonOK nonOK OK 4 Id005 A nonOK nonOK OK 5 Id006 A OK OK OK 6 Id007 A OK nonOK OK 7 Id008 B nonOK nonOK OK 8 Id009 C OK OK OK 9 Id010 B OK OK nonOK 10 Id011 C OK nonOK OK 11 Id012 C OK nonOK OK 12 Id013 C nonOK OK OK 13 Id014 C nonOK nonOK OK 14 Id015 C nonOK nonOK OK  And I'm trying to get this kind of output :  Type Test Speed Efficiency Durability 0 A OK 3 3 3 1 A nonOK 2 2 2 2 B OK 1 1 2 3 B nonOK 3 3 2 4 C OK 3 2 6 5 C nonOK 3 4 0  I tried with df.groupby('Type').agg('count') but it doesn't give the expected output. Is it possible to make this kind of transformation with pandas, please ?

Answer: You could try as follows: out = df.groupby('Type').agg({col:'value_counts' for col in df.columns[2:]})\ .fillna(0).astype(int).sort_index().reset_index().rename( columns={'level_1':'Test'}) print(out) Type Test Speed Efficiency Durability 0 A OK 3 3 3 1 A nonOK 2 2 2 2 B OK 1 1 2 3 B nonOK 3 3 2 4 C OK 3 2 6 5 C nonOK 3 4 0
Question: I am trying to build a model with a layer of torch.nn.linear with same input size and output size, so the layer would be square matrix. In this model, I want the diagonal elements of this matrix fixed to zero. Which means, during training, I don't want the diagonal elements to be changed from zero. I could only think of adding some kind of step that change diag elements to zero for each training epoch, but I am not sure if this is valid or efficient way. Is there a definite way of making this kind of layer which can ensure that the diagonal elements don't change? Sorry if my question is weird.

Answer: You can always implement your own layers. Note that all custom layers should be implemented as classes derived from nn.Module. For example: class LinearWithZeroDiagonal(nn.Module): def __init__(self, num_features, bias): super(LinearWithZeroDiagonal, self).__init__() self.base_linear_layer = nn.Linear(num_features, num_features, bias) def forward(self, x): # first, make sure the diagonal is zero with torch.no_grad(): self.base_linear_layer.weight.fill_diagonal_(0.) return self.base_linear_layer(x)
Question: I have a list of objects, and want to check if part of the list matches a specific pattern. Consider the following lists: l1 = ["foo", "bar"] l2 = [{1, 2},"foo", "bar"] l3 = ["foo", "bar", 5] l4 = [{1,2},"foo", "bar", 5, 6]  How would I match the sequence ["foo", "bar"] in all the different cases? My naive idea is: match l4: case [*_, "foo", "bar", *_]: print("matched!")  Unfortunately this is a SyntaxError: multiple starred names in sequence pattern. The issue is, that I don't know how many elements are leading and trailing the pattern. Edit: I think I need to clarify: "foo", "bar" is just a stand-in for a much more complex pattern. (I am working with an AST object)

Answer: def struct_match(lst_target, lst_pattern): for i in range(len(lst_target)-(len(lst_pattern)-1)): if lst_target[i:i+len(lst_pattern)] == lst_pattern: print('matched!') break  l1 = ["foo", "bar"] l2 = [{1, 2},"foo", "bar"] l3 = ["foo", "bar", 5] l4 = [{1,2},"foo", "bar", 5, 6] l5 = [{1,2},"foo", "baz", "bar", 5, 6] patt = ["foo", "bar"] struct_match(l1, patt) struct_match(l2, patt) struct_match(l3, patt) struct_match(l4, patt) struct_match(l5, patt) # outputs matched! matched! matched! matched!  PS: I just found a beautiful recursive solution here (recursion is always beautiful... if your list is not too long)
Question: After Python 3.6, we have typing.NamedTuple, which is a typed version of collections.namedtuple(), we can inherit it like a class: class Employee(NamedTuple): name: str id: int  Compared with collections.namedtuple, this syntax is more beautiful, but I still can't understand its implementation, whether we look at typing.py file, or do some simple tests, we will find that it is a function rather than a class: # Python 3.10.6 typing.py def NamedTuple(typename, fields=None, /, **kwargs): """...""" if fields is None: fields = kwargs.items() elif kwargs: raise TypeError("Either list of fields or keywords" " can be provided to NamedTuple, not both") try: module = sys._getframe(1).f_globals.get('__name__', '__main__') except (AttributeError, ValueError): module = None return _make_nmtuple(typename, fields, module=module)  >>> type(NamedTuple) <class 'function'>  I understand that it uses some metaclass magic, but I don't understand what happens when using class MyClass(NamedTuple). For this reason, I have tried to customize a function to inherit: >>> def func_for_inherit(*args, **kwargs): ... print(args, kwargs) ... >>> class Foo(func_for_inherit): ... foo: str ... Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: function() argument 'code' must be code, not str  Well, this got a result that I can't understand. When inheriting a user-defined function, it seems that its class was called. What happened behind this?

Answer: typing.NamedTuple uses a really esoteric feature, __mro_entries__:  If a base that appears in class definition is not an instance of type, then an __mro_entries__ method is searched on it. If found, it is called with the original bases tuple. This method must return a tuple of classes that will be used instead of this base. The tuple may be empty, in such case the original base is ignored.  Immediately after the NamedTuple function definition, the following code appears: _NamedTuple = type.__new__(NamedTupleMeta, 'NamedTuple', (), {}) def _namedtuple_mro_entries(bases): if len(bases) > 1: raise TypeError("Multiple inheritance with NamedTuple is not supported") assert bases[0] is NamedTuple return (_NamedTuple,) NamedTuple.__mro_entries__ = _namedtuple_mro_entries  This sets NamedTuple.__mro_entries__ to a function that tells the class creation system to use an actual class, _NamedTuple, as the base class. (_NamedTuple then uses metaclass features to customize the class creation process further, and the end result is a class that directly inherits from tuple.)
Question: I have created the docker image to run python script. Now when I run below command it throwing error. command in powershell: docker build -t pull_request_summary . Error: Traceback (most recent call last): File "/usr/app/src/./listPullRequest.py", line 1, in import requests ModuleNotFoundError: No module named 'requests' Just to confirm the "requests module is already installed on my machine. The script is running fine if I directly run from PowerShell. It just throwing error while running docker images. My question is where it is expecting the "request" module of python?

Answer: your container also must have requests installed on it In your docker file put this line RUN pip install requests
Question: Suppose I have a list/tuple of strings, COLOURS = [ "White", "Black", "Red" "Green", "Blue" ] for c in COLOURS: # rest of the code  Sometimes I forget placing a comma after each entry in the list ("Red" in the above snippet). This results in one "RedGreen" instead of two separate "Red" and "Green" list items. Since this is valid Python, no IDE/text editor shows a warning/error. The incorrect value comes to the limelight only during testing. What writing style or code structure should I use to prevent this?

Answer: You're incorrect that "no IDE/text editor shows a warning/error". Pylint can identify this problem using rule implicit-str-concat (W1404) with flag check-str-concat-over-line-jumps. (And for that matter, there are lots of things that are valid Python that a linter will warn you about, like bare except: for example.) Personally, I'm using VSCode, so I enabled Pylint via the Python extension (python.linting.pylintEnabled) and set up a pylintrc like this: [tool.pylint] check-str-concat-over-line-jumps = yes  Now VSCode gives this warning for your list:  Implicit string concatenation found in list pylint(implicit-str-concat) [Ln 4, Col 1]   Lastly, there are probably other linters that can find the same problem, but Pylint is the first one I found.
Question: when I write a list in pandas when I read it, its dtype is string and not array, is there any way to write a column of list in such a way that be again array type when we read it? Here is what I mean: d=[['d',['A','B','C']],['p',['F','G']]] df=pd.DataFrame(d) df.to_csv('file.csv')  when I run the following code, pd.read_csv('file.csv')['1'].values[0]  the output is:  "['A', 'B', 'C']"  but I want this:  ['A', 'B', 'C'] 

Answer: one solution would be to run it through literal_eval. you make a dictionary with the column name as key and the converter function as value. and pass that into read_csv with the keyword converters Note that if your column has mixed data (also strings and other stuff) you might want to write a custom function which filters and converts the different types from ast import literal_eval df1 = pd.read_csv('file.csv', converters={'1': literal_eval}) df1  output:  type(df1["1"][0])  output:
Question: I'm using FastAPI with Redis. My app looks something like this from fastapi import FastAPI import redis # Instantiate redis client r = redis.Redis(host='localhost', port=6379, db=0, decode_responses=True) # Instantiate fastapi app app = FastAPI() @app.get("/foo/") async def foo(): x = r.get("foo") return {"message": x} @app.get("/bar/") async def bar(): x = r.get("bar") return {"message": x}  Is it bad practice to create r as a module-scoped variable like this? If so what are the drawbacks? In Tiangolo's tutorial on setting up a SQL database connection he uses a dependency, which I guess in my case would look something like this from fastapi import Depends, FastAPI import redis # Instantiate fastapi app app = FastAPI() # Dependency def get_redis(): return redis.Redis(host='localhost', port=6379, db=0, decode_responses=True) @app.get("/foo/") async def foo(r = Depends(get_redis)): x = r.get("foo") return {"message": x} @app.get("/bar/") async def bar(r = Depends(get_redis)): x = r.get("bar") return {"message": x}  I'm a bit confused as to which of these methods (or something else) would be preferred and why.

Answer: Depends will evaluate every time your function got a request, so your second example will create a new connection for each request. As @JarroVGIT said, we can use connection pooling to maintain the connection from FastAPI to Redis and reduce open-closing connection costs. Usually, I create a different file to define the connection. Let's say we have config/db.py: import redis def create_redis(): return redis.ConnectionPool( host='localhost', port=6379, db=0, decode_responses=True ) pool = create_redis()  Then in the main.py from fastapi import Depends, FastAPI import redis from config.db import pool app = FastAPI() def get_redis(): # Here, we re-use our connection pool # not creating a new one return redis.Redis(connection_pool=pool) @app.get("/items/{item_id}") def read_item(item_id: int, cache = Depends(get_redis)): status = cache.get(item_id) return {"item_name": status} @app.put("/items/{item_id}") def update_item(item_id: int, cache = Depends(get_redis)): cache.set(item_id, "available") return {"status": "available", "item_id": item_id}  Usually, I also split the dependencies file like the doc so we can call it from our routing module, but for simplicity, I will leave it like this. You can check this repo to experiment by yourself. It has more comprehensive code and I have already created several scenarios that might help you understand the difference. And it will also cover how your first example may block other endpoints.
Question: I have a dataset like this: a = tf.data.Dataset.range(1, 16) b = tf.data.Dataset.range(16, 32) zipped = tf.data.Dataset.zip((a, b)) list(zipped.as_numpy_iterator()) # output: [(0, 16), (1, 17), (2, 18), (3, 19), (4, 20), (5, 21), (6, 22), (7, 23), (8, 24), (9, 25), (10, 26), (11, 27), (12, 28), (13, 29), (14, 30), (15, 31)]  When I apply batch(4) to it, the expected result is an array of batches, where each batch contains four tuples: [[(0, 16), (1, 17), (2, 18), (3, 19)], [(4, 20), (5, 21), (6, 22), (7, 23)], [(9, 24), (10, 25), (10, 26), (11, 27)], [(12, 28), (13, 29), (14, 30), (15, 31)]]  But this is what I receive instead: batched = zipped.batch(4) list(batched.as_numpy_iterator()) # Output: [(array([0, 1, 2, 3]), array([16, 17, 18, 19])), (array([4, 5, 6, 7]), array([20, 21, 22, 23])), (array([ 8, 9, 10, 11]), array([24, 25, 26, 27])), (array([12, 13, 14, 15]), array([28, 29, 30, 31]))]  I'm following this tutorial, he does the same steps but gets the correct output somehow.  Update: according to the documentation this is the intended behavior:  The components of the resulting element will have an additional outer dimension, which will be batch_size  But it doesn't make any sense. To my understanding, dataset is a list of pieces of data. It doesn't matter the shape of those pieces of data, when we are batching it we are combining the elements [whatever their shape is] into batches, therefore it should always insert the new dimention to the second position ((length, a, b, c) -> (length', batch_size, a, b, c)). So my questions are: I wonder what is the purpose of batch() being implemented this way? And what is the alternative that does what I described?

Answer: One thing you can try doing is something like this: import tensorflow as tf a = tf.data.Dataset.range(16) b = tf.data.Dataset.range(16, 32) zipped = tf.data.Dataset.zip((a, b)).batch(4).map(lambda x, y: tf.transpose([x, y])) list(zipped.as_numpy_iterator())  [array([[ 0, 16], [ 1, 17], [ 2, 18], [ 3, 19]]), array([[ 4, 20], [ 5, 21], [ 6, 22], [ 7, 23]]), array([[ 8, 24], [ 9, 25], [10, 26], [11, 27]]), array([[12, 28], [13, 29], [14, 30], [15, 31]])]  but they are still not tuples. Or: zipped = tf.data.Dataset.zip((a, b)).batch(4).map(lambda x, y: tf.unstack(tf.transpose([x, y]), num = 4))  [(array([ 0, 16]), array([ 1, 17]), array([ 2, 18]), array([ 3, 19])), (array([ 4, 20]), array([ 5, 21]), array([ 6, 22]), array([ 7, 23])), (array([ 8, 24]), array([ 9, 25]), array([10, 26]), array([11, 27])), (array([12, 28]), array([13, 29]), array([14, 30]), array([15, 31]))]
Question: How do I load a custom yolo v-7 model. This is how I know to load a yolo v-5 model : model = torch.hub.load('ultralytics/yolov5', 'custom', path='yolov5/runs/train/exp15/weights/last.pt', force_reload=True)  I saw videos online and they suggested to use this : !python detect.py --weights runs/train/yolov7x-custom/weights/best.pt --conf 0.5 --img-size 640 --source final_test_v1.mp4  But I want it to be loaded like a normal model and give me the bounding box co-ordinates of where ever it found the objects. This is how I did it in yolo v-5: from models.experimental import attempt_load yolov5_weight_file = r'weights/rider_helmet_number_medium.pt' # ... may need full path model = attempt_load(yolov5_weight_file, map_location=device) def object_detection(frame): img = torch.from_numpy(frame) img = img.permute(2, 0, 1).float().to(device) #convert to required shape based on index img /= 255.0 if img.ndimension() == 3: img = img.unsqueeze(0) pred = model(img, augment=False)[0] pred = non_max_suppression(pred, conf_set, 0.20) # prediction, conf, iou # print(pred) detection_result = [] for i, det in enumerate(pred): if len(det): for d in det: # d = (x1, y1, x2, y2, conf, cls) x1 = int(d[0].item()) y1 = int(d[1].item()) x2 = int(d[2].item()) y2 = int(d[3].item()) conf = round(d[4].item(), 2) c = int(d[5].item()) detected_name = names[c] # print(f'Detected: {detected_name} conf: {conf} bbox: x1:{x1} y1:{y1} x2:{x2} y2:{y2}') detection_result.append([x1, y1, x2, y2, conf, c]) frame = cv2.rectangle(frame, (x1, y1), (x2, y2), (255,0,0), 1) # box if c!=1: # if it is not head bbox, then write use putText frame = cv2.putText(frame, f'{names[c]} {str(conf)}', (x1, y1), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0,0,255), 1, cv2.LINE_AA) return (frame, detection_result) 

Answer: You cannot use attempt_load from the Yolov5 repo as this method is pointing to the ultralytics release files. You need to use attempt_load from Yolov7 repo as this one is pointing to the right files. # yolov7 def attempt_download(file, repo='WongKinYiu/yolov7'): # Attempt file download if does not exist file = Path(str(file).strip().replace("'", '').lower()) ...  # yolov5 def attempt_download(file, repo='ultralytics/yolov5', release='v6.2'): # Attempt file download from GitHub release assets if not found locally. release = 'latest', 'v6.2', etc. from utils.general import LOGGER def github_assets(repository, version='latest'): ...  Then you can download it like this: # load yolov7 method from models.experimental import attempt_load model = attempt_load('yolov7.pt', map_location='cuda:0') # load FP32 model
Question: I want to efficiently generate pairs of elements from two lists equal to their Cartesian product with some elements omitted. The elements in each list are unique. The code below does exactly what's needed but I'm looking to optimize it perhaps by replacing the loop. See the comments in the code for details. Any advice would be appreciated. from itertools import product from pprint import pprint as pp def pairs(list1, list2): """ Return all combinations (x,y) from list1 and list2 except: 1. Omit combinations (x,y) where x==y """ tuples = filter(lambda t: t[0] != t[1], product(list1,list2)) """ 2. Include only one of the combinations (x,y) and (y,x) """ result = [] for t in tuples: if not (t[1], t[0]) in result: result.append(t) return result list1 = ['A', 'B', 'C'] list2 = ['A', 'D', 'E'] pp(pairs(list1, list1)) # Test a list with itself pp(pairs(list1, list2)) # Test two lists with some common elements  Output [('A', 'B'), ('A', 'C'), ('B', 'C')] [('A', 'D'), ('A', 'E'), ('B', 'A'), ('B', 'D'), ('B', 'E'), ('C', 'A'), ('C', 'D'), ('C', 'E')] 

Answer: About 5-6 times faster than the fastest in your answer's benchmark. I build sets of values that appear in both lists or just one, and combine them appropriately without further filtering. from itertools import product, combinations def pairs(list1, list2): a = {*list1} b = {*list2} ab = a & b return [ *product(a, b-a), *product(a-b, ab), *combinations(ab, 2) ]  You could also make it an iterator (because unlike previous solutions, I don't need to store the already produced pairs to filter further ones): from itertools import product, combinations, chain def pairs(list1, list2): a = {*list1} b = {*list2} ab = a & b return chain( product(a, b-a), product(a-b, ab), combinations(ab, 2) )
Question: How can one extend __class_getitem__ for a Python Generic class? I want to add arguments to __class_getitem__ while having some be propagated upwards to Generic.__class_getitem__. Please see the below code snippet for an example use case (that doesn't run): from typing import ClassVar, Generic, TypeVar T = TypeVar("T") class Foo(Generic[T]): cls_attr: ClassVar[int] def __class_getitem__(cls, cls_attr: int, item): cls.cls_attr = cls_attr return super().__class_getitem__(item) def __init__(self, arg: T): pass foo = Foo[1, bool](arg=True)  Gives me this TypeError: Traceback (most recent call last): File "/path/to/file.py", line 17, in <module> foo = Foo[1, bool](arg=True) TypeError: Foo.__class_getitem__() missing 1 required positional argument: 'item' 

Answer: As @juanpa.arrivillaga suggests, this is the way to go: from typing import ClassVar, Generic, TypeVar T = TypeVar("T") class Foo(Generic[T]): cls_attr: ClassVar[int] def __class_getitem__(cls, item: tuple[int, T]): cls.cls_attr = item[0] return super().__class_getitem__(item[1]) def __init__(self, arg: T): self.arg = arg foo = Foo[1, bool](arg=True) assert foo.cls_attr == 1 assert foo.arg  Unfortunately, it looks like Python type inspection tooling is not advanced enough to understand this pattern. For example, mypy==0.971 (Sept 2022) doesn't support __class_getitem__ yet per https://github.com/python/mypy/issues/11501.
Question: I was trying to connect my Python files. Didn't go as planned and for some reason started source control without a clue how to get back.  Does anyone have a clue how I exit and just go back to normal?

Answer: You may have inadvertently clicked the Initialize Repository button. You can remove the source control by opening the directory in your File Manager and deleting the .git folder. That will remove the source control from your project. If you want to go further, you can right click on the source control icon:  and then click Hide 'Source Control'.