eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'm looking to troubleshoot my internet issue so I need a way to track both my latency and which application is using how much network bandwidth. I've already sorted out checking latency, but now I need a way to monitor each process' network usage (KB/s), like how it appears in Windows Task Manager. Before you suggest a program, unless it's able to record the values with a timestamp then that's not what I'm looking for. I'm asking for a Pythonic way because I need to record the network bandwidth and latency values at the same time so I can figure out if a specific process is causing latency spikes. So here's the info I need: Time | Process ID | Process Name | Down Usage | Up Usage | Network Latency | Also, please don't link to another Stackoverflow question unless you know their solution works. I've looked through plenty already and none of them work, which is why I'm asking again.

Answer: Following the third section of this guide provided me with all of the information listed in the post, minus latency. Given that you said you already had measuring latency figured out, I assume this isn't an issue. Logging this to csv/json/whatever is pretty easy, as all of the information is stored in panda data frames. As this shows the time the process was created, you can use datetime to generate a new timestamp at the time of logging. I tested this by logging to a csv after the printing_df variable was initialized, and had no issues.
Question: I want to know how can I draw text like this check image  As you can see text is on a green image and text has pink color background My code, this is part of my code I'm using PIL  draw = ImageDraw.Draw(background) font = ImageFont.truetype("assets/font2.ttf", 40) font2 = ImageFont.truetype("assets/font2.ttf", 70) arial = ImageFont.truetype("assets/font2.ttf", 30) name_font = ImageFont.truetype("assets/font.ttf", 30) para = textwrap.wrap(title, width=32) j = 0 draw.text( (10, 10), f"{h}", fill="red", font=name_font ) draw.text( (600, 150), "NOW PLAYING", fill="white", stroke_width=2, stroke_fill="white", font=font2, )   Thanks in advance :-) 

Answer: You can use the draw.textbbox method to get a bounding box for your text string and fill it using the draw.rectangle method. from PIL import Image, ImageDraw, ImageFont image = Image.new("RGB", (500, 100), "white") font = ImageFont.truetype("segoeui.ttf", 40) draw = ImageDraw.Draw(image) position = (10, 10) text = "Hello world" bbox = draw.textbbox(position, text, font=font) draw.rectangle(bbox, fill="red") draw.text(position, text, font=font, fill="black") image.show()   If you want a larger margin for the background rectangle, you can adjust the returned bounding box like so: left, top, right, bottom = draw.textbbox(position, text, font=font) draw.rectangle((left-5, top-5, right+5, bottom+5), fill="red") draw.text(position, text, font=font, fill="black")
Question: I have a folder containing a lot of sub-folders, with PDF files inside. It's a real mess to find information in these files, so I'm making a program to parse these folders and files, searching for a keyword in the PDF files, and returning the names of the PDF files containing the keyword. And it's working. Almost, actually. I have this error: PyPDF2.errors.PdfReadError: PDF starts with '♣▬', but '%PDF-' expected when my program reaches some folders (hard to know which one exactly). From my point of view, all the PDF files in my folders are the same, so I don't understand why my program works with some files and doesn't work with others. Thank you in advance for your responses.

Answer: disclaimer: I am the author of borb, the library mentioned in this answer PDF documents caught in the wild will sometimes start with non-pdf bytes (a header that is not really part of the PDF spec). This can cause all kinds of problems. PDF will (internally) keep track of all the byte offsets of objects in the file (e.g. "object 10 starts at byte 10202"). This header makes it harder to know where an object starts.  Do we start counting at the start of the file? Or at the start of where the file behaves like a PDF?  If you just want to extract text from a PDF (to be able to check it for content and keywords), you can try to use borb. borb will look for the start of the PDF within the first 1MB of the file (thus potentially ignoring your faulty header). If this turns out to corrupt the XREF (cross reference table, containing all byte addresses of objects) it will simply build a new one. This is an example of how to extract text from a PDF using borb: import typing from borb.pdf.document.document import Document from borb.pdf.pdf import PDF from borb.toolkit.text.simple_text_extraction import SimpleTextExtraction def main(): # read the Document doc: typing.Optional[Document] = None l: SimpleTextExtraction = SimpleTextExtraction() with open("output.pdf", "rb") as in_file_handle: doc = PDF.loads(in_file_handle, [l]) # check whether we have read a Document assert doc is not None # print the text on the first Page print(l.get_text_for_page(0)) if __name__ == "__main__": main()  You can find more examples in the examples repository.
Question: I have a Python package with an __init__.py that imports some things to be exposed as the package API. # __init__.py from .mymodule import MyClass # ...  I also want to be able to use the package as a command-line application, as in python -m mypackage, so I have a __main__.py file for that purpose: # __main__.py if __name__ == '__main__': from .main import main main()  So far so good. The problem is that, when the package is run as a program like this, I want to be able to do some stuff before importing any of the submodules - namely changing some environment variables before some third-party dependencies are loaded. I do not know how to do this, at least not in a reasonable way. Ideally, I would like to do something like: # __init__.py def thePackageIsRunningAsAnApplication(): # ??? def prepareEnvironment(): # ... if thePackageIsRunningAsAnApplication(): prepareEnvironment() from .mymodule import MyClass # ...  The problem is I don't think thePackageIsRunningAsAnApplication() can be implemented. The usual __name__ == '__main__' does not work here, because the main module being run is __main__.py, not __init__.py. In fact, I would prefer to define and run prepareEnvironment within __main__.py, but I don't know how to get that to run before the inner modules are loaded by __init__.py. I might (not sure, actually) work around it by lazily loading dependencies on my module, or somehow delaying the internal module loading or something, but I would prefer to avoid doing something like that just for this. EDIT: Thinking more about it, lazy loading probably would not work either. In the example, MyClass is, well, a class, not a submodule, so I cannot lazily load it. Moreover, MyClass happens to inherit from a class from that third-party dependency I was mentioning, so I cannot even define it without loading it.

Answer: It might make sense to add a separate entry point for running your code as a script, rather than using __main__.py, which as you've noticed, can only be run after the package's __init__.py is fully loaded. A simple script like run_mypackage.py located at the top level could contain the environment variable tweaking code, and then could import and run the package afterwards. def prepare_environment(): ... if __name__ == "__main__": prepare_environment() # adjust the environment firstt from mypackage.main import main # then load the package afterwards main()
Question: I have downloaded python for the first time in a new computer(ver 3.10.4). I have download the package python-certifi-win32, after someone suggested it as a solution to a SSL certificate problem in a similar question to a problem I had. Since then, pip has completely stopped working, to the point where i can't not run pip --version Every time the same error is printed, it is mostly seemingly junk(just a deep stack trace), but the file at the end is different. start of the printed log: Traceback (most recent call last): File "C:\Users\---\AppData\Local\Programs\Python\Python310\lib\importlib\_common.py", line 89, in _tempfile os.write(fd, reader()) File "C:\Users\---\AppData\Local\Programs\Python\Python310\lib\importlib\abc.py", line 371, in read_bytes with self.open('rb') as strm: File "C:\Users\---\AppData\Local\Programs\Python\Python310\lib\importlib\_adapters.py", line 54, in open raise ValueError() ValueError During handling of the above exception, another exception occurred:  last row of the printed log: PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\-----\\AppData\\Local\\Temp\\tmpunox3fhw' 

Answer: I found the answer in another question - PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: after installing python-certifi-win32 basically, you should remove two files that initialize python-certifi-win32 when running pip. the files are located in the directory: C:\Users\<username>\AppData\Local\Programs\Python\Python310\Lib\site-packages  and their names are: python-certifi-win32-init.pth distutils-precedence.pth  Shoutout to Richard from the mentioned post :)
Question: I'm writing my own wrapper around boto3 for quick firing functions. I'm trying to type annotate what boto3.session().client('ec2') returns. Debugger says it's <class 'botocore.client.EC2'>, but if I write it down like that, python crashes with a runtime error ec2: botocore.client.EC2 AttributeError: module 'botocore.client' has no attribute 'EC2'  Removing type annotation works for runtime, but it makes linting very limited. Is there a reasonably fast way or a hack to get typing working with this boto3 case? The code I'm talking about is following: class AWS_Client: # debugger says it's <class 'botocore.client.EC2'>, # but mypy says this class does not exist ec2: botocore.client.EC2 asg: botocore.client.AutoScaling region: str instance_id: str def __init__(self, profile_name: Optional[str] = None, instance_id: str = '', region_name: str = '', **kwargs) -> None: global config self.instance_id = instance_id or ec2_meta('instance-id').text self.region = region_name or ec2_meta( 'placement/availability-zone').text[:-1] boto3.set_stream_logger('botocore', level=logging.DEBUG) self.session = call_partial( boto3.Session, region_name=self.region, profile_name=profile_name, **kwargs) self.ec2 = self.session.client('ec2', region_name=self.region, **kwargs) self.asg = self.session.client( 'autoscaling', region_name=self.region, **kwargs) def get_tags(self) -> Dict[str, str]: self.tags = self.ec2.describe_tags(Filters=[{ 'Name': 'resource-id', 'Values': [self.instance_id] }])['Tags'] return self.tags aws = AWS_Client() print(aws.get_tags()) 

Answer: Try mypy-boto3 mypy plugin. You can install it via python -m pip install 'boto3-stubs[ec2]'  import boto3 from mypy_boto3_ec2 import EC2Client def foo(ec2_client: EC2Client) -> None: ... ec2_client = boto3.client("ec2") foo(ec2_client) # OK
Question: I have a plot that looks like this (this is the famous Wine dataset):  As you can see, the x-axis labels overlap and thus I need to be rotated. NB! I am not interested in rotating the x-ticks (as explained here), but the label text, i.e. alcohol, malic_acid, etc. The logic of creating the plot is the following: I create a grid using axd = fig.subplot_mosaic(...) and then for the bottom plots I set the labels with axd[...].set_xlabel("something"). Would be great if set_xlabel would take a rotation parameter, but unfortunately that is not the case.

Answer: Based on the documentation set_xlabel accepts text arguments, of which rotation is one. The example I used to test this is shown below, though . import matplotlib.pyplot as plt import numpy as np plt.plot() plt.gca().set_xlabel('Test', rotation='vertical')
Question: I have a dataframe of lists that looks similar to the one below (fig a). There is a single key column followed by n columns containing lists. My goal is that for each row, I will combine the lists from each column (excluding the key) into a single list in the new column, combined. An example of my desired result is below in figure B. I've tried some methods with iteritems(), but these dataframes have the potential to be hundreds of thousands to millions of rows long which made it incredibly slow. So I am trying to avoid solutions that use that. I'd like to use something like the list comprehension seen in this SO post, but I haven't been able to get it working with pandas. # example data data = {'key': ['1_1', '1_2', '1_3'], 'valueA': [[1, 2, 3, 4, 5, 6], [7, 8, 9, 10, 11, 12], [13, 14, 15, 16, 17, 18]], 'valueB': [[1, 2, 3, 4, 5, 6], [7, 8, 9, 10, 11, 12], [13, 14, 15, 16, 17, 18]], 'valueN': [[1, 2, 3, 4, 5, 6], [7, 8, 9, 10, 11, 12], [13, 14, 15, 16, 17, 18]]} dataSet = pd.DataFrame(data)  Figure A  Figure B  Edit: I really appreciate all the answers I have gotten so far! I'm currently going through and timing each on my full size dataset so I can figure out which one will work best this this case. I'll update with my result shortly! Edit 2: I tested the main solutions provided here on a few of my larger datasets and their average times are below. # Lambda/Apply a nested list comprehension shakiba.mrd: 1.12 s # Sum columns jfaccioni: 2.21 s # Nested list comprehension with iterrows mozway: 0.95 s # Adding column lists together politinsa: 3.50 s  Thanks again to everyone for their contributions!

Answer: You can use a nested list comprehension: dataSet['combined'] = [[e for l in x for e in l] for _,x in dataSet.filter(like='value').iterrows()]  Output:  key valueA valueB valueN combined 0 1_1 [1, 2, 3, 4, 5, 6] [1, 2, 3, 4, 5, 6] [1, 2, 3, 4, 5, 6] [1, 2, 3, 4, 5, 6, 1, 2, 3, 4, 5, 6, 1, 2, 3, 4, 5, 6] 1 1_2 [7, 8, 9, 10, 11, 12] [7, 8, 9, 10, 11, 12] [7, 8, 9, 10, 11, 12] [7, 8, 9, 10, 11, 12, 7, 8, 9, 10, 11, 12, 7, 8, 9, 10, 11, 12] 2 1_3 [13, 14, 15, 16, 17, 18] [13, 14, 15, 16, 17, 18] [13, 14, 15, 16, 17, 18] [13, 14, 15, 16, 17, 18, 13, 14, 15, 16, 17, 18, 13, 14, 15, 16, 17, 18]  Timing comparison with repeated addition (100 rows, 100 columns, 1000 items per list): # repeated addition of the lists 8.66 s ± 309 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) # nested list comprehension 729 ms ± 285 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
Question: I am working with Django REST framework and django-filters and and I'd like to use the reverse relationship annotation_set as one of filters for a GET API that uses the model Detection. The models are the following: class Detection(models.Model): image = models.ImageField(upload_to="detections/images") def local_image_path(self): return os.path.join('images' f"{self.id}.jpg") class Annotation(models.Model): detection = models.ForeignKey(Detection, on_delete=models.CASCADE) attribute = models.CharField(max_length=255)  The serializer is: class DetectionSerializer(UniqueFieldsMixin, serializers.ModelSerializer): local_image_path = serializers.CharField() class Meta: model = Detection fields = '__all__'  And the viewset is:  class DetectionTrainingViewSet( mixins.ListModelMixin, mixins.RetrieveModelMixin, viewsets.GenericViewSet ): queryset = Detection.objects.all() serializer_class = DetectionSerializer filterset_fields = ('annotation_set__id', ) @action(methods=['GET'], detail=False) def list_ids(self, request): queryset = self.get_queryset() filtered_queryset = self.filter_queryset(queryset) return Response(filtered_queryset.values_list('id', flat=True))  When I make a call to the endpoint, I get the error: 'Meta.fields' must not contain non-model field names: annotation_set__id  Shouldn't the field exist? Note: I tried to add other fields to the Annotation model and then use annotation_set__newfield but I still have the error. I can confirm that the newfield exists because it is correctly serialized and return by the API when I comment out the line that set the filterset_fields.

Answer: Apparently I had to explicitly state the name of the reverse relationship: class Annotation(models.Model): detection = models.ForeignKey(Detection, on_delete=models.CASCADE, related_name='annotation_set') attribute = models.CharField(max_length=255)  If anybody knows why, I'd love to know it! Thanks!
Question: When I run mypy checkings I am getting an error. I am no able to ignore it or turn it off the strict optional checking. It there a way to solve this. Here is the line that is throwing the error:  if tree.data.attributes.custom != JAPAN:  where attributes is declared as: class TreeAttributesModel(BaseModel): id: Optional[TreeId] name: Optional[str] = None status: StatusEnum custom: Optional[CustomAttrsModel] = None  and CustomAttrsModel is declared as it follows: class CustomAttrsModel(BaseModel): seller: Optional[str] buyed_at: Optional[datetime] country: Optional[Union[CountryEnum, str]]  Could you please help me with this?

Answer: I had to tweak your snippets a bit to get a MWE, but here we go: import enum import dataclasses from datetime import datetime from typing import Optional, Union class StatusEnum(enum.Enum): OK = enum.auto() NOK = enum.auto() class CountryEnum(enum.Enum): JAPAN = enum.auto() RAPTURE = enum.auto() @dataclasses.dataclass class TreeAttributesModel: id: Optional[str] name: Optional[str] # = None had to remove default, attribs w/o default cannot follow attribs w/ one status: StatusEnum custom: Optional[CustomAttrsModel] = None @dataclasses.dataclass class CustomAttrsModel: seller: Optional[str] buyed_at: Optional[datetime] country: Optional[Union[CountryEnum, str]] custom = CustomAttrsModel(seller="test", buyed_at=None, country=CountryEnum.JAPAN) attribs = TreeAttributesModel(id="test", name="test", status=StatusEnum.OK, custom=custom) assert attribs.custom is not None # this is typed as being optional, so make sure it isn't None assert attribs.custom.country is not None # same as above result = attribs.custom.country != CountryEnum.JAPAN  The message is: just use assert something is not None whenever something is Optional ;)
Question: I am trying to count all the files in a folder and all its subfolders For exemple, if my folder looks like this: file1.txt subfolder1/ ├── file2.txt ├── subfolder2/ │ ├── file3.txt │ ├── file4.txt │ └── subfolder3/ │ └── file5.txt └── file6.txt file7.txt  I would like get the number 7. The first thing I tried is a recursive function who count all files and calls itself for each folder def get_file_count(directory: str) -> int: count = 0 for filename in os.listdir(directory): file = (os.path.join(directory, filename)) if os.path.isfile(file): count += 1 elif os.path.isdir(file): count += get_file_count(file) return count  This way works but takes a lot of time for big directories. I also remembered this post, which shows a quick way to count the total size of a folder using win32com and I wondered if this librairy also offered a way to do what I was looking for. But after searching, I only found this fso = com.Dispatch("Scripting.FileSystemObject") folder = fso.GetFolder(".") size = folder.Files.Count  But this only returns the number of files in only the targeted folder (and not in its subfolders) So, do you know if there is an optimal function in python that returns the number of files in a folder and all its subfolders?

Answer: IIUC, you can just do sum(len(files) for _, _, files in os.walk('path/to/folder'))  or perhaps, to avoid the len for probably slightly better performance: sum(1 for _, _, files in os.walk('folder_test') for f in files)
Question: I was trying to accessing my key vault, but I got always the same error: AppServiceCredential.get_token failed: request() got an unexpected keyword argument 'tenant_id' ManagedIdentityCredential.get_token failed: request() got an unexpected keyword argument 'tenant_id'  This was the code I used in an Azure Machine Learning notebook, copied from the docs: from azure.identity import ManagedIdentityCredential from azure.keyvault.secrets import SecretClient credential = ManagedIdentityCredential() secret_client = SecretClient(vault_url="https://XXXX.vault.azure.net/", credential=credential) secretName = 'test' retrieved_secret = secret_client.get_secret(secretName) # here's the error retrieved_secret  What is wrong? Could you help me? Thank you in advance.

Answer: This error is because of a bug that has since been fixed in azure-identity's ManagedIdentityCredential. Key Vault clients in recent packages include a tenant ID in token requests to support cross-tenant authentication, but some azure-identity credentials didn't correctly handle this keyword argument until the bug was fixed in version 1.8.0. Installing azure-identity>=1.8.0 should fix the error you're getting. (Disclaimer: I work for the Azure SDK for Python)
Question: With even very simple example, backward() cannot work if sparse_grad=True, please see the error below. Is this error expected, or I'm using gather in a wrong way? In [1]: import torch as th In [2]: x = th.rand((3,3), requires_grad=True) # sparse_grad = False, the backward could work as expetecd In [3]: th.gather(x @ x, 1, th.LongTensor([[0], [1]]), sparse_grad=False).sum().backward() # sparse_grad = True, backward CANNOT work In [4]: th.gather(x @ x, 1, th.LongTensor([[0], [1]]), sparse_grad=True).sum().backward() --------------------------------------------------------------------------- RuntimeError Traceback (most recent call last) ----> 1 th.gather(x @ x, 1, th.LongTensor([[0], [1]]), sparse_grad=True).sum().backward() ~/miniconda3/lib/python3.9/site-packages/torch/_tensor.py in backward(self, gradient, retain_graph, create_graph, inputs) 305 create_graph=create_graph, 306 inputs=inputs) --> 307 torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs) 308 309 def register_hook(self, hook): ~/miniconda3/lib/python3.9/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs) 152 retain_graph = create_graph 153 --> 154 Variable._execution_engine.run_backward( 155 tensors, grad_tensors_, retain_graph, create_graph, inputs, 156 allow_unreachable=True, accumulate_grad=True) # allow_unreachable flag RuntimeError: sparse tensors do not have strides 

Answer: I think torch.gather does not support sparse operators: torch.gather(x, 1, torch.LongTensor([[0], [1]]).to_sparse())  Results with:  NotImplementedError: Could not run 'aten::gather.out' with arguments from the 'SparseCPU' backend.   I think you should open an issue or a feature request on pytorch's github.
Question: Problem: I can open VS code and start typing but after ~30s (sometimes minutes) the window freezes showing this message:  I am using python on a jupyter notebook with only a couple of unspectacular lines which I don't show because I tried different content. Last time it crashed after about 2 min of just import pandas as pd, df = pd.read_csv(path_to_file), print(df) I think it does not crash when I don't start typing but just wait, at least not as fast Yesterday everything was still fine. Today with the same setup not. System information: Windows 11 home The only VS extensions installed are Python v2022.7.11331006 Pylance v2022.5.1 Jupyter v2022.5.1001351004 Jupyter Keymap v1.0.0 Jupyter Notebook Renderers Which other information is needed to help? Attempts (not resolving the problem):  Work from a different folder with a new file/code Disabling extensions (but notebook will not work without some) Uninstall/reinstall same version (x64-1.67.1) Uninstall/install previous version (1.66.1) Unistall/install insider version (x64-1.68.0-insider) Disable all online connecting services in VS, disconnecting from internet. Rollback of last windows patch update Delete backup folders in %AppData%/Code/.. Graphic card is up-to-date I am reluctant to disable stuff that makes the dell xps faster (and expensive) 

Answer: This is caused by the latest Pylance extension. You can revert to earlier Pylance extension to solve the problem. This problem will be fixed in a new version in a few days according to the github.