eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: When executing the following code: from IPython.core import ultratb sys.excepthook = ultratb.FormattedTB(mode='Verbose', color_scheme='Linux', call_pdb=1)  In order to catch exceptions, I receive the following error: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/dinari/miniconda3/envs/testenv/lib/python3.8/site-packages/IPython/core/ultratb.py", line 994, in __init__ VerboseTB.__init__(self, color_scheme=color_scheme, call_pdb=call_pdb, File "/Users/dinari/miniconda3/envs/testenv/lib/python3.8/site-packages/IPython/core/ultratb.py", line 638, in __init__ TBTools.__init__( File "/Users/dinari/miniconda3/envs/testenv/lib/python3.8/site-packages/IPython/core/ultratb.py", line 242, in __init__ self.pdb = debugger_cls() TypeError: 'NoneType' object is not callable  Using python 3.8.2 and IPython 8.4.0 pdb otherwise is working fine. Any idea for a fix for this?

Answer: Downgrading IPython to 7.34.0 solved this.
Question: I have a listWidget that contains multiple QListWidgetItem and for simplicity's sake, let's assume that each QListWidgetItem consists of a QWidget with a QPushButton called delete. This is assembled with this code: class widgetItem(QtWidgets.QWidget): def __init__(self, parent): super(widgetItem, self).__init__() uic.loadUi('UIfiles/trainingWidget.ui', self) # Load the .ui file self.listWidgetItem = QtWidgets.QListWidgetItem() self.listWidgetItem.setSizeHint(self.sizeHint()) self.delete.clicked.connect(self.deleteSelf) parent.listWidget.addItem(self.listWidgetItem) parent.listWidget.setItemWidget(self.listWidgetItem, self)  and this is called in the main app with this: def additem(self): self.referenceDict[self.itemID] = widgetItem(self)  Now, my goal is to delete this particular widget from both the referenceDict as well as the listWidget when the button is pressed. Each widgetItem also has their own itemID as a string, and can access the listWidget as well as the referenceDict. How do I write this deleteSelf? I've tried using self.deleteLater but it seems like it only deletes the QWidget but not the QListWidgetItem. Calling self.listWidgetItem.deleteLater raises an attribute error as it cannot be deleted this way. I also tried parent.referenceDict.pop(self.itemID) but for some reason it raises a key error despite both the keys matching when I printed the dict out.

Answer: You can remove listitems using the takeItem method and delete widgets using the deleteLater method. I wasn't to fond of your chosen process for creating the widgets and adding them to the list, so I went ahead and created a minimal example but using QPushButton instead of QWidgets for the itemWidgets. Example: from PyQt5.QtWidgets import * from PyQt5.QtCore import * from PyQt5.QtGui import * class Window(QWidget): def __init__(self, parent=None): super().__init__(parent=parent) self.layout = QVBoxLayout() self.setLayout(self.layout) self.listWidget = QListWidget(self) self.layout.addWidget(self.listWidget) self.addListItems() def addListItems(self): # creates the item widgets for _ in range(5): item = QListWidgetItem(type=0) widget = ListItem("button", self, item) self.listWidget.addItem(item) self.listWidget.setItemWidget(item, widget) def removeWidgetItem(self, item): # removes the item widgets index = self.listWidget.indexFromItem(item).row() item = self.listWidget.takeItem(index) class ListItem(QPushButton): def __init__(self, text, parent, item): super().__init__(text, parent) self.item = item # the ListWidgetItem self._parent = parent # the Window self.clicked.connect(self.deleteSelf) def deleteSelf(self): # slot for button click self._parent.removeWidgetItem(self.item) self.deleteLater() app = QApplication([]) window = Window() window.show() app.exec()
Question: The scenario: I have a locally cloned github repo with multiple branches. Each branch needs potentially different dependencies. The question: I would like to switch between these branches as I work and therefore would like multiple pipenv virtual environments (one per branch). How can I achieve this, given that pipenv by default associates a single virtual environment with the project root folder?

Answer: Check out each branch into a separate directory (e.g., using git worktree). Because each branch would have a separate directory, pipenv would work without any additional changes. Assuming that you're currently in your work tree (let's say on the main branch), and you have additional branches named branch1 and branch2, that might look like: $ git worktree add ../branch1 branch1 $ git worktree add ../branch2 branch2  Now you main checked out in your current directory, branch1 checked out in ../branch1, and branch2 checked out in ../branch2. You can cd between these directories and work on them like normal, and pipenv will do what you want since each branch is now in a separate directory.
Question: I have a pytest project and a want to run tests from TWO python files. The project structure looks like this: at the root of the project there is a "tests" folder, it contains several folders "test_api1", "test_api2", "test_api3", each of them contains conftest.py and a test file. tests:  test_api1: conftest.py, test_api_1 test_api2: conftest.py, test_api_2 test_api3: conftest.py, test_api_3  Usually I run tests like this python -m pytest -vs -k tests (if I want to run all tests from tests directory) or like this python -m pytest -vs -k test_api1.py (if I want to run a certain test). But now I want to run tests from TWO certain files test_api1.py and test_api1.py. How can I do that?

Answer: Just add files that you want to run as positional arguments, e.g.: python -m pytest tests/test_api1.py tests/test_api2.py  NOTE: you need to run pytest without -k flag and hence use paths to the files instead of just filenames.
Question: I get that in Python OpenCV images are numpy arrays, that correspond to cv::Mat in c++. This question is about what type-hint to put into python functions to properly restrict for OpenCV images (maybe even for a specific kind of OpenCV image). What I do now is: import numpy as np import cv2 Mat = np.ndarray def my_fun(image: Mat): cv2.imshow('display', image) cv2.waitKey()  Is there any better way to add typing information for OpenCV images in python?

Answer: UPD: As mentioned in another answer, now, OpenCV has cv2.typing.MatLike. Then, the code would be: import cv2 def my_fun(img: cv2.typing.MatLike) -> None: pass  You can specify it as numpy.typing.NDArray with an entry type. For example, import numpy as np Mat = np.typing.NDArray[np.uint8] def my_fun(img: Mat): pass
Question: Conda can create an environment.yml that specifies both conda packages & pip packages. The problem is, I want to specify a pip package (torch==1.12.1+cu116), that is only available in the following index: https://download.pytorch.org/whl/cu116. How can I specify this in the environment.yml? Or at the very least, when running conda env create -f environment.yml, I would like to specify the extra index for pip.

Answer: This configuration should work, see the advanced-pip-example for other options. name: foo channels: - defaults dependencies: - python - pip - pip: - --extra-index-url https://download.pytorch.org/whl/cu116 - torch==1.12.1+cu116  See also  Combining conda environment.yml with pip requirements.txt Can conda be configured to use a private pypi repo?
Question: How to remove the legend in the seaborn.JoingGrid plot? The reference code is like below: import matplotlib.pyplot as plt import seaborn as sns penguins = sns.load_dataset("penguins") g = sns.JointGrid(data=penguins, x="bill_length_mm", y="bill_depth_mm", hue="species") g.plot_joint(sns.scatterplot) sns.boxplot(data=penguins, x=g.hue, y=g.y, ax=g.ax_marg_y) sns.boxplot(data=penguins, y=g.hue, x=g.x, ax=g.ax_marg_x) plt.show()   I have tried to use the following methods that are known to work on the other seaborn plots, but failed on the jointplot: plt.legend([],[], frameon=False) g._legend.remove() 

Answer:  To remove the legend, the correct part of the sns.JointGrid must be accessed.  In this case g.ax_joint is where the legend is located.   As stated in a comment by mwaskom, Matplotlib axes have .legend (a method that creates a legend) and .legend_ (the resulting object). Don't access variables that start with an underscore (_legend), because it indicates they are private. Tested in python 3.10, matplotlib 3.5.1, seaborn 0.11.2  g.plot_joint(sns.scatterplot, legend=False) may be used instead.    sns.JointGrid import seaborn as sns penguins = sns.load_dataset("penguins") g = sns.JointGrid(data=penguins, x="bill_length_mm", y="bill_depth_mm", hue="species") g.plot_joint(sns.scatterplot) # legend=False can be added instead sns.boxplot(data=penguins, x=g.hue, y=g.y, ax=g.ax_marg_y) sns.boxplot(data=penguins, y=g.hue, x=g.x, ax=g.ax_marg_x) # remove the legend from ax_joint g.ax_joint.legend_.remove()    Moving the JointGrid legend can be done with sns.move_legend, as shown in this answer.  This also requires using g.ax_joint.    penguins = sns.load_dataset("penguins") g = sns.JointGrid(data=penguins, x="bill_length_mm", y="bill_depth_mm", hue="species") g.plot_joint(sns.scatterplot) sns.boxplot(data=penguins, x=g.hue, y=g.y, ax=g.ax_marg_y) sns.boxplot(data=penguins, y=g.hue, x=g.x, ax=g.ax_marg_x) # move the legend in ax_joint sns.move_legend(g.ax_joint, "lower right", title='Species', frameon=False)    With sns.jointplot  g.ax_joint.legend_.remove() can be used, but removing the legend is more easily accomplished by passing legend=False to the plot: sns.jointplot(..., legend=False). g.ax_joint. is still required to move the legend.  penguins = sns.load_dataset("penguins") g = sns.jointplot(data=penguins, x="bill_length_mm", y="bill_depth_mm", hue="species") # legend=False can be added instead # remove the legend g.ax_joint.legend_.remove() # or # move the legend # sns.move_legend(g.ax_joint, "lower right", title='Species', frameon=False)
Question: Every time I boot up terminal on VSCode, I get the following prompt. This does not happen on Terminal.app.  /usr/local/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.  How do I resolve this?

Answer: Install the setuptools 58.2.0 version using the following command: pip install setuptools==58.2.0
Question: I have a json key with whitespace in it: My_dict = {"my key": 1}  I want to create a Model to model it: from pydantic import BaseModel class MyModel(BaseModel): mykey: int # my key isn't a legit variable name # my_key is, but like mykey - it doesn't catch the correct key from the json MyModel(**my_dict)  This doesn't work. I tried playing with the BaseModel.Config, but didn't get anywhere. Didn't see anything on the docs as well. Is this possible? I can use a workaround: Go over the json, replace all key's whitespaces into underscores, and then use pydantic but I would love to not use this...

Answer: Yes, it's possible by using Field's aliases: from pydantic import BaseModel, Field class MyModel(BaseModel): mykey: int = Field(alias='my key') class Config: allow_population_by_field_name = True print(MyModel(**{"my key": 1})) print(MyModel(**{"mykey": 1}))  [Edit @Omer Iftikhar] For Pydantic V2: You need to use populate_by_name instead of allow_population_by_field_name otherwise you will get following warning UserWarning: Valid config keys have changed in V2: * 'allow_population_by_field_name' has been renamed to 'populate_by_name'
Question: I'm learning how to use Protocol classes that have been introduced in Python 3.8 (PEP 544). So typing.Protocol classes are subclasses from ABCMeta and they are treated just like abstract classes are with the added benefit of allowing to use structural subtyping. I was trying to think of what I would use abstract base classes now and I'm drawing a blank. What are the downsides of Protocol classes compared to ABCs (if any)? Maybe they come with a performance hit? Are there any specific cases where an ABC is still the best choice?

Answer: I prefer ABCs beacuse they're explicit. With a Protocol someone reading the code may not know your class is intended to implement an interface in another module or deep in a dependency. Similarly, you can accidentally conform to a Protocol's signature, without conforming to its contract. For example, if a function accepts a class Image(Protocol): def draw() -> None: ...  it's obviously not going to make sense with a class Cowboy: def draw() -> None: ...  but the type checker would happily accept it.
Question: I want to get data which inserted in the last so I used a django code user = CustomUser.objects.filter(email=email).last() So it gives me the last user detail. But then experimentally I used: user = CustomUser.objects.filter(email=email).latest() Then It didn't give me a user object. Now, what is the difference between earliest(), latest, first and last()?

Answer: There are several differences between .first() [Django-doc]/.last() [Django-doc] and .earliest(…) [Django-doc]/.latest(…) [Django-doc]. The main ones are:  .first() and .last() do not take field names (or orderable expressions) to order by, they do not have parameters, .earliest(…) and .latest(…) do; .first() and .last() will work with the ordering of the queryset if there is one, .earliest(…) and .latest(…) will omit any .order_by(…) clause that has already been used; if the queryset is not ordered .first() and .last() will order by the primary key and return the first/last item of that queryset, .earliest(…) and .latest(…) will look for the get_latest_by model option [Django-doc] if no fields are specified; and .first() and .last() will return None in case the queryset is empty; whereas .earliest(…) and .latest(…) will raise a DoesNotExist exception [Django-doc].
Question: So I've a Flask web app that will be exposing some deep learning models. I built the image and everything works fine. the problem is the size of this image is 5.58GB! which is a bit ridiculous. I have some deep learning models that are copied during the build, I thought they might be the culprit but their size combined does not exceed 300MB so that's definately not it. upon checking the history and the size of each layer I discovered this:  RUN /bin/sh -c pip install -r requirements.txt is taking up 771MB. RUN /bin/sh -c pip install torch==1.10.2 is taking up 2.8GB! RUN /bin/sh -c apt-get install ffmpeg libsm6 libxext6 is taking up 400MB.  so how do I incorporate these libraries while keeping image size reasonable? is it ok to have images of these size when deploying ml models in python? below is the root directory:  Dockerfile: FROM python:3.7.13 WORKDIR /app COPY ["rdm.pt", "autosort_model.pt", "rotated_model.pt", "yolov5x6.pt", "/app/"] RUN pip install torch==1.10.2 COPY requirements.txt /app/requirements.txt RUN pip install -r requirements.txt RUN apt-get update RUN apt-get install ffmpeg libsm6 libxext6 -y COPY . /app CMD python ./app.py  .dockerignore: Dockerfile README.md __pycache__ 

Answer: By default torch will package CUDA packages and stuff. Add --extra-index-url https://download.pytorch.org/whl/cpu and --no-cache-dir to pip install command if you do not require CUDA. RUN pip install --no-cache-dir -r requirements.txt --extra-index-url https://download.pytorch.org/whl/cpu Also it's good practice to remove the apt list cache: RUN apt-get update \ && apt-get install -y \ ffmpeg \ libsm6 \ libxext6 \ && rm -rf /var/lib/apt/lists/*
Question: I am trying to call a a superclass method from a dataclass with slots=True in Python 3.10.5. from dataclasses import dataclass @dataclass(slots=True) class Base: def hi(self): print("Hi") @dataclass(slots=True) class Sub(Base): def hi(self): super().hi() Sub().hi()  I get the following error. Traceback (most recent call last): File "...", line 16, in <module> Sub().hi() File "...", line 13, in hi super().hi() TypeError: super(type, obj): obj must be an instance or subtype of type  It works fine if I remove slots=True from Sub, or make it a non-dataclass with __slots__ manually. The error remains if I instead do these to Base. Sub.__mro__ is (<class '__main__.Sub'>, <class '__main__.Base'>, <class 'object'>) and isinstance(Sub(), Base) is True.

Answer: As seen here, the dataclass decorator creates a new class object, and so the __closure__ attached to hi() is different to the one attached to the decorated class, and therefore the super() call cannot work without arguments due to relying on the __closure__. Therefore, you need to change super().hi() to super(Sub, self).hi().
Question: My server has two GPUs, How can I use two GPUs for training at the same time to maximize their computing power? Is my code below correct? Does it allow my model to be properly trained? class MyModel(nn.Module): def __init__(self): super(MyModel, self).__init__() self.bert = pretrained_model # for param in self.bert.parameters(): # param.requires_grad = True self.linear = nn.Linear(2048, 4) #def forward(self, input_ids, token_type_ids, attention_mask): def forward(self, input_ids, attention_mask): batch = input_ids.size(0) #output = self.bert(input_ids, token_type_ids, attention_mask).pooler_output output = self.bert(input_ids, attention_mask).last_hidden_state print('last_hidden_state',output.shape) # torch.Size([1, 768]) #output = output.view(batch, -1) # output = output[:,-1,:]#(batch_size, hidden_size*2)(batch_size,1024) output = self.linear(output) return output device = torch.device("cuda" if torch.cuda.is_available() else "cpu") if torch.cuda.device_count() > 1: print("Use", torch.cuda.device_count(), 'gpus') model = MyModel() model = nn.DataParallel(model) model = model.to(device) 

Answer: There are two different ways to train on multiple GPUs:  Data Parallelism = splitting a large batch that can't fit into a single GPU memory into multiple GPUs, so every GPU will process a small batch that can fit into its GPU Model Parallelism = splitting the layers within the model into different devices is a bit tricky to manage and deal with.  Please refer to this post for more information To do Data Parallelism in pure PyTorch, please refer to this example that I created a while back to the latest changes of PyTorch (as of today, 1.12). To utilize other libraries to do multi-GPU training without engineering many things, I would suggest using PyTorch Lightning as it has a straightforward API and good documentation to learn how to do multi-GPU training using Data Parallelism.  Update: 2022/10/25 Here is a video explaining in much details about different types of distributed training: https://youtu.be/BPYOsDCZbno?t=1011
Question: I use VS Code for a python project but recently whenever I launch the debugger it immediately exits. The debug UI will pop up for half a second then disappear. I can't hit a breakpoint no matter where it's placed in the current file. The project has the expected normal behavior when run in non-debug mode. I vaguely remember a command being inserted into the terminal window when I used to click debug but now I see nothing. I opened a totally different project but debugger still exits immediately. Any advice? Anywhere I can find logs for the debugger run? My launch.json file: { "version": "0.2.0", "configurations": [ { "name": "Python: Current File", "type": "python", "request": "launch", "program": "${file}", "console": "integratedTerminal", "justMyCode": true } ] }  I have tried: running app as admin, reinstalling vs code, reinstalling python extension, restarting app, restarting computer, disabling all non-essential extensions, deleting launch.json, launching a file with only print statement.

Answer: Please install Python 3.7 or later. If you must use Python 3.6 or earlier, rollback the Python extension to version 2022.08.0.
Question: I'm running vscode release (1.69.2 dated 7/18/22) with the python extension on Ubuntu 20.04. When I try to run some code with the debugger I get an exception because it can't find my_venv_dir/lib/python3.8/dist-packages. I've read a bit about Ubuntu's use of dist-packages and site-packages but I haven't found any information that suggests that there should be a dist-packages directory in my venv. There isn't one. When I run the code without the debugger I don't see the exception. When I run it in the debugger, the exception occurs before I hit a very early breakpoint. The exception seems to happen in startup code. So I don't know if there's a venv issue or a vscode/python-extension issue. I created the virtual env using python -m venv Here's the call stack: islink (/usr/lib/python3.8/posixpath.py:167) _joinrealpath (/usr/lib/python3.8/posixpath.py:425) realpath (/usr/lib/python3.8/posixpath.py:391) _run_code (/usr/lib/python3.8/runpy.py:87) _run_module_as_main (/usr/lib/python3.8/runpy.py:194)  Maybe this is more useful: Exception has occurred: FileNotFoundError [Errno 2] No such file or directory: '/home/my/venv/dir/lib/python3.8/dist-packages' File "/usr/lib/python3.8/posixpath.py", line 167, in islink st = os.lstat(path) File "/usr/lib/python3.8/posixpath.py", line 425, in _joinrealpath if not islink(newpath): File "/usr/lib/python3.8/posixpath.py", line 391, in realpath path, ok = _joinrealpath(filename[:0], filename, {}) File "/usr/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None,  launch.json: { // Use IntelliSense to learn about possible attributes. // Hover to view descriptions of existing attributes. // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387 "version": "0.2.0", "configurations": [ { "name": "Python: Current File", "type": "python", "request": "launch", "program": "${file}", "console": "integratedTerminal", "justMyCode": false } ] }  This is the sys.path: (myvenv) host:~/dev/myprj$ python Python 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> print(sys.path) ['', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/my/venv/dir/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] >>> 

Answer: It looks like something was corrupted in the vscode configuration. I tried uninstalling and reinstalling vscode but noticed that when I started up again, vscode remembered where I left off. When I deleted the ~/.config/Code directory the problem was fixed. I suspect that the corruption happened when I updated to a newer release but I can't be sure.