akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How does STE realize in quantification aware training?

Rahul_Xie: <p>How does STE realize in quantification aware training?</p>

etrommer: <p>You could pass the tensor that you want to quantize through a <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.Function" rel="noopener nofollow ugc">custom torch.autograd.Function</a><br>
Here is an example where this is done for the DoReFa quantization: <a href="https://github.com/KwangHoonAn/PACT/blob/025370908c347ed7b461ccf1db6860a81250f72c/module.py#L41-L54" class="inline-onebox" rel="noopener nofollow ugc">PACT/module.py at 025370908c347ed7b461ccf1db6860a81250f72c · KwangHoonAn/PACT · GitHub</a></p>
<p>In the forward pass, implement whichever quantization scheme you are interested in. Note that the backward pass is made explicit and simply returns the upstream gradient as the gradient for the operation’s input tensor.</p>

Rahul_Xie: <p>Thanks for your reply.</p>
Lead MLE role at Spectrum Labs

Nina_Lopatina: <p>We are looking to add a new Lead Machine Learning Engineer to our thriving remote-first team at Spectrum Labs. The role blends state-of-the-art applied NLP with scalable software systems to make the internet a safer place. If you are looking for a challenging opportunity to use science to drive a meaningful impact and have a strong deep learning background, we would love to talk to you!</p>
<p>Please apply below if you are interested.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://app.trinethire.com/companies/38694-spectrum-labs/jobs/63162-lead-machine-learning-engineer">
  <header class="source">
      <img src="https://dg4guz6k8vg2i.cloudfront.net/assets/transparent-icon-1x1-63ef318d96b5d0d0ceba6e04a4e622b1158335cdc67c49e27839132c6f655058.png" class="site-icon" width="1" height="1">

      <a href="https://app.trinethire.com/companies/38694-spectrum-labs/jobs/63162-lead-machine-learning-engineer" target="_blank" rel="noopener nofollow ugc">TriNet Applicant Tracking</a>
  </header>

  <article class="onebox-body">
    <img src="https://dg4guz6k8vg2i.cloudfront.net/uploads/company/logo/38694/thumb_spectrum-labs-1641607086.png" class="thumbnail onebox-avatar" width="150" height="150">

<h3><a href="https://app.trinethire.com/companies/38694-spectrum-labs/jobs/63162-lead-machine-learning-engineer" target="_blank" rel="noopener nofollow ugc">Spectrum Labs | Lead Machine Learning Engineer</a></h3>

  <p>Overview

Are you well-versed in and excited by the unique challenges of
applying NLP and NLU to content moderation? Do you regularly
leverage deep learning frameworks such as Tensorflow, PyTorch,
and ONNX to build and maintain reliable, fast,...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Missing COCO Annotations

Bassie_Witkin: <p>I am using a pretrained Image segmentation on a custom COCO Dataset. I used the COCO Detection class:</p>
<pre><code class="lang-auto">class CocoDetection():
    """`MS Coco Detection &lt;http://mscoco.org/dataset/#detections-challenge2016&gt;`_ Dataset.

    Args:
        root (string): Root directory where images are downloaded to.
        annFile (string): Path to json annotation file.
        transform (callable, optional): A function/transform that  takes in an PIL image
            and returns a transformed version. E.g, ``transforms.ToTensor``
        target_transform (callable, optional): A function/transform that takes in the
            target and transforms it.
    """

    def __init__(self, root, annFile, transform=None, target_transform=None):
        from pycocotools.coco import COCO
        self.root = root
        self.coco = COCO(annFile)
        self.ids = list(self.coco.imgs.keys())
        self.transform = transform
        self.target_transform = target_transform

    def __getitem__(self, index):
        """
        Args:
            index (int): Index

        Returns:
            tuple: Tuple (image, target). target is the object returned by ``coco.loadAnns``.
        """
        coco = self.coco
        img_id = self.ids[index]
        ann_ids = coco.getAnnIds(imgIds=img_id)
        target = coco.loadAnns(ann_ids)
        for x in target:
            target=x
        path = coco.loadImgs(img_id)[0]['file_name']

        img = Image.open(os.path.join(self.root, path)).convert('RGB')
        if self.transform is not None:
            img = self.transform(img)

        if self.target_transform is not None:
            target = self.target_transform(target)

        return img, target

</code></pre>
<p>I realized that I am missing annotations because there are a lot of images that have more than one annotation per image. Is there another function or class that takes this issue into account or is this something that I have to work out manually?</p>
Could it possible that mps is made as easy as cuda?

AkechiKogoro: <p>There are a couple of things I cannot do on my Mac M1 machine now.</p>
<p>For example, I cannot check how many GPUs I have in order to do parallel training. I have many GPUs and I want to make full use of them.</p>
<p>Also, if I have a tensor x, I can easily write “x.cuda()” to move x to GPU on a machine with NVIDIA, however this is not possible with mps. We have to do x.to(torch.device(‘mps’)).</p>
<p>There are a lot more functions that CUDA has but mps does not. I wish they could be added in later updates.</p>
<p>Thank the developers.</p>

albanD: <p>Hi,</p>
<blockquote>
<p>For example, I cannot check how many GPUs I have in order to do parallel training. I have many GPUs and I want to make full use of them.</p>
</blockquote>
<p>Could you give details on how you have many GPUs? M1 machines are only single chip.</p>
<blockquote>
<p>Also, if I have a tensor x, I can easily write “x.cuda()” to move x to GPU on a machine with NVIDIA, however this is not possible with mps. We have to do x.to(torch.device(‘mps’)).</p>
</blockquote>
<p>This is an oversight on our end I’m afraid. Could you open an issue on github so that we can track adding this method?</p>
<blockquote>
<p>There are a lot more functions that CUDA has but mps does not. I wish they could be added in later updates.</p>
</blockquote>
<p>Yes, PyTorch has 2000+ ops. So it is quite a bit of work to support them all. We’re working on this!</p>
Why is jit::script needed at all in libtorch?

botelho: <p>Hi there,</p>
<p>My goal is simply to load a <code>.pt</code> model saved in Python into a <code>nn::Module</code> in libtorch. Is this impossible to do programmatically?</p>
<p>I work on a libtorch-based application in which I can load HDF5 models from Keras. Since the HDF5 contains both the network graph (as a json string) and the weights, I can programmatically construct the <code>nn::Module</code> and populate its trainable parameters. All without going through any format conversion! And, ironically, I cannot do the same with a model saved from <strong>Pytorch itself</strong> in python?</p>
<p>Am I correct in concluding that I must use <code>jit::script::Module</code> instead for that task, and that there’s no conversion between that and <code>nn::Module</code>? Can any Pytorch dev explain why not go the much easier route used by Keras to save/load models, toss <code>jit::script</code> altogether and keep only <code>nn</code>?</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> , <a class="mention" href="/u/glaringlee">@glaringlee</a> , <a class="mention" href="/u/tom">@tom</a> , <a class="mention" href="/u/yf225">@yf225</a> : please help out here! I’ve struggled with this issue for months, and any pointer will be greatly appreciated. Thanks!</p>

ptrblck: <p>Please don’t tag specific users as it can be demotivating for others to answer your question and you might also tag users unfamiliar with your use case.<br>
Also, this seems to be a double post from <a href="https://discuss.pytorch.org/t/loading-python-model-into-nn-module-in-libtorch/154159">here</a>, where other users are already responding, or what is the difference between these topics?</p>

botelho: <p>Thank you, <a class="mention" href="/u/ptrblck">@ptrblck</a>  Won’t tag specific users in the future. The reason I did is because I’m interested specifically in feedback from Pytorch developers regarding this design decision, and ways to work around it. In particular, the question is:</p>
<p>Is it impossible to load a python model directly into <code>nn::Module</code>? If so, why did you guys decide to go this way, and develop an orthogonal API for it (<code>jit::script</code>)?</p>
<p>Thank you</p>
Find Jaccard similarity between 2 vectors of different lengths

saswat_72444: <p>I have two tensors (1D array of integers) of unequal lengths and I want to calculate the jaccard similarity between them in PyTorch.</p>
<p>My current implementation is valid for same-length arrays only</p>
<pre><code class="lang-auto">def js_pt(list1, list2):
    intersection = (list1 * list2).abs().sum(dim=-1)
    sum_ = torch.sum(list1.abs() + list2.abs(), dim=-1)
    jac = (intersection) / (sum_ - intersection)
    return (1 - jac)
</code></pre>
<p>How can I modify it for tensors with unequal lengths?</p>
Feed forward entire dataset on model

rjsdebug: <p>For my project I want to run some experiments after certain epochs have finished training. For example, after the 3rd epoch I want to get the logit values of a network pre-softmax, so I need to run my entire dataset (MNIST in this case) through my model, just forward. This seems it lends itself to parallelization, is there a way to run the entire model on my data array without a for loop?</p>

Karthik_Ganesan: <p>Does you model have a softmax at the end? If not, you should get logit values from the final FC layer already. If you do have a softmax, you can add a <a href="https://web.stanford.edu/~nanbhas/blog/forward-hooks-pytorch/#forward-hooks-101" rel="noopener nofollow ugc">forward hook</a> after the final FC layer to get the activations from that before the softmax.</p>
<p>You should be able to get the outputs from your model without a for loop by just providing all the inputs you want the model to classify.</p>
<p><code> outputs = model(inputs)</code></p>
<p>You can then use <code>outputs</code> for whatever you want.</p>

rjsdebug: <p>Thank you. When I try to pass my dataset into model() it says I cannot pass in an MNIST object (it must be a tensor). Do I have to make a separate dataloader for this, or is there a quick way to make my dataset into a tensor?</p>

Karthik_Ganesan: <p>Yes you have to use a dataloader whenever you want to provide input to your model. Your dataloader is setup with a given batch size so each time you will get a tensor with that many inputs and labels.</p>
<pre><code class="lang-auto">for images, labels in testloader:
    outputs = model(images)
</code></pre>
Unbalanced audio classification

Gorgen: <p>Hello everyone. In the audio classification task, the data is seriously unbalanced, and there is a similar long-tail phenomenon, resulting in very few samples that cannot be identified. how to solve this problem? thank you.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/0/4028818cc648ad562a47f0bc9fa4de97c8ac0353.png" alt="image" data-base62-sha1="99zlcns5PnoBGF6rYmZ085EgCyv" width="452" height="472"></p>
Looking for literature references of autoencoder architectures

Hadi_Nayebi: <p>I am writing a paper on a model I build for analyzing a dataset. The model is an autoencoder with an additional unit attached to the embedding layer. Similar to an adversarial autoencoder, except, in this case, the new unit can train for any prediction; binary or multi-class classification, another decoder, regression model, LSTM, and so on. The embeddings feed into the new unit or units and train in a supervised style.</p>
<p>The training goes through multiple passes; based on the available input and output, it can feedforward in various paths. For example, if I use it with a classifier attached, the classifier weights will update if the labels are available. Otherwise, it will only update the encoder and decoder.</p>
<p>It’s doing a good job on my dataset. I have looked in the literature to find similar architectures and reference them in mine, but the most similar case I found is the adversarial autoencoder. I would appreciate it if anyone gave me feedback on this and let me know if they have read any papers with similar architecture.</p>
<p>Thanks.</p>
Try to assign parameters to the model, but

noName_111: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/4/c4b0f3de7e47e16a58d7d6928c89f59e4989a129.png" alt="image" data-base62-sha1="s40HC07KGO6qULDIXYndbQp1NgZ" width="519" height="68"><br>
I try to assign the parameters to the model, and there are two options, which line I should use? The first line or the second line, and what is the difference between them? Also, If I use the first line, it will cause my DDP stuck at the second epoch, is there any solution If I have to use the first line?</p>

initmahesh: <p><a class="mention" href="/u/h-huang">@H-Huang</a> to triage further</p>

H-Huang: <p>Assigning/updating parameters (line 1) to the model during DDP will cause a hang as the original parameter has been already assigned to the Model parameters() and updating this removes the reference to it and the gradients.</p>
<p>(line 2) data is an internal field in tensors that just changes the value of it. It does not update the Model parameters() and will not cause a hang.</p>
<p>Is there any reason you are updating the parameters manually like this during training? It is not very conventional. If you need to load previous state you should look into torch.load/torch.save (<a href="https://pytorch.org/docs/stable/generated/torch.load.html?highlight=load#torch.load" class="inline-onebox" rel="noopener nofollow ugc">torch.load — PyTorch 1.11.0 documentation</a>)</p>
Cannot install torch 1.7.0 on WSL2/Debian

jlgarnier: <p>Hi all!</p>
<p>I’m using WSL2 Debian distro to install and compile (<strong>not run</strong>) a software which requires torch 1.7.0. Python is v3.9.2, there’s no GPU. Unfortunately I’m unable to install the lib.</p>
<p>Here’s what I’ve tried so far:</p>
<ul>
<li>
<code>python3 -m pip install torch==1.7.0 -f https://download.pytorch.org/whl/torch_stable.html</code> → ERROR: No matching distribution found for torch==1.7.0</li>
<li>
<code>pip3 install https://download.pytorch.org/whl/cpu/torch-1.7.0%2Bcpu-cp38-cp38-linux_x86_64.whl</code> → ERROR: torch-1.7.0+cpu-cp38-cp38-linux_x86_64.whl is not a supported wheel on this platform.</li>
<li>
<code>pip install torch==1.7.0+cpu torchvision==0.8.0+cpu torchaudio==0.7.0 -f https://download.pytorch.org/whl/torch_stable.html</code> → ERROR: No matching distribution found for torch==1.7.0+cpu</li>
</ul>
<p>I’m a bit lost: can anyone point me to the proper command?<br>
Reminder: I just want to install and compile the app, then copy it to a Nvidia Jetson Nano target to check whether it runs or not…</p>
<p>Thanks in advance for any help!</p>
<p>---- Environment<br>
PyTorch version: N/A<br>
Is debug build: N/A<br>
CUDA used to build PyTorch: N/A<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: Debian GNU/Linux 11 (bullseye) (x86_64)<br>
GCC version: (Debian 10.2.1-6) 10.2.1 20210110<br>
Clang version: Could not collect<br>
CMake version: version 3.18.4<br>
Libc version: glibc-2.31</p>
<p>Python version: 3.9.2 (default, Feb 28 2021, 17:03:44)  [GCC 10.2.1 20210110] (64-bit runtime)<br>
Python platform: Linux-4.4.0-19041-Microsoft-x86_64-with-glibc2.31<br>
Is CUDA available: N/A<br>
CUDA runtime version: Could not collect<br>
GPU models and configuration: Could not collect<br>
Nvidia driver version: Could not collect<br>
cuDNN version: Could not collect<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A<br>
Is XNNPACK available: N/A</p>
<p>Versions of relevant libraries:<br>
[pip3] numpy==1.22.4<br>
[conda] Could not collect</p>
Program hangs after NCCL process group is destroyed and restarted

fot: <p>Hi all!</p>
<p>I train a model using DistributedDataParallel with the NCCL backend. At a specific point of the training, I want to reinitialize the process group, therefore I do:</p>
<ul>
<li>dist.destroy_process_group</li>
<li>del self.model (the DDP model)</li>
<li>del self.optimizer</li>
<li>self.store = dist.TCPStore(host_name=master_ip, port=int(master_port), is_master=False) (the master is properly setup)</li>
<li>dist.init_process_group(“nccl”, world_size=world_size, rank=rank, store=self.store, group_name=gname, timeout=timedelta(seconds=10))</li>
<li>self.model = … (whatever model i have here)</li>
<li>self.model = DDP(model, device_ids=[self.local_rank])</li>
</ul>
<p>Although the process group seems to be initialized (the store barrier is passed), the program always hangs but not at a specific  point.</p>
<p>Could this be an issue with non-aborted NCCL communicators? Does anyone know how to solve it?</p>
<p>Thank you</p>

eqy: <p>Could you post a script that reproduces this issue? Is it happening on with multiple devices on a single node, multiple nodes, or both?</p>
<p>Additionally, have you checked the output when running with e.g., <code>NCCL_DEBUG=INFO</code> to see if that uncovers a specific failure? <a href="https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html" class="inline-onebox" rel="noopener nofollow ugc">Environment Variables — NCCL 2.12.12 documentation</a></p>

fot: <p>Hi,</p>
<p>Thank you for your message!<br>
The problem was due to some NCCL communicators that were not aborted. This caused subsequent calls to <a href="https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__STREAM.html#group__CUDART__STREAM_1g82b5784f674c17c6df64affe618bf45e" rel="noopener nofollow ugc">cudaStreamSynchronize</a> to block. I solved it by explicitly removing all the “old” communicators.</p>
Overfitting Under Class Imbalance

ALMOUDI: <p>Hello,<br>
I am working with semantic segmentation on dataset with 5 classes including a null class including unlabeled pixels due to uncertainty, I am ignoring the null class in loss function computation, one of the classes consider is under represented , I have tried weighted loss, over and under sampling , augmentation but the model is  overfitting the minority class with IoU of 93% and on test it is 16%, I tried a lot of loss functions, and a paper considered asymmetric techniques with mixup and adversarial training but nothing seems to work. Do you have any suggestion to deal with this problem. I am using mIoU for comparing results as the main metric</p>
What should I use for declaring a layers __init__()?

yakhyo: <p>I used to use declaring all <code>ReLU</code> and <code>MaxPooling2d</code> layers in <code>__init__()</code> part of the model. However these these two functions have no learnable parameters? So Do I need to declare only <code>Conv2d</code> and <code>BatchNorm2d</code> in the <code>__init__()</code> part of the model? This below example is one of my code snippet of creating a Conv by stackin Convolutional, Normalization and activation functions:</p>
<pre><code class="lang-auto">class Conv(nn.Module):
    def __init__(self, in_channels, out_channels, kernel_size=3, stride=1, padding=1, dilation=1, groups=1,
                 bias=False) -&gt; None:
        super().__init__()
        self.conv = nn.Conv2d(in_channels=in_channels,
                              out_channels=out_channels,
                              kernel_size=kernel_size,
                              stride=stride,
                              padding=padding,
                              dilation=dilation,
                              groups=groups,
                              bias=bias)
        self.norm = nn.BatchNorm2d(num_features=out_channels)
        self.relu = nn.ReLU(inplace=False)

    def forward(self, x: torch.Tensor) -&gt; torch.Tensor:
        x = self.conv(x)
        x = self.norm(x)
        x = self.relu(x)

        return x
</code></pre>
<p>I just want to know what is the best practices for declaring <code>nn.MaxPooling2d</code> and <code>nn.ReLU</code>? Is it inside <code>__init__()</code> or outside of it?</p>

ptrblck: <p>I think it depends on your use case and eventually also depends on your coding style.<br>
E.g. if you plan to replace specific layers later, using the module approach might be easier than manipulating the <code>forward</code> method. <a href="https://discuss.pytorch.org/t/beginner-should-relu-sigmoid-be-called-in-the-init-method/18689/6">This post</a> explain the different approaches in more detail.</p>

yakhyo: <p>Thanks for the reply and that post. It was really helpful.</p>
THC and THCUNN not found on my machine

danyaljj: <p>The PyTorch code hs the following core libraries (as also mentioned in this post):</p>
<pre><code class="lang-auto"> TH, THC, THNN, THCUNN
</code></pre>
<p>When I am checking the installation on my own machine, I see the followings:</p>
<pre><code class="lang-auto">$ pwd
/Users/daniel/miniconda3/lib/python3.7/site-packages/torch/lib/include

$ ls -lha
total 0
drwxr-xr-x    9 daniel  staff   288B Dec 30 15:48 .
drwxr-xr-x   20 daniel  staff   640B Dec 30 15:48 ..
drwxr-xr-x  143 daniel  staff   4.5K Dec 30 15:48 ATen
drwxr-xr-x   39 daniel  staff   1.2K Dec 30 15:48 TH
drwxr-xr-x    4 daniel  staff   128B Nov  1 16:26 THNN
drwxr-xr-x   14 daniel  staff   448B Dec 30 15:48 c10
drwxr-xr-x    5 daniel  staff   160B Dec 30 15:48 caffe2
drwxr-xr-x   21 daniel  staff   672B Dec 30 15:48 pybind11
drwxr-xr-x    6 daniel  staff   192B Dec 30 15:48 torch
</code></pre>
<p>There are two core libraries missing: <code>THC</code>  and <code>THCUNN</code>. I checked my Linux installation and it contained the missing libraries.</p>

orange-416: <p>Hi,have you ever solved this problem? I also meet the dilemma that my Pytorch do not have include dir. And maybe your solution will help.</p>

ptrblck: <p>The <code>TH</code> namespace was merged to e.g. <code>Aten</code> so these files shouldn’t be there anymore.</p>
How to learn multiple binary classifiers?

se7en.dreams: <p>Hi, I have an input of shape <code>14 x 10 x 128 x 128</code>, where <code>14</code> is <code>batch_size</code>, <code>10</code> is the <code>sequence_length</code> and each item in the sequence is of shape <code>128 x 128</code>. I want to learn to map this input to output of shape <code>14 x 10 x 128</code>, i.e., for each item in the sequence I want to learn 128-binary classifiers.</p>
<p>Does the following model make sense? So, first I reshape my input to <code>140 x 128 x 128</code> and then pass it through the model and reshape the output back to <code>14 x 10 x 128</code> .</p>
<pre><code class="lang-python">classifier = nn.Sequential(
    nn.Conv1d(128, 128, 1),
    nn.ReLU(),
    nn.BatchNorm1d(128),
    nn.Conv1d(128, 128, 1),
    nn.ReLU(),
    nn.BatchNorm1d(128),
    nn.Conv1d(128, 1, 1)
)
</code></pre>
<p>Thank you.</p>

ptrblck: <p>The model architecture could work and the conv layers could be replaced with inear layers (but the current architecture might be easier to apply the batchnorm layers so you might want to keep it).<br>
The last reshaping might not be necessary as you are treating each time step as a separate sample in the model so could also do the same in the loss calculation, but again it also doesn’t seem to be wrong.</p>

se7en.dreams: <p>Thank you so much for your valuable feedback. Much appreciated!</p>
<p>Could you please take a quick look at the loss I’m calculating for the above model to see if I’m doing it correctly: <a href="https://discuss.pytorch.org/t/multi-label-classifier-loss/154149" class="inline-onebox">Multi-label Classifier Loss</a></p>
Performance and Confusion matrix

Fathima: <p>I have one training loop for train and valid. I want the accuracy, confusion matrix for train, and validation set.  I also want the matshow plot to display the numbers inside the graph. This is my code.</p>
<pre><code class="lang-auto">import numpy as np
batch_size = 32
epochs = 50
min_valid_loss = np.inf # track change in validation loss

train_losses, valid_losses = [], []
batch_train_losses, batch_valid_losses = [],[]
epoch_train_losses,epoch_valid_losses=[],[]

M = ConfusionMetrics()

for e in range(epochs):
    train_loss = 0.0
    model.train()     # Optional when not using Model Specific layer
    for data, target in train_loader:
         # Transfer Data to GPU if available
        if torch.cuda.is_available():
            data, target = data.cuda(), target.cuda()
            #print("targets.data", targets.item())
        
        
         # Clear the gradients
        optimizer.zero_grad()
        
        # Forward Pass
        output = (model(data))
        squeezed_output = torch.squeeze(output)
        
        # Find the Loss
        loss = criterion(squeezed_output.view(-1,1),target.view(-1,1))
        
        # Calculate gradients 
        loss.backward()
        # Update Weights
        optimizer.step()
        # Calculate Loss
        train_loss += loss.item()
        
        # calculate average losses
        #train_loss = train_loss/len(train_loader.dataset)
        train_losses.append(train_loss)
        epoch_train_loss =torch.tensor(train_losses).mean()
        epoch_train_losses.append(epoch_train_loss)
        
     
    #valid_loss = 0.0
    model.eval()     # Optional when not using Model Specific layer
    for data, target in valid_loader:
        if torch.cuda.is_available():
            data, target = data.cuda(), target.cuda()
        
        output = model(data)
        squeezed_output = torch.squeeze(output)
        print("=====LOSS===========",target)
        
        #loss = criterion(squeezed_output,target)
        loss = criterion(squeezed_output.view(-1,1),target.view(-1,1))
        valid_loss = loss.item() * data.size(0)
        
        
        #valid_loss = valid_loss/len(valid_loader.dataset)
        valid_losses.append(valid_loss)
        epoch_valid_loss=torch.tensor(valid_losses).mean()
        epoch_valid_losses.append(epoch_valid_loss)
        
        
        
        print('total confusion matrix')
        M.matrix()
        print('Accuracy',M.accuracy())
        print('Sensitivity', M.se())
        print('PPV', M.ppv())
        print(M.accumulate(target,output))
       

    print(f'Epoch {e+1} \t\t Training Loss: {train_loss / len(train_loader)} \t\t Validation Loss: {valid_loss / len(valid_loader)}')
    
   
    plt.matshow(M.matrix())
    reset = M.reset()
   

    if min_valid_loss &gt; valid_loss:
        print(f'Validation Loss Decreased({min_valid_loss:.6f}---&gt;{valid_loss:.6f}) \t Saving The Model')
        min_valid_loss = valid_loss
        
</code></pre>
Loss for Multi-label Classifier

se7en.dreams: <p>Hi, I am working on a multi-label classification problem. My gt labels are of shape <code>14 x 10 x 128</code>, where <code>14</code> is the <code>batch_size</code>, <code>10</code> is the <code>sequence_length</code>, and <code>128</code> is the vector with values <code>1</code> if the item in sequence belongs to the object and <code>0</code> otherwise.</p>
<p>My output is also of same shape: <code>14 x 10 x 128</code>. Since, my input sequence was of varying length I had to pad it to make it of fixed length <code>10</code>. I’m trying to find the loss of the model as follows:</p>
<pre><code class="lang-python">total_loss = 0.0
unpadded_seq_lengths = [3, 4, 5, 7, 9, 3, 2, 8, 5, 3, 5, 7, 7, ...] # true lengths of sequences

optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
criterion = nn.BCEWithLogitsLoss()

for data in training_dataloader:
    optimizer.zero_grad()

    # shape of input 14 x 10 x 128
    output = model(data)
    
    batch_loss = 0.0
    for batch_idx, sequence in enumerate(output):
        # sequence shape is 10 x 128
        true_seq_len = unpadded_seq_lengths[batch_idx]
        
        # only keep unpadded gt and predicted labels since we don't want loss to be influenced by padded values
        predicted_labels = sequence[:true_seq_len, :] # for example, 3 x 128
        gt_labels =  gt_labels_padded[batch_idx, :true_seq_len, :] # same shape as above, gt_labels_padded has shape 14 x 10 x 128
        
        # loop through unpadded predicted and gt labels and calculate loss  
        for item_idx, predicted_labels_seq_item in enumerate(predicted_labels):
            # predicted_labels_seq_item and gt_labels_seq_item are 1D vectors of length 128
            gt_labels_seq_item = gt_labels[item_idx]
            current_loss = criterion(predicted_labels_seq_item, gt_labels_seq_item)                     
            total_loss += current_loss
            batch_loss += current_loss


    batch_loss.backward()
    optimizer.step()

</code></pre>
<p>Can anybody please check to see if I’m calculating loss correctly. Thanks</p>
StopIteration: Caught StopIteration in replica 0 on device 0

ojipadeson: <pre><code class="lang-auto">Original Traceback (most recent call last):
  File "/home/jjrv/miniconda3/envs/layoutlm/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker
    output = module(*input, **kwargs)
  File "/home/jjrv/miniconda3/envs/layoutlm/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/data_nas/jjrv/P_LayoutLM-CW/model/modeling_bert.py", line 493, in forward
    head_mask=None,cx = cx ,cy = cy,height = height)
  File "/home/jjrv/miniconda3/envs/layoutlm/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/data_nas/jjrv/P_LayoutLM-CW/model/modeling_bert.py", line 336, in forward
    extended_attention_mask = extended_attention_mask.to(dtype=next(self.parameters()).dtype)  # fp16 compatibility
StopIteration
</code></pre>
<p><code>next(self.parameters())</code> caused the error. I wonder if it’s related to pytorch version?</p>

ptrblck: <p>I don’t think the <code>parameters()</code> method changed in the last year(s) so wouldn’t think the error is caused by an update in the PyTorch version.<br>
Your post doesn’t have any details so properly debug the issue, so I would recommend to check the <code>self</code> object (I assume it should be a model) and check if parameters are registered at all. Empty models would directly raise the <code>StopIteration</code> as seen here:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        
    def forward(self, x):
        return x

model = MyModel()
print(list(model.parameters()))
# []
next(model.parameters())
# StopIteration
</code></pre>
Is there any way to convert jit::script::Module to nn::Module

crissallan: <p>Hi,<br>
I rencently tried to load a trained Pytorch model which could do “Image Style transfer” to a C++ model.<br>
The style transfer needs the gradient of loss w.r.t the input tensor, but it seems that the jit::script::Module can’t pass the gradients to the input tensor.</p>
<p>So I’d like to know is there any way to convert a jit::script::Module to nn::Moudle conveniently?</p>
<p>Many thanks!</p>

eellison: <p>Hi, there is no way to convert a jit:script::Module to an nn::Module. However,  we should work on making it so that jit::script::Module can be used interchangeably. Can you comment on <a href="https://github.com/pytorch/pytorch/issues/46460" class="inline-onebox" rel="noopener nofollow ugc">C++ API: Training and inference of torchscript modules on multiple GPU · Issue #46460 · pytorch/pytorch · GitHub</a> with your specific use case ? Thanks.</p>

botelho: <p>Hi <a class="mention" href="/u/eellison">@eellison</a> , could you please help me understand why <code>jit::script::Module</code> is needed at all in libtorch? In other words, why isn’t it a simple thing for libtorch to read a model file saved in Python and build a <code>nn::Module</code> directly from it? I’ve struggled with this issue for years now, and never found a good answer.</p>
<p>For example, I have a libtorch-based application for which I have written code to load HDF5 models from Keras. Since the HDF5 contains both the network graph (as a json string) and the weights, I can programmatically construct the <code>nn::Module</code> from it and populate its trainable parameters. All without going through any format conversion! And, ironically, I cannot do the same with a model saved from Pytorch in python!! Any insights would be appreciated. Thanks.</p>
Using GradCAM in XceptionNet

universer: <p>Dear All:</p>
<p>I want to use GradCAM to visualize the feature map in XceptionNet, but I do not exactly how to use it.<br>
My part code is showing below:</p>
<pre><code class="lang-auto">target_layer = [model.conv4[-1]]. // I do not know which is the target_layer to XceptionNet

rgb_img = Image.open('267.jpg').convert('RGB')
rgb_img = xception_default_data_transforms['test'](rgb_img)
rgb_img = torch.unsqueeze(rgb_img, dim=0)
input_tensor = rgb_img

cam = GradCAM(model=model, target_layers=target_layer, use_cuda=False)

target_category = None
grayscale_cam = cam(input_tensor=input_tensor, target_category=target_category)  
grayscale_cam = grayscale_cam[0]
visualization = show_cam_on_image(rgb_img, grayscale_cam)  
cv2.imshow('image.jpg', visualization)
</code></pre>
<p>Thanks.</p>
<p>GradCAM Reference: <a href="https://github.com/jacobgil/pytorch-grad-cam" class="inline-onebox" rel="noopener nofollow ugc">GitHub - jacobgil/pytorch-grad-cam: Many Class Activation Map methods implemented in Pytorch for CNNs and Vision Transformers. Examples for classification, object detection, segmentation, embedding networks and more. Including Grad-CAM, Grad-CAM++, Score-CAM, Ablation-CAM and XGrad-CAM</a><br>
XceptionNet Reference: <a href="https://github.com/tstandley/Xception-PyTorch/blob/master/xception.py" class="inline-onebox" rel="noopener nofollow ugc">Xception-PyTorch/xception.py at master · tstandley/Xception-PyTorch · GitHub</a></p>
Can I use CPU mem to save GPU mem at sequential model?

luofan: <p>For example, I have a big module “BigNet”, and two gpus. Each gpu’s memory can only allow me to train one module. I know I can do this</p>
<pre><code class="lang-auto">net1=BigNet().to(gpu1) # optimizer 1
net2=BigNet().to(gpu2) # optimizer 2
X=X.to(gpu1)
y=net1(X)
y=net2(y.to(gpu2))
loss=loss_fn(y, label)
loss.backward()
optim1.step()
optim2.step()
</code></pre>
<p>By this way, I can train a bigger model with two modules. gpu1 is idle when gpu2 is operating. Can I back up the mem of gpu1 to CPU mem at this time, train the net3 on gpu1, and copy net1 from memory to GPU when the loss is back-propagated to net1? For example:</p>
<pre><code class="lang-auto">net1=BigNet().to(gpu1) # optimizer 1
net2=BigNet().to(gpu2) # optimizer 2
net3=BigNet() # optimizer 3
X=X.to(gpu1)
y=net1(X)
y=net2(y.to(gpu2))
net1.to(cpu)
net2.to(gpu1)
y=net3(y.to(gpu1))
loss=loss_fn(y, label)
loss.backward(Control) # control loss backward at net3, net2
optim3.step()
optim2.step()
net3.to(cpu)
net1.to(gpu1)
loss.backward(Control) # control loss backword at net1
optim1.step()
</code></pre>
<p>Thanks</p>

ptrblck: <p>You could check the <a href="https://pytorch.org/docs/master/autograd.html#saved-tensors-default-hooks">CPU offloading</a> for your use case.</p>

luofan: <p>Thanks, the hook, and “save on CPU” works for me.</p>
Min requirements for torchserve

skoenman: <p>Morning Guys</p>
<p>What is the min requirements for deploying torchserve?<br>
My current system isnt low spec but seems to be hanging every time i up the workers specs as follows.</p>
<p>Ubuntu 18.04<br>
8gig mem<br>
I5 6th gen<br>
Ssd<br>
2x 3060ti<br>
Model maskrcnn<br>
Compiled with cudo 11.6<br>
No docker.</p>

marksaroufim: <p>The <code>maskrcnn</code> model has some known issues where it hangs otherwise your minimum requirements seem fine, I use a less fancy machine than you</p>
<p>If you’re still getting an issue with other models then please open an issue here and tag me, would be happy to help <a href="https://github.com/pytorch/serve" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/serve: Serve, optimize and scale PyTorch models in production</a></p>

skoenman: <p>Thanks for the reply… will give it a go. And try another model… any idea on how to speedup the inference when using torchserve? Im trying to use it on machine at the moment but performance seems to be severely limited by the number of workers? Not sure if its maybe because in using cud116?</p>

marksaroufim: <p>Hard to say without more information but I’d start with checking how slow your model is outside of Torchserve - this performance guide may help <a href="https://github.com/pytorch/serve/blob/master/docs/performance_guide.md" class="inline-onebox" rel="noopener nofollow ugc">serve/performance_guide.md at master · pytorch/serve · GitHub</a> and our benchmark tool <a href="https://github.com/pytorch/serve/tree/master/benchmarks" class="inline-onebox" rel="noopener nofollow ugc">serve/benchmarks at master · pytorch/serve · GitHub</a></p>
Why do apps using pytorch get size errors when deployed on heroku?

YuhskeHujisaki: <p>When deploying an api using pytorch to heroku, I get Compiled slug size: XXX M is too large.  What should I do?</p>

andrewor: <p>Hi Yuhske,</p>
<p>Are you using the pytorch quantization API? How large is your model? Without further information it’ll be difficult to identify what the issue is. An alternative is to maybe solve this from the heroku side, e.g. I found some settings you might be able to try here: <a href="https://stackoverflow.com/questions/30199856/heroku-compiled-slug-size-is-too-large" class="inline-onebox" rel="noopener nofollow ugc">ruby on rails - Heroku: Compiled Slug Size is too large - Stack Overflow</a>.</p>
<p>Best,<br>
-Andrew</p>
How to sync a manually updated lookup table in DDP

origin_of_symmetry: <p>Hi,</p>
<p>I want to implement a Module that has a lookup table which is manually updated with module outputs. The idea is in this paper: <a href="https://arxiv.org/pdf/2008.01466.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/2008.01466.pdf</a></p>
<p>If I use DDP, different GPU copies of the table (nn.Embedding, require_grad=False) could be different. But I want a synced table among different GPUs. What’s the best way to implement this?</p>
<p>I can think of keeping this table in CPU memory, and exchange data between CPU and GPUs, but no sure if this will affect the training speed a lot.</p>
<p>Thanks</p>

initmahesh: <aside class="quote no-group" data-username="origin_of_symmetry" data-post="1" data-topic="153052">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/origin_of_symmetry/48/29831_2.png" class="avatar"> origin_of_symmetry:</div>
<blockquote>
<p>I can think of keeping this table in CPU memory, and exchange data between CPU and GPUs, but no sure if this will affect the training speed a lot.</p>
</blockquote>
</aside>
<p><a class="mention" href="/u/pritamdamania87">@pritamdamania87</a> to follow up here.</p>

harpone: <p>ouch sounds nasty… if the tables are pytorch tensors, then I think you should be able to do that with <code>torch.distributed.reduce</code> and gather etc (<a href="https://github.com/facebookresearch/detectron2/blob/main/MODEL_ZOO.md" class="inline-onebox" rel="noopener nofollow ugc">detectron2/MODEL_ZOO.md at main · facebookresearch/detectron2 · GitHub</a>)</p>
How to initialize weights in nn.Sequential container?

sceamms: <p>How do I use nn.init.xavier_normal() to initialize weights inside nn.Sequential container like the one below? Thanks for your help!</p>
<pre><code>class CNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Sequential(    # input shape (1, 60, 60)
            nn.Conv2d(
                in_channels=1,    # input channel
                out_channels=32,  # output channels - &gt; 32
                kernel_size=5,    # filter size
                stride=1,         # step size
                padding=2,        # padding = (kernel size - 1)/2
            ),                    # output (32, 60, 60)
            nn.BatchNorm2d(32),   
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2),  # same output (32, 30, 30) as stride =2 
/kernel size
            nn.Dropout2d(0.7), 
        )
        self.conv2 = nn.Sequential(     # input (32, 30, 30)
            nn.Conv2d(
                in_channels=32,
                out_channels=64,
                kernel_size=3,
                stride=1,
                padding=1,
            ),
            nn.BatchNorm2d(64),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2),  # output (64, 15, 15)
            nn.Dropout2d(0.7),   
        )
        # before processing to the dense layer, it need to be flatenned into: batch size, 
64*15*15
        self.fc1 = nn.Sequential(
            nn.Linear(64*15*15,400),  # need to change this, when picture size is changed.
            nn.BatchNorm1d(400),
            nn.ReLU(),
            nn.Dropout(0.7),
        )
        self.fc2 = nn.Sequential(
            nn.Linear(400, 7),
            nn.Dropout(0.5), 
            nn.LogSoftmax()
        )

    def forward(self, x):
        x = self.conv1(x)
        x = self.conv2(x)
        x = x.view(x.size(0), -1)
        x = self.fc1(x)
        output = self.fc2(x)
        return output</code></pre>

smth: <p>You can loop over container’s modules.</p>
<p>See here for an example (follow the usage of weights_init function): <a href="https://github.com/pytorch/examples/blob/master/dcgan/main.py#L95">https://github.com/pytorch/examples/blob/master/dcgan/main.py#L95</a></p>

sceamms: <p>thanks. actually I find the piece in the the standard document is directly answering my questions.</p>
<blockquote>
<blockquote>
<blockquote>
<p>def init_weights(m):<br>
print(m)<br>
if type(m) == nn.Linear:<br>
m.weight.data.fill_(1.0)<br>
print(m.weight)</p>
<p>net = nn.Sequential(nn.Linear(2, 2), nn.Linear(2, 2))<br>
net.apply(init_weights)</p>
</blockquote>
</blockquote>
</blockquote>

hull11: <p>What does the parameter m here signify?</p>

Wan-ZL: <p>it inputs layers in net one by one.</p>
Divide gradients for pretrained subnetwork

scientia: <p>I’m using a pretrained network, built some layers on top of it, and then finetune. But I wonder if we can divide the gradients for the subnetwork only (the lower layers) by some factor to avoid catastrophic forgetting as soon as we start the training.</p>

ptrblck: <p>Yes, you could scale the gradients before the <code>optimizer.step()</code> method and after calling <code>loss.backward()</code> by accessing the <code>.grad</code> attributes of the desired parameters (the gradient clipping utilities might also be interesting for your use case).<br>
Alternatively, creating different optimizers with different learning rates (smaller learning rate for the “lower layers”) could also work.</p>
0D or 1D, multi target not supported

ifunayaScript: <pre><code class="lang-auto">import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

class WineDataset(Dataset):
    def __init__(self):
        # data loading
        xy = np.loadtxt(r'C:\Users\USER\Desktop\MyDatasets\wine_data.csv', delimiter=',', dtype=np.float32, skiprows=1)
        self.x = torch.from_numpy(xy[:, 1:])
        self.y = torch.from_numpy(xy[:, 0])
        self.n_samples = xy.shape[0]
    def __getitem__(self, index):
        return self.x[index], self.y[index]
    def __len__(self):
        return self.n_samples
        
dataset = WineDataset()
dataloader = DataLoader(dataset=dataset, batch_size=4, shuffle=True)

samples = iter(dataloader)
features, targets = samples.next()

input_features = features.shape[1]
hidden_size = 40
num_classes = len(np.unique(targets))

class Model(nn.Module):
    def __init__(self, input_features, hidden_size, num_classes):
        super(Model, self).__init__()
        self.l1 = nn.Linear(input_features, hidden_size)
        self.relu = nn.ReLU()
        self.l2 = nn.Linear(hidden_size, num_classes)
    
    def forward(self, x):
        out = self.l1(x)
        out = self.relu(out)
        out = self.l2(out)
        return out

# Define some training parameters
model = Model(input_features, hidden_size, num_classes)
learning_rate = 0.01
epochs = 10
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)

# Training loop
n_total_steps = len(dataloader)
for epoch in range(epochs):
    for i, (features, targets) in enumerate(dataloader):
        
        # forward pass and loss calculation
        prediction = model(features)
        #print(prediction.shape, targets.shape)
        loss = criterion(prediction, targets)
        
        # empty gradients and backward pass
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        
        if (i+1) % 100 == 0:
            print(f'epoch: {epoch+1}/{epochs}, step: {i+1}/{n_total_steps}, loss: {loss.item():.4f}')
           
            
# ifunanyaScript           

</code></pre>

ptrblck: <p>Check the shape and values of the <code>prediciton</code> and <code>targets</code> tensors.<br>
In a standard multi-class classification use case <code>nn.CrossEntropyLoss</code> expects a model output in the shape <code>[batch_size, nb_classes]</code> containing logits and a target tensor in the shape <code>[batch_size]</code> containing class indices in the range <code>[0, nb_classes-1]</code>.</p>
Find which line is using pytorch operation

whoab: <p>I got this result from the profiler:</p>
<p><code>                              aten::ne         5.64%     372.377ms        16.59%        1.095s      39.548us     390.012ms         5.91%        1.099s      39.721us         27680</code></p>
<p>As far as I can tell through searching my code, I am not calling <code>torch.ne</code> anywhere. How can I trace exactly which lines are using this function?</p>

harpone: <p>Probably some other function is calling <code>torch.ne</code>, i.e. comparing if tensor elements are smaller than or equal elementwise…</p>

ptrblck: <p>You could check if the profiler shows this method in its stacktrace as shown <a href="https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html#examining-stack-traces">here</a> and might then be able to narrow it down.</p>
Convert pretrained resnet for segmentation

randomuser19: <p>Hi all,</p>
<p>I have a pretrained resnet model that I have used for classification. Is it possible to convert the output of the model for a semantic segmentation task? If so, how? I would just like to evaluate the model for semantic segmentation, not entirely retrain the model. The output from the model is just the batch size x feature number. I’m wondering just how to convert this to segmentation. Many thanks!</p>

ptrblck: <p>One difference between your classification and a segmentation model would be the output shape and thus the last used layer. You would need to make sure the logits for each pixel are given and not only for the current sample. Replacing the “classifier” (usually linear layers with relus etc.) with conv layers could work, but you would still need to fine-tune these newly initialized layers.<br>
Since your current output has less values than the segmentation use case would need, I don’t think there exist a proper way to convert it to a segmentation use case without changing the model architecture.</p>

randomuser19: <p>Thanks for the response! I think I would probably implement fine-tuning. Do you have any recommendation to go about retrieving the pixel-level logits?</p>

ptrblck: <p>You could start by replacing the classifier with its linear layers with a new one using conv layers.<br>
This would make sure the output of 4-dimensional as <code>[batch_size, nb_classes, height, width]</code>. Depending on the used architecture, the spatial sizes (height and width) would often be smaller in the later activation maps, so your output would either also be smaller in its spatial size (compared to the input), you could try to upscale the activation in the new classifier, or change the entire model architecture to avoid aggressive pooling.</p>