akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Use the BatchNorm and forward the batch sample by sample

Torcione: <p>For my project I have to record the gradient associated to each single sample of a generic batch (not the mean/sum given by the propagation of the whole group). I am working on a VGG16 architecture with <code>nn.BatchNorm2d</code> layers.<br>
If I split the batch in single elements to record gradients time by time and finally sum them togheter before the <code>optimizer.step()</code> I cannot use <code>nn.BatchNorm2d</code> (single element batch). Is there a way to keep the <code>nn.BatchNorm2d</code> layer and still forward samples one by one?<br>
(for example making a first forwarding with the whole batch, fixing by it (somehow) the Batch Norm state associated to the batch and then repeat the propagation sample by sample)</p>
Getting Info on Connections Between Layers in Models with Split Connections

J_Johnson: <p>I am clear on how to get various layer information via iterating:</p>
<pre><code class="lang-auto">for child, name in zip(model.children(), model.named_children()):
print(child, name, type(child), name[0])
</code></pre>
<p>etc.</p>
<p>And I am clear on how to get the number of parameters and their values via:</p>
<pre><code class="lang-auto">for layer in model.parameters():
    print(layer.data.size())
</code></pre>
<p>In a sequential model, the connection sizes will be equal to the layer parameters in their corresponding connected dimension.</p>
<p>But what I am not clear on is how to find the name and sizes of the next connected layers in a non-sequential model.</p>
<p>For instance, suppose we have the following dummy model with a split connection:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

class Split_Model(nn.Module):
    def __init__(self):
        super(Split_Model, self).__init__()
        self.fc1 = nn.Linear(1, 20)
        self.fc2a = nn.Linear(10, 10)
        self.fc2b = nn.Linear(10, 10)
        self.fc3 = nn.Linear(20, 1)

    def forward(self, x):
        x = self.fc1(x)
        y, x = x[:, :10], x[:, 10:20]
        x = self.fc2a(x)
        y = self.fc2b(y)
        x = torch.cat([x, y], dim=1)
        x = self.fc3(x)
        return x


model= Split_Model()
</code></pre>
<p>Split connections are becoming more useful as they better interconnect a model and eliminate vanishing gradients in very long models. How would I be able to find the next layers in the graph that fc1 are connected to? Let’s say I wanted something like:</p>
<pre><code class="lang-auto">print(model.fc1.???)
&gt;&gt;&gt; fc2a: 10 , fc2b: 10 #something like this is what I would ideally like  for an output
</code></pre>
<p>How would I get this information out of a defined model with split connections?</p>
<p>Thank you!</p>
How to save memory by not tracking some variable in training process?

Ferd: <p>I am trying to save some memory in training process to avoid OOM.<br>
Here is the original code block:</p>
<pre><code class="lang-auto">entropy = torch.sum(-(pred1_softmax + 1e-9).log() * pred1_softmax, dim=1)   # pred1_softmax size: [1, 192, 384, 1248]
</code></pre>
<p>Running the line above will cause an increase of 700+ MB in memory occupancy. I guess it is because pytorch saves -(pred1_softmax + 1e-9).log(),  which is a big tensor with the same size of pred1_softmax, in order to compute grad in the backward pass.  So, I want to wrap the block with torch.no_grad. Here is the modified code:</p>
<pre><code class="lang-auto">with torch.no_grad():
    entropy = torch.sum(-(pred1_softmax + 1e-9).log() * pred1_softmax, dim=1)
</code></pre>
<p>However, it doesn’t work. It still occupies more than 700MB memory and I am confused.</p>
<p>Any help would be appreciated.</p>

techkang: <p>use torch.no_grad() will only disable gradient, but <code>entropy</code> is still in GPU. You can use following code to profile you GPU memory usage.</p>
<pre><code class="lang-auto">import torch as t

data = t.rand(1, 192, 384, 1248)
print(t.cuda.memory_reserved(0))
data = data.cuda()
print(t.cuda.memory_reserved(0))
data = t.softmax(data, -1)
print(t.cuda.memory_reserved(0))
data.requires_grad = True
print(t.cuda.memory_reserved(0))
# entropy = t.sum(-(data + 1e-9).log() * data, dim=1)
# print(t.cuda.memory_reserved(0))
# del entropy
# print(t.cuda.memory_reserved(0))
with t.no_grad():
    entropy = t.sum(-(data + 1e-9).log() * data, dim=1)
    print(t.cuda.memory_reserved(0))
</code></pre>

Ferd: <p>Thank you for your reply!<br>
I know <em><strong>entropy</strong></em> is still in GPU, what I want to do is to eliminate the intermediate tensor <em><strong>-(pred1_softmax + 1e-9).log()</strong></em>, cause it is much larger than <em><strong>entropy</strong></em> itself.</p>
<pre><code class="lang-auto"># data represent for the intermediate tensor of shape [1, 192, 384, 1248]
# entropy is the tensor of shape [1, 384, 1248]

&gt;&gt;&gt; data = torch.rand([1, 192, 384, 1248], device='cuda')
&gt;&gt;&gt; torch.cuda.memory_allocated() / 1024 / 1024    # convert to MB
352

&gt;&gt;&gt; entropy = torch.rand([1, 384, 1248], device='cuda')
&gt;&gt;&gt; torch.cuda.memory_allocated() / 1024 / 1024
1.828125
</code></pre>

Ferd: <p>Again, really appreciate your help!  Your answer inspired me.<br>
<em><strong>torch.no_grad()</strong></em> actually works, the intermediate tensor is not in GPU anymore. However, the occupancy does increase a lot. I find that part of memory is cached.</p>
<p>And using</p>
<pre><code class="lang-auto">torch.cuda.empty_cache()
</code></pre>
<p>can release the memory.</p>
Running pytorch model on android. 

Alexey322: <p>Hi. I want to use my model built for pytorch for android. For windows, during inference the model takes about 5-6 GB of GPU memory. Can you please tell me how to find out if it can run on an android device? Will the model use the android GPU in the same way as it does on the pc?<br>
I would also be grateful if you could give me materials to read for this task. Thanks.</p>
Fast differentiable blurring

ZimoNitrome: <p>How can I efficiently blur a binary image using a floating-point parameter for the amount of blur?</p>

ZimoNitrome: <p>This might work:</p>
<pre><code class="lang-python">import torch.nn.functional as F
import torch

def make_gaussian_kernel(kernel_size, sigma):
    ts = torch.linspace(-kernel_size // 2, kernel_size // 2 + 1, kernel_size)
    gauss = torch.exp((-(ts / sigma)**2 / 2))
    kernel = gauss / gauss.sum()
    return kernel

def fast_gaussian_blur(img: torch.Tensor, sigma: float) -&gt; torch.Tensor:
    trailing_dims = img.shape[:-3]
    kernel_size = int(sigma * 5)
    if kernel_size % 2 == 0:
        kernel_size += 1
    kernel = make_gaussian_kernel(kernel_size, sigma)

    # padding = (left, right, top, bottom)
    padding = [kernel_size // 2, kernel_size // 2, kernel_size // 2, kernel_size // 2]
    img = F.pad(img, padding, mode="constant", value=0)

    # Separable 2d conv
    kernel = kernel.view(*trailing_dims, 1, kernel_size, 1)
    img = F.conv1d(img, kernel)
    kernel = kernel.view(*trailing_dims, 1, 1, kernel_size)
    img = F.conv1d(img, kernel)

    return img
</code></pre>
<p>I have not fully tried it out yet. Any and all feedback is appreciated <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Shapes cannot be multiplied

KanZa: <p>I am training my model on ImageNet dataset the backbone network is very big so I decreased the batch size my new error is given below:</p>
<p>RuntimeError: mat1 and mat2 shapes cannot be multiplied (16x25088 and 512x1000)</p>
<p>What changes do I need to make to train my network on ImageNet</p>
<p>class LossNet(nn.Module):<br>
def <strong>init</strong>(self, feature_sizes=[32, 16, 8, 4], num_channels=[64, 128, 256, 512], interm_dim=128):<br>
super(LossNet, self).<strong>init</strong>()</p>
<pre><code>    self.GAP1 = nn.AvgPool2d(feature_sizes[0])
    self.GAP2 = nn.AvgPool2d(feature_sizes[1])
    self.GAP3 = nn.AvgPool2d(feature_sizes[2])
    self.GAP4 = nn.AvgPool2d(feature_sizes[3])

    self.FC1 = nn.Linear(num_channels[0], interm_dim)
    self.FC2 = nn.Linear(num_channels[1], interm_dim)
    self.FC3 = nn.Linear(num_channels[2], interm_dim)
    self.FC4 = nn.Linear(num_channels[3], interm_dim)

    self.linear = nn.Linear(4 * interm_dim, 1)

def forward(self, features):
    out1 = self.GAP1(features[0])
    out1 = out1.view(out1.size(0), -1)
    out1 = F.relu(self.FC1(out1))

    out2 = self.GAP2(features[1])
    out2 = out2.view(out2.size(0), -1)
    out2 = F.relu(self.FC2(out2))

    out3 = self.GAP3(features[2])
    out3 = out3.view(out3.size(0), -1)
    out3 = F.relu(self.FC3(out3))

    out4 = self.GAP4(features[3])
    out4 = out4.view(out4.size(0), -1)
    out4 = F.relu(self.FC4(out4))

    out = self.linear(torch.cat((out1, out2, out3, out4), 1))
    return out</code></pre>

tom: <p>This says that a linear layer gets the wrong number of features (and you’d know from the backtrace which). I would propose to print all shapes of interest in the forward to check on it.<br>
Quite likely this is the average pool not doing what you think it does, maybe AdaptiveAveragePool (or just calling “features[0].mean([2, 3])”) is a better fit for your purpose.</p>
<p>Best regards</p>
<p>Thomas</p>
Is there pytorch function which is similar to numpy.in1d?

MinJeong_Kim: <p>Hi everyone,</p>
<p>I would like to compare the two arrays whether the element in one array exists in the other array especially for the one dimensional array which works same as <a href="https://docs.scipy.org/doc/numpy/reference/generated/numpy.in1d.html" rel="nofollow noopener">numpy.in1d</a>.<br>
Is there any pytorch function corresponding to this?</p>

smth: <p>Here’s what you want:</p>
<pre><code class="lang-auto">a.eq(b)
</code></pre>

MinJeong_Kim: <p>Thank you very much!</p>

Jacob005: <p>Hi, I checked the document of eq. It is not equivalent to numpy.in1d.</p>

Linux-cpp-lisp: <p>A year later, is there now such a function? Or an efficient way of implementing similar behavior?</p>

Goodwin: <p>Thanks for the information Keep suggesting this type of post. looking for the same info.</p>

Zayd: <p><code>isin</code> has native support starting in <code>torch</code> 1.10.  See: <a href="https://pytorch.org/docs/master/generated/torch.isin.html?highlight=isin#torch.isin" class="inline-onebox" rel="noopener nofollow ugc">torch.isin — PyTorch master documentation</a></p>
RuntimeError: no deterministic convolution algorithms available in CuDNN

Ford: <p>When lauched a multi-machine-multi-card training program with the help of horovod module, I encountered the error “RuntimeError: no deterministic convolution algorithms available in CuDNN”. My pytorch version is pytorch 1.4. Can anyone know  how to fix this error? Thanks</p>

ptrblck: <p>Could you update PyTorch to the latest stable release and rerun the code, please?<br>
In case you are still hitting the issue, could you then post a minimal, executable code snippet so that we could try to reproduce it?</p>

Ford: <p>ok, thanks, I will try it</p>
Dropout in LSTM during eval mode

helloybz: <p>Hi.</p>
<p>In pytorch implementation, LSTM takes droupout argument for its constructor, which determines the probability of dropout.</p>
<p>Any ideas on whether dropouts are ignored in evaluation mode? Ex) model.eval()</p>

ptrblck: <p>Yes, it should be disabled:</p>
<pre><code class="lang-python"># no dropout
model = nn.LSTM(10, 10, 2, dropout=0.0)
x = torch.randn(10, 10, 10)
out1, _ = model(x)
out2, _ = model(x)
print((out1 - out2).abs().max())
&gt; tensor(0., grad_fn=&lt;MaxBackward1&gt;)

# dropout
model = nn.LSTM(10, 10, 2, dropout=0.5)
x = torch.randn(10, 10, 10)
out1, _ = model(x)
out2, _ = model(x)
print((out1 - out2).abs().max())
&gt; tensor(0.1524, grad_fn=&lt;MaxBackward1&gt;)

# dropout + eval
model.eval()
out1, _ = model(x)
out2, _ = model(x)
print((out1 - out2).abs().max())
&gt; tensor(0., grad_fn=&lt;MaxBackward1&gt;)
</code></pre>

uddeshya_upadhyay: <p>How to activate dropouts in LSTM during <code>eval</code> mode?</p>

uddeshya_upadhyay: <p>Oh got it. Just don’t use <code>eval</code> mode. Just use <code>torch.no_grad()</code></p>
Does Pytorch have something similar to np.isin?

qiminchen: <p>Hi, I want to replace all elements in a tensor that are <strong>equal</strong>  to some values. In NumPy, I can simply do <code>arr[np.isin(arr, some_values)]</code> where <code>some_values</code> here is a list. Below can achieve what I need but is there any way not to use the <code>for</code> loop?</p>
<pre><code class="lang-auto">for v in [some_values]:
    arr[arr == v] = 0
</code></pre>

qiminchen: <p><code>arr[(arr[..., None] == [some_values]).any(-1)] = 0</code> works</p>

Zayd: <p>Update on this. <code>torch</code> 1.10 now includes native support for <code>isin</code>.  See: <a href="https://github.com/pytorch/pytorch/issues/3025" class="inline-onebox" rel="noopener nofollow ugc">Function request: np.isin · Issue #3025 · pytorch/pytorch · GitHub</a></p>
How can I dropout for torch.complex data type?

leeseokho: <p>Can I use nn.Dropout() to cfloat dtype?<br>
Always the warning sign, “fused_dropout” not implemented for ‘ComplexFloat’, is popped up.</p>

tom: <p>It seems that it is not implemented yet.</p>
<p>I’d probably work around it as</p>
<pre><code class="lang-auto">def dropout_complex(x):
    # work around unimplemented dropout for complex
    if x.is_complex():
        mask = torch.nn.functional.dropout(torch.ones_like(x.real))
        return x * mask
    else:
        return torch.nn.functional.dropout(x)

a = torch.randn(5, 5, dtype=torch.cfloat)
print(dropout_complex(a))
</code></pre>
<p>Best regards</p>
<p>Thomas</p>

leeseokho: <p>How did you do that? you are genius <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to create multiple nn.Linear() using a loop?

Audrey: <p>This question may be shallow. I am implementing a GNN model, and for each relation type I want to create a weight matrix <strong>W</strong>r using <code>nn.Linear(dim, dim)</code>.  This is a piece of my code:</p>
<pre><code class="lang-auto">class GNN(nn.Module):
    def __init__(self, entity_num, relation_num, dim, args):
        super(GNN, self).__init__()
        self.entity_num = entity_num
        self.relation_num = relation_num
        self.dim = dim
        # create entity and relation embedding lookup  table
        self.entity_embedding = torch.nn.Embedding(self.entity_num, self.dim, padding_idx=self.pad_index)
        self.relation_embedding = torch.nn.Embedding(self.relation_num, self.dim)
        # The next line is a single weight matrix W, but I want to create multiple W
        # self.W = nn.Linear(self.dim, self.dim)
        self.W_list = ?  # [nn.Linear(self.dim, self.dim) for i in range(relation_num)] 
</code></pre>
<p>How to implement <code>self.W_list = ?</code></p>

a_d: <p>You might want to use a module list.</p>
<pre><code class="lang-auto">self.W_list =  nn.ModuleList()

for i  in range(relation_num):
    self.W_list.append(nn.Linear(self.dim, self.dim))
</code></pre>

Audrey: <p>This’s viable! Thank you. <a class="mention" href="/u/a_d">@a_d</a></p>
Create final image with CNN

randino: <p>I divided the original image (100*100) into (5.5) patches , and I took train and test percentages of the original image , but I want to find the test and train  result in forma image?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/9/09f0ccd0d1bc1c7747d0ced179d493e966d849d2.png" alt="image" data-base62-sha1="1pWcXlexl1T8VudkqCIJRsu6yP0" width="457" height="97"></p>
<pre><code class="lang-auto">train_losses = []
    # empty list to store validation losses
    val_losses = []
    correct = 0
    total = 0
    predictions =[]
    actuals=[]
    predictiontest=[]
    # training the model
    m1=0
    m2=0
    for epoch in range(n_epochs):
      train(epoch)
    plt.plot(train_losses, label='Training loss')
    plt.plot(val_losses, label='Validation loss')
    plt.legend()
    plt.show()
    with torch.no_grad():
        output1 = model((train_x.float()),(train_x1.float()))     
        softmax1 = torch.exp(output1).cpu()    
        prob1 = list(softmax1.numpy())
        predictions1 = np.argmax(prob1, axis=1)
        print('Validation accuracy train: {:.4f}%'.format(float(accuracy_score(train_y, predictions1)) * 100))
    with torch.no_grad():    
        output = model((val_x.float()),(val_x1.float()))
        val_y = val_y.to(device)    
        softmax = torch.exp(output).cpu()       
        prob = list(softmax.numpy())
        predictions = np.argmax(prob, axis=1)
        
        total += val_y.size(0)
        for i in range(predictions.size) : 
          if (predictions[i] == val_y[i]): 
             correct= correct+1
        
        print(correct)
        
        print('Validation accuracy test: {:.4f}%'.format(float(accuracy_score(val_y, predictions)) * 100))
j=0;
for i in range(predictions.size) :
  val_y[i] = predictions[i]   
plt.imshow(val_y)  
conf_matrix  = confusion_matrix(val_y, predictions)     
print(conf_matrix)
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/3/e34a7867bcf8fdddd53e489ae26328dff303472c.png" alt="image" data-base62-sha1="wqHW8GJRCk5RI7LUYw7CacTWFc0" width="602" height="189"></p>
<p>help me plz</p>
Can Pytorch run natively on Mac M1 chip & use M1's GPU?

tnguyen87: <p>Greetings,</p>
<p>I wonder if I Pytorch run natively on mac M1 chip? I read some threads and saw comments that it was not compatible yet, but those comments were back in March 2021.</p>
<p>Furthermore, can I run Pytorch with GPU of M1 (no CUDA here) rather than CPU only?</p>
<p>I would very much appreciate it if these questions can be answered. I am planning to get a MacBook to run Pytorch, especially with GPU.</p>
<p>Thank you,</p>

tom: <p>The current state is summarized by <a href="https://github.com/pytorch/pytorch/issues/47702#issuecomment-953074900" rel="noopener nofollow ugc">Soumith in this bug report</a>:</p>
<blockquote>
<p>we’ll hopefully communicate a better update later this week, or next week.<br>
Co-working with hardware companies to add backends always results in a bit of tightly controlled communication, so sorry we can’t keep our usual fully-open spirit.</p>
</blockquote>
<p>Best regards</p>
<p>Thomas</p>
How to append to a tensor in a loop

ck172: <p>I have a for loop where each iteration will give a tensor in this shape: tensor([[[0., 1., 1., 2., 1., 1., 0.],<br>
[0., 2., 1., 1., 2., 1., 0.],<br>
[0., 1., 5., 3., 1., 1., 0.]]])</p>
<p>How can I append new tensors to this one and form a tensor in this shape at last:<br>
(for example, in second iteration the tensor will be<br>
tensor([[[0., 0., 1., 2., 1., 0., 0.],<br>
[0., 0., 2., 3., 3., 0., 0.],<br>
[0., 0., 1., 2., 2., 0., 0.]]]))<br>
The expected output at last is:</p>
<p>tensor([[[0., 1., 1., 2., 1., 1., 0.],<br>
[0., 0., 1., 2., 1., 0., 0.]],<br>
[[0., 2., 1., 1., 2., 1., 0.],<br>
[0., 0., 2., 3., 3., 0., 0.]],<br>
[[0., 1., 5., 3., 1., 1., 0.],<br>
[0., 0., 1., 2., 2., 0., 0.]]])</p>

thecho7: <p>You can concatenate the tensors along the specific dimension.<br>
Your question can be briefly expressed like below,</p>
<pre><code>a = torch.Size(1, 3, 7)
b = torch.Size(1, 3, 7)

result = torch.cat((a, b), dim=1)
</code></pre>
<p>Then, you can get  the result tensor size of (1, 6, 7)<br>
The sample code</p>
<pre><code>for i in range(it):
    try:
        a = torch.cat((a, new_a), dim=1)
    except:
        a = new_a</code></pre>
Problem with my input data structure using Torchvision RetinaNet? Minimal example included

asi212: <p>An error occurs while trying to compute loss in the classification head. To the best I can tell, my input data is in the structure required by the torch vision RetinaNet. However, I still get this error that prevents me from training.</p>
<p>I’ve replaced the example minimally below. The error message you receive should be “index 1 is out of bounds for dimension 1 with size 1” in compute_loss(args) method of the classification head.</p>
<p>Thank you in advance for any help.</p>
<p>Minimal code:</p>
<pre><code class="lang-auto">
''' Adapted from example in PyTorch code '''
import torch
import torchvision
from torchvision.models.detection.anchor_utils import AnchorGenerator
from torchvision.models.detection.backbone_utils import resnet_fpn_backbone
from torchvision.models.detection import RetinaNet

''' Backbone '''
backbone = resnet_fpn_backbone('resnet18', pretrained=False, trainable_layers=4)
backbone.out_channels = 256

''' Anchor Generator '''
anchor_sizes = ((32,), (64,), (128,), (256,), (512,)) 
aspect_ratios = ((0.5, 1.0, 2.0),) * len(anchor_sizes) 
anchor_generator = AnchorGenerator(sizes=anchor_sizes, aspect_ratios=aspect_ratios)

''' Model '''
model = RetinaNet(backbone,
                  num_classes=1,
                  anchor_generator=anchor_generator)

def __getitem__():
    img = torch.rand(3, 256, 256)
    bboxes = []
    bboxes = [[15, 15, 20, 20]]*20
    bboxes = torch.FloatTensor(bboxes)
    labels = torch.LongTensor(np.ones(len(bboxes), dtype=int))
    targets = {'boxes':bboxes, 'labels':torch.LongTensor(labels)}
    return img, targets

def retinanet_collate_fn(batch_size=2):
    img_batch = []
    targets_batch = []
    for i in range(batch_size):
        img, targets = __getitem__()
        img_batch.append(img)
        targets_batch.append(targets)
    return img_batch, targets_batch
    

img_batch, targets_batch = retinanet_collate_fn(batch_size=2)

outputs = model(img_batch, targets_batch)
</code></pre>

ptrblck: <p>I think the error is raised in the unexpected target containing ones (so two classes):</p>
<pre><code class="lang-python">labels = torch.LongTensor(np.ones(len(bboxes), dtype=int))
</code></pre>
<p>while the model is initialized with <code>num_classes=1</code>. Change it to <code>num_classes=2</code> and it should work.</p>

asi212: <p>You are absolutely right. I assumed the background would have been class 0. That makes sense. Thank you.</p>
WeightedRandomSampler only sampling from one class

shabamee: <p>I’m running into an issue where WeightedRandomSampler is only sampling data from one class.</p>
<p>I’ve tried switching it for shuffle=True, and that works just fine, but I need to use weighted sampling since my data is imbalanced.<br>
Here’s the relevant code:</p>
<pre><code class="lang-auto">batch_size = 64

transform = torch.jit.script(nn.Sequential(
    RandAugment(num_ops=5, magnitude=10),
    NormalizeImage()
))

sample_count = [len(os.listdir(f"../../data/train_data/{i}")) for i in range(9)]
weights = torch.tensor(sample_count).reciprocal()

dataset = ImageFolder(root="../../data/train_data/", loader=read_image, transform=transform)
loader = DataLoader(dataset, batch_size=batch_size, sampler=WeightedRandomSampler(weights=weights, num_samples=batch_size), pin_memory=True)

data, labels = next(iter(loader))
data = data.cuda()
labels = labels.cuda()
print(labels)
</code></pre>
<p>The output of this is</p>
<pre><code class="lang-auto">tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], device='cuda:0')
</code></pre>
<p>No Bueno <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=10" title=":frowning:" class="emoji" alt=":frowning:"></p>
<p>If it’s helpful, my sample_count is</p>
<pre><code class="lang-auto">[1025, 964, 931, 922, 962, 909, 974, 916, 8306] 
</code></pre>
<p>and my weights are</p>
<pre><code class="lang-auto">tensor([0.0010, 0.0010, 0.0011, 0.0011, 0.0010, 0.0011, 0.0010, 0.0011, 0.0001])
</code></pre>
<p>Also, like I mentioned before, if I change my data loader to shuffle=True</p>
<pre><code class="lang-auto">loader = DataLoader(dataset, batch_size=batch_size, shuffle=True, pin_memory=True)
</code></pre>
<p>I get the expected outcome of</p>
<pre><code class="lang-auto">tensor([8, 3, 8, 8, 8, 8, 8, 2, 8, 8, 8, 2, 4, 8, 8, 8, 1, 8, 8, 3, 0, 8, 3, 2,
        1, 8, 8, 5, 8, 8, 8, 8, 8, 8, 7, 5, 8, 4, 3, 8, 8, 8, 8, 8, 3, 3, 4, 0,
        7, 8, 3, 2, 8, 8, 5, 5, 3, 8, 8, 8, 8, 4, 2, 5], device='cuda:0')
</code></pre>
<p>(For clarity, I have 9 classes numbered 0 through 8 each with their own subfolder in the train_data directory)<br>
Any help would be appreciated!</p>

ptrblck: <p><code>WeightedRandomSampler</code> expects a <code>weight</code> tensor assigning the weight values to each sample, not the class index.<br>
Have a look at <a href="https://discuss.pytorch.org/t/how-to-handle-imbalanced-classes/11264/2">this post</a> which gives you an example.</p>
Model Created With *list, .children(), and nn.sequential Produces Different Output Tensors

Stanley_C: <p>I’m currently trying to use a pretrained DenseNet in my model. I’m following this tutorial: <a>https://pytorch.org/hub/pytorch_vision_densenet/,</a> and it works well, with an input of [1,3,244,244], it returns a [1,1000] tensor, exactly as expected.<br>
However, currently I’m using this code to load a pretrained Densenet into my model, and use it as a “feature extraction” model. This is the code in the init function</p>
<pre><code class="lang-auto">base_model = torch.hub.load('pytorch/vision:v0.10.0', 'densenet121', pretrained=True)

self.base_model = nn.Sequential(*list(base_model.children())[:-1])
</code></pre>
<p>And it is being used like this in the forward function</p>
<pre><code class="lang-auto">x = self.base_model(x)

</code></pre>
<p>This however, taking the same input, returns a tensor of the size: ([1, 1024, 7, 7]). I can not figure out what is not working, I think it is due to the fact that DenseNet connects all the layers together, but I do not know how to get it to work in the same method. Any tips in how to use pretrained DenseNet in my own model?</p>

ptrblck: <p>Using the mentioned approach by re-wrapping modules into an <code>nn.Sequential</code> container might break for models which are using the functional API in their <code>forward</code> or any other approach which is not initializing and applying all layers in a sequential way.<br>
In your case, you are missing <a href="https://github.com/pytorch/vision/blob/b37c8a3ca4c9c626cdac763c6be697231665b0f8/torchvision/models/densenet.py#L214-L216">these functional API calls</a>, which is why the output shape differs.</p>

Stanley_C: <p>Thank you for your response! I guess next time I should take a look at the source code to see which layers are missing.</p>
How to sample from a softmax distribution

Ayush_Chakravarthy: <pre><code class="lang-auto">dist = torch.randn((100, 100))
softmax = nn.Softmax(dim=1)
out = softmax(dist)
</code></pre>
<p>This is all pretty standard and makes sense, but I am unable to figure out how to draw random samples from this distribution. In particular, I want to randomly find indices from dist which should be sampled following the softmax distribution.</p>
<p>Is there a way to do this?</p>

arman-yekkehkhani: <p>You can obtain the probability of sampling for each object by softmax, but you have to have the actual list of objects. Here, I simply assume the list comprises numbers from 0 to 100.</p>
<pre><code class="lang-auto">obj_list = list(range(100))
np.random.choice(obj_list, p=dist[0])
</code></pre>
<p>note: I  used <code>dist[0]</code> since <code>out</code> contains probability distribution for several independent PDFs. You can do similar for the rest.</p>

tom: <p>Note that torch.multinomial might be better if you want to stay in PyTorch.<br>
I would recommend to avoid mixing random number generators from different frameworks if you can.</p>
<p>(To me using two RNGs from two different frameworks falls under “being clever” and my rule of thumb here is “<a href="https://www.johndcook.com/blog/2016/01/29/random-number-generator-seed-mistakes/" rel="noopener nofollow ugc">don’t try to be clever with random number generation unless you absolutely know what you are doing</a>”… The linked anecdote is not immediately applicable, but it is a trap I have seen people fall into. The other day the big surprise was that PyTorch dataloaders didn’t initialize the numpy seed in workers, that has been fixed in recent PyTorch, but I’d venture that it is a good habit to stay with PyTorch unless you have to use something else.)</p>
<p>Best regards</p>
<p>Thomas</p>

Ayush_Chakravarthy: <p>Awesome, that worked for me, I used torch.multinomial and got the behaviour I was looking for.</p>
<p>Thanks!</p>
What are the stages for training a neural net for speech recognition?

hananelroe: <p>is that:</p>
<ol>
<li>creating a mel spectrogram</li>
<li>feed it to the neural net</li>
<li>backprop(and by the way what loss function should I use?)</li>
<li>repeat until the loss decreases</li>
</ol>
<p>or should I pass in batches or something? i’m super newbie and its my first project, so I dont know how to create an audio dataser and pass in batches… the tutorials that pytorch provides cant help me unfortunately… maybe can someone explain the steps in the comments or provide a good tutorial?</p>

tom: <p>There also are models working directly on waveforms, e.g. <a href="https://pytorch.org/audio/master/models.html#torchaudio.models.Wav2Letter" rel="noopener nofollow ugc">wav2letter</a>.<br>
For an even simpler entry, you might look at the <a href="https://pytorch.org/tutorials/intermediate/speech_command_recognition_with_torchaudio_tutorial.html" rel="noopener nofollow ugc">Speech Commands tutorial</a> where the waveform is translated into class predictions for keywords (a much simpler problem and relatively fast to train).</p>
<p>Best regards</p>
<p>Thomas</p>
Balancing Datasets

romhem: <p>I’m working on Detection of pneumonia project but I’m stuck at the first step on how to balance datasets which have images only. Can I get the code to balance the datasets using Pytorch . I have training , testing and validation sets which have images only.<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/5/e5d9fbacea994cddbe68874d0f9f6baf30bc98de.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/e5d9fbacea994cddbe68874d0f9f6baf30bc98de" title="Screenshot_2021-10-31-20-56-43-788_com.android.chrome"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/5/e5d9fbacea994cddbe68874d0f9f6baf30bc98de_2_250x500.jpeg" alt="Screenshot_2021-10-31-20-56-43-788_com.android.chrome" data-base62-sha1="wNmmMepXcWWjjPVVN9KH0uK2NDo" width="250" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/5/e5d9fbacea994cddbe68874d0f9f6baf30bc98de_2_250x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/e/5/e5d9fbacea994cddbe68874d0f9f6baf30bc98de_2_375x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/5/e5d9fbacea994cddbe68874d0f9f6baf30bc98de_2_500x1000.jpeg 2x" data-dominant-color="CCD4D7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot_2021-10-31-20-56-43-788_com.android.chrome</span><span class="informations">1080×2160 124 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

arman-yekkehkhani: <p>There is no general code for balancing a dataset. Yet there are methods for this purpose like data augmentation which means that you add artificial samples to your dataset so that each class has the same number of items. <a href="https://github.com/aleju/imgaug" rel="noopener nofollow ugc">imgaug</a> is a python library specifically developed for augmenting image data. You can use it to balance your datatset.</p>
Differentiable affine transformation on patches of images in pytorch

azadef: <p>I have a tensor of object bounding boxes, e.g. with the shape of [10,4] which correspond to a batch of images e.g. with shape [2,3,64,64] and transformation matrices for each object with shape [10,6] and a vector that defines which object index belongs to which image. I would like to apply the affine transformations on patches of the images and replace those patches after applying the transformations. I am doing this using a for loop now, but the way I am doing it is not differntiable (I get the in place operation error from pytorch). I wanted to know if there is a differntiable way to do this. e.g. via grid_sample?</p>
<p>Here is my current code:</p>
<pre><code class="lang-auto">for obj_num in range(obj_vecs.shape[0]): #batch_size
    im_id = obj_to_img[obj_num]
    x1, y1, x2, y2 = boxes_pred[obj_num]
    im_patch = img[im_id, :, x1:x2, y1:y2]
    im_patch = im_patch[None, :, :, :]
    img[im_id, :, x1:x2, y1:y2] = self.VITAE.stn(im_patch, theta_mean[obj_num], inverse=False)[0]
</code></pre>

azadef: <p>using im_batch.clone() instead of feeding it directly to STN solved the issue.</p>
Broadcast Tensor to Have Certain Size

rambo: <p>Hi!</p>
<p>I have a tensor variable <code>x</code>, that may come in sizes <code>(,)</code>, <code>(n,)</code>, <code>(1, n)</code> or <code>(m,n)</code>.<br>
Independently of the particular case, I would like to explicitly broadcast <code>x</code> to be two-dimensional.<br>
In other words, is there a single call like <code>y = x.broadcast_to_dim(2)</code> that will do nothing if <code>x</code> has size <code>(1,n)</code> or <code>(m,n)</code>, and will insert an axis on the left if <code>x</code> has size <code>(n,</code>) and two axes on the left if <code>x</code> has size <code>(,)</code>, making <code>x</code> of size <code>(1,n)</code> in both cases?</p>
<p>Thanks a lot!</p>

tom: <p>Does something like</p>
<pre><code class="lang-auto">if x.dim() == 1:
    x = x.epxand(1, n)
</code></pre>
<p>work?</p>
<p>Best regards</p>
<p>Tomas</p>

rambo: <p>Thanks!</p>
<p>I now did</p>
<pre><code class="lang-auto">if x.dim() &lt;= 1:
    x = x.expand(1, *(x.size() or [1]))
</code></pre>
<p>which does the job, but unfortunately must be part of my forward method.<br>
What kind of toll does this check have on performance?</p>

tom: <p>Not much, as it is metadata only. But you might try expand(1, -1) instead of the complicated thing or treat dim() == 0 and 1 separately.</p>
Extraction patches cnn

randino: <p>when I get patches, how do I get the (x, y) axes of the pixel ?</p>
<pre><code class="lang-auto">pixel=image.reshape(image.shape[0], image.shape[1]*image.shape[2])
</code></pre>
<p>how do I know the axes of the pixel (x, y) plz</p>

tom: <p>if <code>c, h, w = image.shape</code> and the index into the <code>pixel</code> tensor is <code>i, j</code>, then <code>y = j // w</code>, <code>x = j % w</code>.</p>
<p>Best regards</p>
<p>Thomas</p>

randino: <p><a class="mention" href="/u/tom">@tom</a> how do I know the i and j of the pixel?</p>

tom: <p>The instruction you posted “flattens” the image, so you transform the x y index into a a single j index. You can then use any j between 0 and h * w - 1.</p>

randino: <p><a class="mention" href="/u/tom">@tom</a> , I made a patch extraction in an original image but I want to know  i and j of the pixel 1 ?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/8/3896467acb1cef0880d353293d87fba7771aee41.png" alt="image" data-base62-sha1="84AHvocL60UVwdCWi84mNg80v4d" width="274" height="89"></p>
How to fix NaN in the Bert layer?

Odin_NI: <p>Hello everyone,</p>
<p>I train custom Bert model and observe the following problems.</p>
<p>These points are relevant for AMP case, training was fine for Float32 case.</p>
<ol>
<li>Attention mask starts to include -inf values and after dropout returns NaN for -inf.</li>
</ol>
<pre><code class="lang-auto">drop = nn.Dropout()
drop(torch.tensor([-np.inf, 1], dtype=torch.float))
</code></pre>
<ol start="2">
<li>If attention has -inf then matmul between -inf and zero return NaN</li>
</ol>
<p>How do I deal with those -inf values? Simply excluding or replacing them is not possible because it’s not a math operation.</p>

Odin_NI: <p>Is it right use torch.nan_to_num() after each operation?</p>

eqy: <p>Generally when there are NaNs or Inf values in a given training step, it is not possible to “recover” from the training step; a common practice is to simply reject or skip the weight update of that step to avoid propagating the issue to the model weights (so <code>nan_to_num</code> wouldn’t really help). However, if you are finding that the training is consistently producing NaN values, then there may be further tuning necessary in the training pipeline (e.g., gradient scaling, etc.)</p>

tom: <p>The -inf should not be multiplied with anything directly but only go into softmax. Typically attention dropout is applied afterwards (at least in bert/gpt).  Your application of dropout here is very nonstandard.</p>
<p>Best regards</p>
<p>Thomas</p>
Sum along axis with bins

robertwmacy: <p>I am trying to sum a tensor along a single dimension while grouping outputs by a separate label tensor.<br>
The functionality I’m looking for is similar to torch.bincount if it could be applied along a single dimension. Is there a torch function that has this functionality?<br>
E.g.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; a
tensor([[ 1,  2,  3],
        [ 4,  5,  6],
        [ 7,  8,  9],
        [10, 11, 12]])
&gt;&gt;&gt; b
tensor([0, 0, 1, 2])
# And I want to produce the following.
tensor([[ 5,  7,  9],
        [ 7,  8,  9],
        [10, 11, 12]])
</code></pre>
<p>Thanks for any help!</p>

tom: <p>You can use <code>Tensor.index_add</code> (or here its inplace version) for this:</p>
<pre><code class="lang-python">a = torch.arange(1, 13.).view(4, 3)
b = torch.tensor([0, 0, 1, 2])

c = torch.zeros(3, 3)
c.index_add_(0, b, a)
print(c)
</code></pre>
<p>Best regards</p>
<p>Thomas</p>
How to have vector of gradient for each parameter?

gemsanyou: <p>I know that we can’t call backward on vector Tensor without reducing it to a scalar. However, I also need the gradient to be a vector, with each column is the gradient of each column of the Loss with respect to the parameters, i.e.,</p>
<pre><code class="lang-auto">import torch
x = torch.tensor([4.0], requires_grad=True)
L1 = torch.sin(x) * torch.cos(x)
L2 = torch.sin(x**2) * torch.cos(x/2)
L = torch.stack([L1, L2])
L.backward()
print(x.grad)
</code></pre>
<p>I hope that x.grad equals [dL1/dx, dL2/dx].<br>
Is there any way I can do this in a simple manner, without copy and pasting to a temporary variable multiple times (as many as the length of the Loss Tensor)?</p>

KFrank: <p>Hi Kadek!</p>
<aside class="quote no-group" data-username="gemsanyou" data-post="1" data-topic="135458" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gemsanyou/48/34969_2.png" class="avatar"> gemsanyou:</div>
<blockquote>
<p>However, I also need the gradient to be a vector, with each column is the gradient of each column of the Loss with respect to the parameters</p>
</blockquote>
</aside>
<p>Does autograd’s <a href="https://pytorch.org/docs/1.10.0/generated/torch.autograd.functional.jacobian.html" rel="noopener nofollow ugc">jacobian()</a> do what you want?</p>
<p>Best.</p>
<p>K. Frank</p>

gemsanyou: <p>Yes I think for the above toy example, it turns out <a href="https://pytorch.org/docs/1.10.0/generated/torch.autograd.functional.jacobian.html" rel="noopener nofollow ugc">Jacobian</a> is enough. However, for my real use-case which is for an nn.Module, it’s not enough. As the thread here suggest <a href="https://discuss.pytorch.org/t/get-gradient-and-jacobian-wrt-the-parameters/98240" class="inline-onebox">Get gradient and Jacobian wrt the parameters</a> , we have to make a new function that takes the model params as the input, because <a href="https://pytorch.org/docs/1.10.0/generated/torch.autograd.functional.jacobian.html" rel="noopener nofollow ugc">Jacobian</a> computed the gradient w.r.t. input. I will try it first. Thank you</p>
AttributeError: 'Example' object has no attribute '<field_name>'

glitch: <p>In an effort to build a Dataset using <code>torchtext</code><br>
I am getting this error:</p>
<blockquote>
<p>AttributeError: ‘Example’ object has no attribute ‘tweet’</p>
</blockquote>
<p>Code to build the dataset looks like:</p>
<blockquote>
<p>tweetField=torchtext.data.Field(sequential=True, tokenize=clean_tweets)<br>
sentimentField=torchtext.data.LabelField(sequential=False)</p>
<p>trainData, valData, testData= torchtext.data.TabularDataset.splits(<br>
path=‘./’,train=‘train.csv’,validation=‘valid.csv’,test=‘test.csv’,format=‘csv’,<br>
skip_header=True,fields=[(‘tweet’,tweetField),(‘sentiment’,sentimentField)]<br>
)<br>
tweetField.build_vocab(trainData,vectors=torchtext.vocab.GloVe(name=“6B”,dim=50))</p>
<p>sentimentField.build_vocab(trainData)</p>
</blockquote>
<p>The Dataset inside the <code>.csv</code> looks like:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/7/97b79471618e523d98a1a1549e8420c1245892f9.png" alt="image" data-base62-sha1="lE9pNlxcfdDhZYx8HqceQAYIRKx" width="319" height="417"></p>
<p>Please Help !</p>

woqingdoua: <p>I am sure whether it is a version problem.<br>
I fixed this problem by<br>
‘tweetField.build_vocab(trainData.fields,vectors=torchtext.vocab.GloVe(name=“6B”,dim=50))’</p>
Is there any way to get IoU values from Mask R-CNN per prediction?

Garfish: <p>Hi All,</p>
<p>I would be very grateful if someone could point me in the right direction on this, because I’m having trouble understanding how I can achieve what I need.</p>
<p>I’m using a slightly modified version of this notebook:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/github/pytorch/tutorials/blob/gh-pages/_downloads/torchvision_finetuning_instance_segmentation.ipynb">
  <header class="source">
      <img src="https://colab.research.google.com/img/favicon.ico?vrz=colab-20211028-060058-RC00_406083766" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/github/pytorch/tutorials/blob/gh-pages/_downloads/torchvision_finetuning_instance_segmentation.ipynb" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="260" height="260">

<h3><a href="https://colab.research.google.com/github/pytorch/tutorials/blob/gh-pages/_downloads/torchvision_finetuning_instance_segmentation.ipynb" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I have modified it to use my own dataset. Another modification I have made is to this block of code :</p>
<pre><code class="lang-auto"># pick one image from the test set
img, _ = dataset_test[0]
# put the model in evaluation mode
model.eval()
with torch.no_grad():
    prediction = model([img.to(device)])
</code></pre>
<p>To:</p>
<pre><code class="lang-auto">### To iterate through dataset_test:
#put model in prediction mode

model.eval()

#function to iterate through test set and return a list of predictions

def prediction_iterator():
  predictions = []
  for img, _ in dataset_test:
    with torch.no_grad():
      prediction = model([img.to(device)])
    predictions.append(prediction)
  return predictions

prediction_list = prediction_iterator()
</code></pre>
<p>This change is just so that I can make a series of predictions on the test set. I’m not totally sure if this is the best way to go about this, but it seems to work as far as I can see.</p>
<p>My issue is now that I want to get the IoUs for each prediction, as I want to compute True/False positive, True/False negative confusion matrices, Precision and Recall, F1, etc.</p>
<p>I realise the evaluate function spits out average precision and recall, but as I say I want to do a bit more than that.</p>
<p>I did use torch.save and then torch.load to save out the predictions (prediction_list) as a .csv, but the result was just that all tensors were listed as a string in one column. I considered using RegEx to split this string up, but I would really prefer if I could get it in the right format to begin with so I can scale this easily.</p>
<p>Apologies if I’m not explaining very well or if this is an obvious problem - if there are any more details you need, please let me know.</p>
How to get the output of a specific input through all layers of the model?

Sarah: <p>Hello,<br>
I have got an autoencoder model, and I want to be able to get the output of each input of every batch in every layer of the module and store it to compare it with something else.<br>
How can I do that<br>
I know that I can use hooks to get the activation of the intermediate layers in my module, now I need to use it for every input!!</p>

my3bikaht: <p>There’s another way, without hooks. You can directly adjust forward() method of your model’s class and return intermediate outputs with the actual output as a dict, for example.</p>
<p>As for hooks, take a look here: <a href="https://blog.paperspace.com/pytorch-hooks-gradient-clipping-debugging/" class="inline-onebox" rel="noopener nofollow ugc">Debugging and Visualisation in PyTorch using Hooks</a></p>

Sarah: <p>Thank you, but can you elaborate more on how to adjust the forward method in the model to get the output of each layer for every input.<br>
I am new to deep learning and PyTorch</p>

my3bikaht: <p>(this option ain’t pretty, it’s more like a hack). For example here’s a forward method of resnet class:</p>
<pre><code class="lang-auto">    def _forward_impl(self, x):
        #out = {}
        x = self.conv1(x)
        #out['conv1'] = x.detach()
        x = self.bn1(x)
        x = self.relu(x)
        x = self.maxpool(x)
        #out['maxpool'] = x.detach()
        x = self.layer1(x)
        #out['layer1'] = x.detach()
        x = self.layer2(x)
        #out['layer2'] = x.detach()
        x = self.layer3(x)
        #out['layer3'] = x.detach()
        x = self.layer4(x)
        #out['layer4'] = x.detach()
        x = self.avgpool(x)
        #out['avgpool'] = x.detach()
        x = torch.flatten(x, 1)
        x = self.fc(x)
        return x, out #instead of return x


</code></pre>
<p>commented lines will create dict with copies of intermediate values, detached from graph</p>

Sarah: <p>Thank you, this is really helpful.<br>
I tried it and it worked.</p>
Why my accuracy suddenly decrease?

jhp: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/f/cf1b659da8474613deac269376e7afcb45094815.png" data-download-href="https://discuss.pytorch.org/uploads/default/cf1b659da8474613deac269376e7afcb45094815" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cf1b659da8474613deac269376e7afcb45094815_2_690x221.png" alt="image" data-base62-sha1="ty9x6SXVKna5okQNzAVzldiIU1n" width="690" height="221" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cf1b659da8474613deac269376e7afcb45094815_2_690x221.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cf1b659da8474613deac269376e7afcb45094815_2_1035x331.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cf1b659da8474613deac269376e7afcb45094815_2_1380x442.png 2x" data-dominant-color="FEFFFF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2431×779 31.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
As shown in above learning curve, suddenly MAP drops at certain epoch.<br>
Also I found that training loss is gradually increasing at this moment but no gradient exploding since I use Adam optimizer and grad cliping.</p>
<p>What makes this kind of situation happen? Does simple generalization tricks (i.e increasing weight decay) can be the solution?</p>

J_Johnson: <p>Is that on test data or train data? If test data, your model is likely overfitting to the train data.</p>