akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Pass Indices of 'ones' in target to the Loss Function for Multi Label Prediction

smpotdar: <p>Consider the following situation for Multi Label Classification.</p>
<pre><code class="lang-auto">output = torch.tensor([[0.3416, 0.6336, 0.3775, 0.2556, 0.6288]])
target = torch.tensor([[1,0,1,0,1]])
target_indices = torch.tensor([[0,2,4]])
</code></pre>
<p>Is there a loss function where in I can calculate the CrossEntropyLoss in the following way:<br>
<code>loss(output, target_indices)</code> ?</p>

ptrblck: <p>For a multi-label classification, you could use <code>nn.BCEWithLogitsLoss</code> with your <code>target</code> tensor directly (after casting it to a <code>FloatTensor</code>):</p>
<pre><code class="lang-python">output = torch.tensor([[0.3416, 0.6336, 0.3775, 0.2556, 0.6288]])
target = torch.tensor([[1,0,1,0,1]]).float()

criterion = nn.BCEWithLogitsLoss()
loss = criterion(output, target)
</code></pre>

smpotdar: <p>Thanks for replying. But I want to pass the target indices, that is a constraint as I have a sparse matrix representation of binary vector and I wish to use that. Is there anyway to do it?</p>

ptrblck: <p>I don’t think you can use the index  approach with the current <code>nn.BCE(WithLogits)Loss</code> implementation, since the target is a <code>FloatTensor</code> and thus contains the target for each data sample.</p>
<p>Would you be able to create the right target using <code>scatter</code>?</p>
<pre><code class="lang-python">output = torch.tensor([[0.3416, 0.6336, 0.3775, 0.2556, 0.6288]])
target = torch.zeros_like(output)
target_indices = torch.tensor([[0,2,4]])
target.scatter_(1, target_indices, 1)
</code></pre>

smpotdar: <p>Okay. Thanks. I’ll try that.</p>
Can someone debug my implementation of Policy Gradients (REINFORCE) for playing Atari breakout?

deadbeef: <p>I’m learning about RL, and I’m struggling to get the simple REINFORCE algorithm working for playing Atari Breakout.</p>
<p>Self-contained code here: <a href="https://colab.research.google.com/drive/1x1vM8l1kXT3SuOeOR1E_a9I943_t5GcK" rel="nofollow noopener">https://colab.research.google.com/drive/1x1vM8l1kXT3SuOeOR1E_a9I943_t5GcK</a></p>
<p>My problem is that the algorithm overfits one action, meaning that one action is played nearly every step (probabilistic due to the sampling operation when choosing an action). Therefore it doesn’t learn to play properly.</p>
<p>I must have an implementation error somewhere because as I understand it actions which lead to negative reward should be relatively discouraged, but they’re not.</p>
<p>Is my implementation of the loss correct? My understanding is that you only backprop through the neuron that codes the action you actually selected, but the gradient gets distributed to all other neurons by the softmax?</p>
<p>If someone can figure out what I did wrong I’d be very grateful. It would also be great to know how you went about figuring out what was wrong.</p>
How to make changes in a mac unit of a kernel in cnn

Anmol_Gautam: <p>we want to access a mac unit in a kernel of each layer of a pretrained network while prediction to perform changes in it. we are new to pytorch and are having issues in implementing the same. we also did post training quantization of weights and activation to 8bits. Our defined cnn is as follows:</p>
<p>class Simplenet(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Simplenet, self).<strong>init</strong>()<br>
self.conv1 = nn.Conv2d(3, 6, 5)<br>
self.relu_conv1 = nn.ReLU()<br>
self.pool1 = nn.MaxPool2d(2, 2)<br>
self.conv2 = nn.Conv2d(6, 16, 5)<br>
self.relu_conv2 = nn.ReLU()<br>
self.pool2 = nn.MaxPool2d(2, 2)<br>
self.fc1 = nn.Linear(16 * 5 * 5, 120)<br>
self.relu_fc1 = nn.ReLU()<br>
self.fc2 = nn.Linear(120, 84)<br>
self.relu_fc2 = nn.ReLU()<br>
self.fc3 = nn.Linear(84, 10)</p>
<pre><code>def forward(self, x):
    x = self.pool1(self.relu_conv1(self.conv1(x)))
    x = self.pool2(self.relu_conv2(self.conv2(x)))
    x = x.view(-1, 16 * 5 * 5)
    x = self.relu_fc1(self.fc1(x))
    x = self.relu_fc2(self.fc2(x))
    x = self.fc3(x)
    return F.log_softmax(x, dim=1)</code></pre>

ptrblck: <p>It depends a bit on what you would like to manipulate.<br>
If you want to change the underlying computation (multiplication of kernel and patch + bias), you would most likely have to write your own convolution (called e.g. in <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/Convolution.cpp" rel="nofollow noopener">Convolution.cpp</a>).</p>
<p>If you want to run some experiments first and make sure your method works correctly, I would recommend to use <code>.unfold</code> to create patches of your input activation, and apply your custom operations in a similar manner as your conv layer would do.<br>
This approach would be slower than the optimized conv code, but might be good enough for your experiments.</p>
Torchvision 0.3, deeplab models not taking in batch size of 1

gtm2122: <p>Hi, I recently installed torchvision 0.3 on linux from source</p>
<p>my below code does not work during training mode but works during eval mode. It gives the error<br>
“Expected more than 1 value per channel when training, got input size torch.Size([1,256,1,1])”</p>
<p>import torchvision.models.segmentation as seg_model<br>
import torch</p>
<p>img = torch.zeros((1,1,300,300))<br>
m1 = seg_model.deeplabv3_resnet50(num_classes=2,pretrained=False)<br>
m2 = seg_model.deeplabv3_resnet101(num_classes=2,pretrained=False)</p>
<p>o = m1.cuda()(img.cuda()) # Error<br>
o = m2.cuda()(img.cuda()) # Also error</p>
<p>This doesn’t happen during eval mode. It only happens when the 0th dim is less than 2 during train mode. I was just wondering to train this do I just keep minimum batch size as 2 or does the model expect something else like like some other information along with the input tensor image ? I am using the code from <a href="https://github.com/pytorch/vision/blob/v0.3.0/references/segmentation/train.py" rel="nofollow noopener">https://github.com/pytorch/vision/blob/v0.3.0/references/segmentation/train.py</a> as reference</p>

ptrblck: <p>Most likely you’ll see this error in a batchnorm layer, which cannot calculate the running batch statistics from a single value in each channel.<br>
It’s similar to the <code>Inception3()</code> from <code>torchvision</code>, which also expects at least two samples in a batch during training.</p>
<p>Would it be possible to provide more than a single sample or are you running out of memory?</p>

gtm2122: <p>Yes, that is correct. The traceback ends at the definition for the functional of batchnorm. I will just set the minimum batch size as 2 in that case. Thank you!</p>

Dane_Mitrev: <p>In my case, I am running out of memory with batch 2. Is is possible to somehow bypass this limitation which expects at least two samples for BN layers?</p>

ptrblck: <p>You could try to use <code>torch.utils.checkpointing</code> to trade compute for memory.<br>
However, you even if it’s working with a batch size of 2, the running estimates might be quite off due to the small batch size. You might need to tune the <code>momentum</code> or rather switch to another normalization layer, e.g. <code>InstanceNorm</code>.</p>
Dynamically weighing Binary Cross Entropy with Logits based on batch

dhecloud: <p>is it possible to pass your own weights into the loss function during each batch such that it affects how the loss is calculated? Note: I don’t want to use the same weights for every batch (which is the <code>weight</code> argument), I want to weigh the loss from each output neuron dynamically based on a rule from the ground truth labels.</p>
<p>So for example, a 2 neuron final layer can have loss weighing <code>[[1,0.5],[0,0.5]]</code> for the first batch of 2, and the second batch of 2 can have <code>[[0,0],[1,1]]</code>… and so on.<br>
and the code might look something like this in the training loop:</p>
<pre><code class="lang-auto">target = torch.randn([2, 2], dtype=torch.float32) #placeholder, will be taken from data loader
weights = create_weights_from_target(target)
output = model(inputs)
criterion = torch.nn.BCEWithLogitsLoss()
criterion(output, target, weights=weights) #extra arg to specify weights for each batch
</code></pre>
<p>Short of reimplementing the whole loss function, is there a better way to do this?<br>
Please let me know if you need me to clarify anything. Thank you very much!!!</p>

ptrblck: <p>If you use the functional API, you would avoid recreating the instance of the loss function.<br>
Alternatively, you could pass <code>reduction='none'</code> to the criterion and multiply the loss output manually with your weights. After the multiplication you could take the mean or normalize the loss values as you wish.</p>
Differentiating where operator

Bruno1: <p>Hi,</p>
<p>I’m trying to make the result of the output of the piece of code below differentiable with respect to epsilon. I know that <code>where</code> operator is not differentiable wrt to its condition.</p>
<p>Is there any way to make the output differentiable wrt. to epsilon ?</p>
<pre><code class="lang-auto">eps = nn.Parameter(torch.ones(1,))
nn.init.constant_(eps,0.5)

x = torch.rand(5,5)
y = torch.zeros(5,5)
z = torch.ones(5,5)

out = torch.where(x &gt; eps, z, y)
</code></pre>
<p>Thank you</p>

tom: <p>The non-differentiability is from the discontinuity of the function. So you would have to <em>relax</em> (smoothen) the expression.<br>
For non-NaN/inf, <code>where(x &gt; eps, z, y)</code> is <code>(x - eps &gt; 0) * (z - y) + y</code> which you could change to  <code>torch.sigmoid(x - eps) * T) * (z - y) + y</code> for some tuneable float  t.</p>
<p>Best regards</p>
<p>Thomas</p>

Bruno1: <p>I see, thanks a lot  it really helps !</p>
How to calculate running loss/training loss while training a CNN model

anonym: <p>Can anyone please tell me if there is a specific method to calculate the running loss/training loss? I came a few formulae while looking for it</p>
<ol>
<li>train_loss = train_loss + ((1 / (batch + 1)) * (loss.data - train_loss))</li>
<li>train_loss += loss.item().</li>
</ol>
<p>Below is the code while using it:<br>
<span class="hashtag">#train</span> model<br>
def train(n_epochs,model,loader,optimizer,criterion,save_path):</p>
<pre><code>for epoch in range(n_epochs):
    train_loss = 0

    net.train()
    for batch, (data,target) in enumerate(loaders['train']):
        
        target = target.view(target.size(0),-1)
        target = Variable(target)

        optimizer.zero_grad()

        outputs = net(data)
        
        loss = criterion(outputs,target)
        loss.backward()
        optimizer.step()
        #calculate training loss
        train_loss = train_loss + ((1 / (batch + 1)) * (loss.data - train_loss))

        #print results
        if batch % 100 == 0:
            print("Epoch: {}, Batch: {}, Training Loss: {}".format(epoch+1, batch, train_loss/1000))

print("Finished Training") 
</code></pre>
<p>I want to know if there is a particular way to calculate it?</p>

ptrblck: <p>I like the approach used in the <a href="https://github.com/pytorch/examples/blob/1de2ff9338bacaaffa123d03ce53d7522d5dcc2e/imagenet/main.py#L287" rel="nofollow noopener">ImageNet example</a> using an <code>AverageMeter</code> and updating it on the fly.</p>

Ludwig_Friborg: <p>I think its common to calculate it like this:</p>
<pre><code class="lang-auto">batch_loss += loss.item() * batch_size
</code></pre>
<p>Then at the end of the epoch we divide the number with the number of steps.</p>
<p>We multiply it with the batch size since <code>loss.item()</code> returns the average loss for each sample within the batch, so to get per sample overall we have to correct for that.</p>
Input and output data are not same features of lstm is possible?

pahome: <p>Taking stock price prediction by LSTM for example, Can I output prices but input has no prices related features?</p>
<p>Ex:<br>
input features are basic like timestamps, stock volumes, stock id.<br>
output is the stock price</p>
<p>I don’t know if that make sense in time-series prediction or not. Can anyone explain  is that possible?</p>
Custom Dataset in Torchtext with variable number of Fields

leewee: <p>A custom dataset in Torchtext where each data is (text1, text2, and label) can be written something like below:</p>
<pre><code class="lang-auto">class CustomDataset(ttdata.Dataset):
    'Reads a JSON file. Each line has text1, text2, and the associated label.'

    def __init__(self, path, max_seq_length=100, min_seq_length=3, add_eos=True):

        text1_field = Field(lower=True, tokenize='spacy')
        text2_field = Field(lower=True, tokenize='spacy')
        label_field = ttdata.Field(sequential=False, use_vocab=False)
        fields = [('text1', text1_field), ('text2', text2_field), ('label', label_field)]

        examples = []

        with open(path, 'r') as f:
            for item in f:
                item = json.loads(item)
                text1, text2, label = item['text1'], item['text2'], item['label']
                examples.append(ttdata.Example.fromlist([text1, text2, label], fields))


        super(CustomDataset, self).__init__(examples, fields)
</code></pre>
<p>But in my case rather than text1 and text2, I have 2 lists of strings of variable length (text1 = [“some text”, “some text”, …] and text2 = [“yet some text”, “yet some text”, …])and a label for the 2 lists.</p>
<p>How can I write a dataset class in this case? I think the main problem is now I cannot define fixed number of Field objects. Can NestedField be used in some way?</p>
How to handle variable length sentences with lstm?

Rafael_R: <p>Hi,</p>
<p>Is there anything wrong with the below code? the sentence is already padded to a maximum length.</p>
<pre><code class="lang-auto">         embedded = self.embedding(sentence)
        # Each batch has the same maxlen, how to make data loader with custom maxlen?
        input_lengths = [sentence.shape[1]]* sentence.shape[0] 
        packed = torch.nn.utils.rnn.pack_padded_sequence(embedded, input_lengths, batch_first=True)
        output, hidden = self.text_LSTM(packed, None)
        output, _ = torch.nn.utils.rnn.pad_packed_sequence(output, batch_first=True)
</code></pre>
<p>However my model is not learning with the final representation to:</p>
<blockquote>
<p>output[:,-1,:]</p>
</blockquote>
<p>extracted from the lstm, i have fixed the input lengths already (since sentences are padded).</p>
Loading a pretrained model without freezing layers? Is it wrong to do?

alx: <p>Hi guys, I’m using a resnet on a dataset with pretrained = True.</p>
<p>When I freeze the last layer, my model does not learn and plateau’s at 74% acc.</p>
<p>When I simply load the model without freezing any layer I get 81% acc.</p>
<p>Is it a wrong method to use?</p>

ptrblck: <p>Usually you would freeze the first layers and finetune the last one.<br>
What’s your use case to only freeze the last layer?</p>

alx: <p>Hey <a class="mention" href="/u/ptrblck">@ptrblck</a>, sorry for the late reply, you’re right.</p>
<p>I’m simply trying to use<br>
<code>model = models.resnet18(pretrained=True)</code></p>
<p>To classify images. When I freeze the layers my acc starts at 69%, when I don’t it starts at 74%.</p>
How to create mask with pair (begin index and end index)

l4zyf9x: <p>Hi all, I have tensor 2D data.size=[4,8]. For each row in data, i want create a mask from begin_index to end_index, eg:</p>
<pre><code class="lang-auto">pair_index = torch.LongTensor([[0,2],[2,7],[1,3],[0,5]])
tensor([[0, 2],
        [2, 7],
        [1, 3],
        [0, 5]])
</code></pre>
<p>So my expected mask output will be:</p>
<pre><code class="lang-auto">tensor([[1, 1, 0, 0, 0, 0, 0, 0],
        [0, 0, 1, 1, 1, 1, 1, 0],
        [0, 1, 1, 0, 0, 0, 0, 0],
        [1, 1, 1, 1, 1, 0, 0, 0]])
</code></pre>

KFrank: <p>Hello Trinh!</p>
<aside class="quote no-group" data-username="l4zyf9x" data-post="1" data-topic="59646" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/c0e974/48.png" class="avatar"> l4zyf9x:</div>
<blockquote>
<p>Hi all, I have tensor 2D data.size=[4,8]. For each row in data, i want create a mask from begin_index to end_index, eg:</p>
<pre><code class="lang-auto">pair_index = torch.LongTensor([[0,2],[2,7],[1,3],[0,5]])
tensor([[0, 2],
        [2, 7],
        [1, 3],
        [0, 5]])
</code></pre>
<p>So my expected mask output will be:</p>
<pre><code class="lang-auto">tensor([[1, 1, 0, 0, 0, 0, 0, 0],
        [0, 0, 1, 1, 1, 1, 1, 0],
        [0, 1, 1, 0, 0, 0, 0, 0],
        [1, 1, 1, 1, 1, 0, 0, 0]])
</code></pre>
</blockquote>
</aside>
<p>There might be a cleaner way, but here’s the best I could do:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
print (torch.__version__)
pair_index = torch.LongTensor([[0,2],[2,7],[1,3],[0,5]])
cols = torch.LongTensor (range (8)).repeat (4, 1)
beg = pair_index[:,0].unsqueeze (1).repeat (1, 8)
end = pair_index[:,1].unsqueeze (1).repeat (1, 8)
mask = cols.ge (beg) &amp; cols.lt (end)
print (mask)
</code></pre>
<p>Here’s the output:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
0.3.0b0+591e73e
&gt;&gt;&gt; pair_index = torch.LongTensor([[0,2],[2,7],[1,3],[0,5]])
&gt;&gt;&gt; cols = torch.LongTensor (range (8)).repeat (4, 1)
&gt;&gt;&gt; beg = pair_index[:,0].unsqueeze (1).repeat (1, 8)
&gt;&gt;&gt; end = pair_index[:,1].unsqueeze (1).repeat (1, 8)
&gt;&gt;&gt; mask = cols.ge (beg) &amp; cols.lt (end)
&gt;&gt;&gt; print (mask)

    1     1     0     0     0     0     0     0
    0     0     1     1     1     1     1     0
    0     1     1     0     0     0     0     0
    1     1     1     1     1     0     0     0
[torch.ByteTensor of size 4x8]
</code></pre>
<p>(It’s easy and straightforward to do this with a loop, but I always<br>
assume these questions are about using tensor operations and<br>
avoiding loops.)</p>
<p>Good luck.</p>
<p>K. Frank</p>

l4zyf9x: <p>Thank for your response. This is what exactly i need</p>
Splitting output and backpropagating separately

hash-ir: <p>I have a problem where I am regressing over a vector of length 8. The first four values of the vector represent one output and the next four represent another output. Since these are two different quantities with different scales, I want to split my output into two, such that the loss is calculated individually for each output quantity and then summed up when backpropagated.</p>
<p>Can I write something like this in the forward function</p>
<pre><code class="lang-auto">def forward(self, x):
    x = self.layer1(x) # feature computation
    ...
    x = self.fc(x) #last layer (fully connected)
    y, z = x.split(2)
    return y, z
</code></pre>
<p>I don’t know if it will be handled well by the loss function or not. And, does it also require two <code>nn.Linear</code> layers at the end in parallel?</p>

tom: <p>You can just return a tuple (but you’d want <code>x.chunk(2, dim=-1)</code> or somesuch to split into two chunks. One fc probably works <em>if your optimizer considers tensor entries individually</em> (e.g. Adam is OK, but LAMB will consider parameters as large tensors and the “size” of the change is proportional to the “length” (both Frobenius norm) of the parameters as vectors).<br>
It doesn’t make much much sense to do the backpropagation completely separate, it is usually much better to scale the loss terms to some sort of equilibrium (either of the losses or the gradients or so) and then backpropagate in one go.</p>
<p>Best regards</p>
<p>Thomas</p>

Paulo_Mann: <p>Also, try using the tensorboard to inspect the backpropagation. You will be more precise about the result being calculated correctly.</p>
Is torch.Graph executable?

sublee: <p>Hi, I’m interested in the optimization of a PyTorch JIT graph. For this purpose, I want to find a way to execute an existing <code>torch.Graph</code> directly. Is there any official or unofficial way for this purpose?</p>

sublee: <p>I found a private API <code>torch._C._create_function_from_graph(name, graph)</code> which reconstructs a <code>torch.Function</code> from an existing JIT graph.</p>

tom: <p>The usual thing to do is to transform the graph. This is what the various passes in the JIT do and you can also register custom passes. For example, the difference between <code>fn.graph</code> and <code>fn.graph_for(*inputs)</code> comes from the passes executed after specialization of the input arguments.</p>
<p>Best regards</p>
<p>Thomas</p>