akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
No padding when using torchtext.data.Field with a TabularDataset

username: ribhu
cooked: <p>I have been trying to implement the pytorch seq-to-seq transformer tutorial (<code>https://pytorch.org/tutorials/beginner/transformer_tutorial.html</code>). But I wanted to use my own data, and hence tried to load the csvs using the TabularDatasets while using the same Field that was defined in the tutorial.</p>
<p>While doing the batchify() step I encountered the error</p>
<blockquote>
<p>ValueError: expected sequence of length 13 at dim 1 (got 6)</p>
</blockquote>
<p>On looking deeper I found that the sequences were not getting padded nor was it being tokenized properly (no eos, pad or init tokens). I am a little confused as I specified the fixed length as well when defining the field but I keep getting the same error. It would be great if someone can point out where exactly I am going wrong or if I should use a different approach. The code is below:</p>
<pre><code class="lang-auto"># Defining the Field
TEXT = torchtext.data.Field(tokenize=get_tokenizer("basic_english"),
                            init_token='&lt;sos&gt;',
                            eos_token='&lt;eos&gt;',
                            pad_token='&lt;pad&gt;',
                            lower=True,
                            fix_length=20)

# create tuples representing the columns
fields = [
  (None, None), # ignore first column
  (None, None), # ignore id column
  ('Exp_text', TEXT),
]

# load the dataset from csvs
train_ds, val_ds = torchtext.data.TabularDataset.splits(
   path = './',
   train = 'lm_data_train.csv',
   validation = 'lm_data_val.csv',
   format = 'csv',
   fields = fields,
   skip_header = True
)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# Create batches
def batchify(data, bsz):
    data = TEXT.numericalize([d.Exp_text for d in vars(data)['examples']])
    # Divide the dataset into bsz parts.
    nbatch = data.size(0) // bsz
    # Trim off any extra elements that wouldn't cleanly fit (remainders).
    data = data.narrow(0, 0, nbatch * bsz)
    # Evenly divide the data across the bsz batches.
    data = data.view(bsz, -1).t().contiguous()
    return data.to(device)

batch_size = 20
eval_batch_size = 10
train_data = batchify(train_ds, batch_size)
val_data = batchify(val_ds, eval_batch_size)
</code></pre>
Dataset map-style vs iterable-style

username: aerinykim
cooked: <p>A map-style dataset in Pytorch has the  <code>__getitem__()</code>  and  <code>__len__()</code>  and iterable-style datasets has  <code>__iter__()</code>  protocol. If we use map-style, we can access the data with  <code>dataset[idx]</code>  which is great, however with the iterable dataset we can’t.</p>
<p>My question is why this distinction was necessary? What makes the data random read so expensive or even improbable?</p>

username: ptrblck
cooked: <p>I understood the main difference between these datasets, that the <code>IterableDataset</code> provides a clean way to yield data from e.g. a stream, i.e. where the length of the dataset is unknown or cannot be simply calculated. Inside the <code>__iter__</code> method you would thus have to make sure to exit the iteration at some point, e.g. if your data stream is empty.</p>

username: shartzog
cooked: <p>It’s pretty much the same distinction as a generator (iterable-style) and a list (map-style), correct?</p>

username: ptrblck
cooked: <p>That’s my understanding, yes.</p>
How to use transform.fivecrop(size) in custom dataset

username: Mohsin_Ullah
cooked: <p>When I use transforms.FiveCrop(20) in the custom dataset, Then pytorch dataloader gives me (128,5,3,20,20) shapes of batches where 128 is the batchsize, 5 comes from croping an image at 5 different places and remaining is channel, width and height.<br>
What I want is that I get only 128 images in the batch.</p>

username: ptrblck
cooked: <p>In that case you could:</p>
<ul>
<li>manually crop at a specific location only</li>
<li>use <code>FiveCrop</code> and just return a single location (randomly selected or fixed)</li>
<li>reduce the batch size by a factor of ~5 and collapse the crops into the batch size to get ~128 samples per batch</li>
</ul>
Loop will decrease the utilization of the GPU?

username: 111357
cooked: <p>I have a loop in my DeepLearning Pipeline’s forward part to normalize the intermediate result.</p>
<p>Do it run on CPU and decrease the utilization of the GPU?</p>
<p>some snippet as follow:</p>
<pre><code class="lang-auto">def forward(self):
    ...
    for b in range(batch_size):
        self.points[b] = self.unit_cube(self.points[b])
    ....
</code></pre>

username: ptrblck
cooked: <p>The Python script including the loops will run on the CPU.<br>
Depending on which device <code>self.points</code> is stored and what is used inside <code>self.unit_cube</code> the GPU might be used. In this case a kernel will be executed in each iteration.</p>
Cross Validation with Lightning

username: gilbertocunha
cooked: <p>Hi! I have implemented my PyTorch Lightning model and I was wondering if it was possible to perform cross validation.<br>
From my understanding, both the train_dataloader and val_dataloader methods are only called in the very beginning of the .fit() trainer method, so I wanted to know if I could somehow change the folders at each epoch.</p>
<p>Thanks in advance</p>

username: ptrblck
cooked: <p>CC <a class="mention" href="/u/williamfalcon">@williamFalcon</a> as a Lightning dev <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Gather command for a 4-D tensor

username: arturbeg
cooked: <p>Hi everyone,</p>
<p>I have a lookup/indexing tensor of size (2, 4). And I also have an <strong>input</strong> tensor of size (2, 4, 2, 4).</p>
<p>For example, the lookup/indexing tensor is the following: <strong>index</strong> = [ [0, 0, 0, 2], [0, 0, 1, 2] ]</p>
<p>My goal is to generate an output tensor of the same size as the input tensor (2, 4, 2, 4) by gathering according to the lookup indices along the dim = 1. To illustrate, in this case I want</p>
<pre><code class="lang-auto">output[0] = torch.cat( [input[0, index[0,0], :, :], input[0, index[0,1], :, :], input[0, index[0,2], :, :], input[0, index[0,3], :, :], dim=1)
output[1] = torch.cat( [input[1, index[1,0], :, :], input[1, index[1,1], :, :], input[1, index[1,2], :, :], input[1, index[1,3], :, :], dim=1)
</code></pre>
<p>which should yield (for the particular <strong>index</strong> mentioned above):</p>
<pre><code class="lang-auto">output[0] = torch.cat( [input[0, 0, :, :], input[0, 0, :, :], input[0, 0, :, :], input[0, 2, :, :], dim=1)
output[1] = torch.cat( [input[1, 0, :, :], input[1, 0, :, :], input[1, 1, :, :], input[1, 2, :, :], dim=1)
</code></pre>
<p>Is there a neat way to do this using torch.gather()?</p>
<p>Thank you:)</p>

username: ptrblck
cooked: <p>You could directly index the tensor without using <code>gather</code>.<br>
I’ve changed the shape a bit (used <code>stack</code> instead of <code>cat</code> and created a final <code>output</code> tensor) for a better comparison:</p>
<pre><code class="lang-python">index = torch.tensor([[0, 0, 0, 2], [0, 0, 1, 2]])
input = torch.randn(2, 3, 2, 3)

output0 = torch.stack( (input[0, index[0,0], :, :], input[0, index[0,1], :, :], input[0, index[0,2], :, :], input[0, index[0,3], :, :]))
output1 = torch.stack( (input[1, index[1,0], :, :], input[1, index[1,1], :, :], input[1, index[1,2], :, :], input[1, index[1,3], :, :]))

output = torch.stack((output0, output1))

res = input[torch.arange(2).unsqueeze(1), index]
print((res == output).all())
&gt; tensor(True)
</code></pre>
Can gradients be computed on tensors obtained from a foward hook?

username: eqy
cooked: <p>Hi, I’m trying to see if I can finetune a pretrained vision model for a downstream task using intermediate activations as features. Since it’s cumbersome to manually save activations by editing model definitions to output multiple tensors, I was trying to do this by just adding hooks to save activations from relevant layers.</p>
<p>As a litmus test, I wanted to see if I could train a model by grabbing the last layer’s output from a hook rather than directly from the model’s output:</p>
<pre><code class="lang-auto">def save_act(name, mod, inp, out):
    activations[name].append(out)
</code></pre>
<p>and using the saved output from the linear layer to try and compute the gradients:</p>
<pre><code class="lang-auto">for name, m in model.named_modules():
    if type(m) == nn.Linear:
        m.register_forward_hook(partial(save_act, name))
...
activations[name].append(out)
...
model(images)
output = activations['module.fc'][0]
loss = criterion(output, target)
optimizer.zero_grad()
loss.backward()
</code></pre>
<p>At first, after trying this, I get:</p>
<pre><code class="lang-auto">RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.
</code></pre>
<p>But after adding <code>retain_graph=True</code> to <code>loss.backward()</code>, I get</p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [512, 196]], which is output 0 of TBackward, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>
<p>which makes it seem like the Tensor that was saved is somehow stale or different from the one I should be using. Is there another way/fix for computing gradients on tensors obtained in a similar way?</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>Could you try to assign the <code>out</code> tensor to <code>activations[name]</code> instead of appending it?<br>
If I understand your use case correctly, you would only need to current output not all outputs from previous iterations (which might cause this error, if their computation graph was already freed).</p>

username: eqy
cooked: <p>Yes, you were right, it turns out using the first tensor in the list over and over was the problem and not even the actual intent. Thanks!</p>
Calculate training accuracy if model output is a dict of loss values

username: CplDunn
cooked: <p>Hi,</p>
<p>Im very new to machine learning and I found quite an interesting implementation of a Faster-RCNN with resnet101 from torchvision.<br>
While tinkering around, I tried to calculate the accuracy from the output during training.<br>
Problem is, the model only returns a dictionary composed of loss values:</p>
<pre><code class="lang-auto">    for batch_index, (images, targets) in enumerate(train_data_loader):
        # move the images and targets to device
        images = list(image.to(DEVICE) for image in images)
        targets = [{k: v.to(DEVICE) for k, v in t.items()} for t in targets]
        loss_dict = model(images,targets)
        loss = sum(loss for loss in loss_dict.values())
</code></pre>
<p>Model:</p>
<pre><code class="lang-auto">    backbone = resnet_fpn_backbone('resnet152', pretrained=True)
    model = FasterRCNN(backbone, num_classes=2)
</code></pre>
<p>loss_dict output:</p>
<pre><code class="lang-auto">{'loss_classifier': tensor(0.6970, device='cuda:0', grad_fn=&lt;NllLossBackward&gt;), 'loss_box_reg': tensor(0.1849, device='cuda:0', grad_fn=&lt;DivBackward0&gt;), 'loss_objectness': tensor(0.6929, device='cuda:0', grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward&gt;), 'loss_rpn_box_reg': tensor(0.3251, device='cuda:0', grad_fn=&lt;DivBackward0&gt;)}
</code></pre>
<p>How do I calculate accuracy from this?</p>
<p>Anyone can point me in the right directions?</p>
Learnable parameter in Softplus

username: DKSG
cooked: <p>Hi, I would like to know how to set the scale parameter ‘beta’ to be learnable in torch.nn.Softplus( ).</p>
<p>Please suggest.<br>
Thank you.</p>

username: tom
cooked: <p>I think you can just make a new module (and probably also define the function yourself to get gradients).</p>
<pre><code class="lang-auto">class LearnedSoftPlus(torch.nn.Module):
    def __init__(self, init_beta=1.0, threshold=20):
        super().__init__()
        # keep beta &gt; 0
        self.log_beta = torch.nn.Parameter(torch.tensor(float(init_beta)).log())
        self.threshold = 20
    def forward(self, x):
        beta = self.log_beta.exp()
        beta_x = beta * x
        return torch.where(beta_x &lt; 20, torch.log1p(beta_x.exp()) / beta, x)
</code></pre>
<p>or somesuch (didn’t test it, really).</p>
<p>Best regards</p>
<p>Thomas</p>

username: DKSG
cooked: <aside class="quote no-group quote-modified" data-username="tom" data-post="2" data-topic="60371">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar"> tom:</div>
<blockquote>
<p>class LearnedSoftPlus(torch.nn.Module): def <strong>init</strong>(self, init_beta=1.0, threshold=20): # keep beta &gt; 0 self.log_beta = torch.nn.Parameter(torch.tensor(float(init_beta)).log()) self.threshold = 20 def forward(self, x): beta = self.log_beta.exp() beta_x = beta * x return torch.where(beta_x &lt; 20, torch.log1p(beta_x.exp()) / beta, x)</p>
</blockquote>
</aside>
<p>Thank you. I have checked it and it seems to work correctly (just missed the super().<strong>init</strong>() line.)</p>

username: tom
cooked: <p>Oh, indeed, thanks for pointing that out. I added that in just in case someone else tries to use the code.</p>

username: alldbi
cooked: <p>Hi, thanks a lot for your code.<br>
It would be perfect if you could replace self.threshold with the numerical value of 20 in the forward path.</p>
[Blog Post] Latent Space Exploration with StyleGAN2

username: Epoching
cooked: <p>I made a new blog post (my second one)!</p>
<p>It’s about exploring the latent space with StyleGan2.</p>
<p>Featuring Jeremy Howard and Barack Obama.</p>
<p>Inside you’ll find lots of information and super cool visualizations about:</p>
<ul>
<li>brief intro to GANs &amp; latent codes.</li>
<li>generating images with stylegan2</li>
<li>latent interpolations to “morph” people together</li>
<li>projecting your own images into the latent space</li>
<li>control facial features using latent directions</li>
<li>interactive widget-app to play with latent directions</li>
</ul>
<p>Check it out here <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> : <a href="https://amarsaini.github.io/Epoching-Blog/jupyter/2020/08/10/Latent-Space-Exploration-with-StyleGAN2.html" rel="nofollow noopener">Latent-Space-Exploration-with-StyleGAN2</a></p>
<p>Also, here’s my previous blog post: <a href="https://amarsaini.github.io/Epoching-Blog/jupyter/2020/03/23/Self-Supervision-with-FastAI.html" rel="nofollow noopener">Self-Supervision-with-FastAI</a></p>
How to add a custom classifier block to a pre-trained Network using nn.Module?

username: Wologman
cooked: <p>I’m still fairly new to PyTorch, and grappling with this problem for my own understanding:   I am trying to insert a new untrained classifier block at the end of densenet 121, replacing the existing one and defining it using nn.Module instead of nn.Sequential.</p>
<p>It works fine if I do something along these lines:</p>
<pre><code>classifier = nn.Sequential(
                      nn.Linear(1024, 500),
                      nn.ReLU(),
                      nn.Linear(500, 1),
                      nn.Sigmoid())
</code></pre>
<p>I’d like to try and define a nn.Module from scratch instead.  In this case I was trying something like:</p>
<pre><code>class New_Classifier(nn.Module):
    def __init__(self):
        super(New_Classifier, self).__init__()
        self.fc1 = nn.Linear(feature_length,hidden_1)
        self.fc2 = nn.Linear(hidden_1, hidden_2)
        self.fc3 = nn.Linear(hidden_2, 1)   # should output the final score
        # dropout layer
        self.dropout = nn.Dropout(0.2)

def forward(self, x):
    x = F.relu(self.fc1(x)) # fully connected layer 1
    x = self.dropout(x)
    x = F.relu(self.fc2(x)) # fully connected layer 2
    x = torch.sigmoid(self.fc3(x))   # Return a value between 0 and 1
    return x
</code></pre>
<p>model.classifier = New_Classifier</p>
<p>I realise this isn’t very elegant or necessary, I just wanted to understand how one would go about this if the new classifier wasn’t nicely definable with nn.Sequential or similar.  I get the following error:</p>
<p>TypeError: cannot assign ‘<strong>main</strong>.New_Classifier’ as child module ‘classifier’ (torch.nn.Module or None expected)</p>
<p>Any suggestions what caused this error?  I suspect the ‘child module’ is a clue.   Or if I’m trying to do something that just can’t be done?</p>

username: mariosasko
cooked: <p>Replace the line <code>model.classifier = New_Classifier</code> with <code>model.classifier = New_Classifier()</code> (just append <strong>()</strong> at the end) so you actually create an instance of the module.</p>

username: Wologman
cooked: <p>Thanks very much Mariosasko, that was awesome.  I threw in the () and it worked flawlessly!  Now I’m a little embarrassed I missed that!</p>
Storing task specific layers output for further use in multi-task learning

username: theRabiulAwal
cooked: <p>I am trying to train two tasks one after another in multitask learning. I have two CNNs for two tasks. While I am computing task 1 layers I also need to use CNN layers output from the other task. I am getting retain_graph = True error as I initialized all layers output as tensors! I don’t know how to fix it!</p>
<pre><code class="lang-auto"># Codes for https://www.aclweb.org/anthology/N18-2114/ / Gated Multi-Task Network for Text Classification
class GCNN_MTL(nn.Module):
    def __init__(self, lins, opt):
        super(GCNN_MTL, self).__init__()
        self.opt = opt
        self.tasks = 2
        self.cnns = clones(CNN_Part(opt.EMB_DIM, opt.NUM_HIDDEN, opt), self.tasks)
        self.linears = lins
        self.sig_gate = Sigma_Gate()
        self.conv_out = [[torch.zeros(self.opt.NUM_FILTER, self.opt.LENGTH - 1).cuda(),
                         torch.zeros(self.opt.NUM_FILTER, self.opt.LENGTH - 2).cuda(),
                         torch.zeros(self.opt.NUM_FILTER, self.opt.LENGTH - 3).cuda()]] * self.tasks
        self.pool_out = [[torch.zeros(self.opt.NUM_FILTER).cuda(), torch.zeros(self.opt.NUM_FILTER).cuda(),
                         torch.zeros(self.opt.NUM_FILTER).cuda()]] * self.tasks
        self.fc_out = [torch.zeros(self.opt.MID_DIM).cuda()] * self.tasks

    def forward(self, w_emb, num_task):
        self.conv_out[num_task] = self.cnns[num_task](w_emb)  # [B, Co, L] * K
        g_conv_out = [self.sig_gate(x, y) for x, y in zip(self.conv_out[num_task], self.conv_out[1 - num_task])]
        self.pool_out[num_task] = [F.max_pool1d(x, x.size(2)).squeeze(2) for x in g_conv_out]  # [B, Co] * K
        g_pool_out = [self.sig_gate(x, y) for x, y in zip(self.pool_out[num_task], self.pool_out[1 - num_task])]
        g_pool_out = torch.cat(g_pool_out, dim=1)  # [B, Co * K]

        self.fc_out[num_task] = self.linears[num_task](g_pool_out)
        gated_fc_out = self.sig_gate(self.fc_out[num_task], self.fc_out[1 - num_task])
        return gated_fc_out

</code></pre>
How to copy part of tensor?

username: Stanley_C
cooked: <p>How do I copy, for example, the 1 index of the first dimension of a 3d tensor, and create a new tensor with   only that information?</p>
<p>For example in a tensor of [5,25,25], how do I copy [x,25,25] and create a new tensor [1,25,25]?</p>

username: user_123454321
cooked: <p>Sorry, is this what you are asking (the size 25 is changed to 3 to illustrate)…</p>
<pre><code class="lang-auto">&gt;&gt;&gt; a = torch.randn(3, 3, 3)
&gt;&gt;&gt; a
tensor([[[ 0.0637, -0.7271,  0.9669],
         [-1.5508, -0.5795,  0.7472],
         [-0.9258,  0.6464,  0.7641]],

        [[-0.4879,  0.8104,  1.3433],
         [-0.3241, -1.3570, -0.2355],
         [ 2.1369,  0.1334,  1.2353]],

        [[-0.6005,  1.5156, -0.4654],
         [ 0.3012,  0.7669, -0.1073],
         [-0.2972,  0.2404, -1.3757]]])
&gt;&gt;&gt; x = 1
&gt;&gt;&gt; b = a[x, :, :].clone()
&gt;&gt;&gt; b
tensor([[-0.4879,  0.8104,  1.3433],
        [-0.3241, -1.3570, -0.2355],
        [ 2.1369,  0.1334,  1.2353]])

</code></pre>

username: Stanley_C
cooked: <p>Yes! Thank you, that is really helpful!</p>
Update weight with other netowork's output

username: hyuntae
cooked: <p>I have two networks.</p>
<p>One network is a diffusion network that makes diffused images.<br>
The other network is the classifier.</p>
<p>If the Classifier network uses the output of the diffusion network, is the diffusion network’s weight also updated?</p>
<p>The classifier network’s loss is <strong>CrossEntropy(predict_label, true label)</strong>.<br>
And predict label is the output of the classification network.<br>
Classification network’s input is the diffusion network’s output.</p>

username: mariosasko
cooked: <p>After the <code>backward()</code> call on the output of the classifier network, the parameters of the diffusion network will get their gradients populated (or accumulated if the gradients are already set to some value). The classifier network just adds operations on top of the first network.</p>
<p>This sort of behaviour can be easily disabled by calling <code>detach()</code> on the output of the first network, before passing it as an input to the second network.</p>
<pre><code class="lang-python">output_classifier = classifier(output_diffusion.detach())
</code></pre>
<p><code>detach()</code> returns a new tensor that points to the same underlying storage, but no longer tracks gradient (<code>requires_grad</code> is set to False). This means that the gradient that flows through the second network can no longer be propageted through the first network.</p>
[Quick question] How to write Local Response Normalization (LRN) layer in pytroch?

username: xiaodongww
cooked: <p>I am trying to reimplement a work in caffe using pytorch but can not find how to write the Local Response Normalization (LRN) layer.</p>
<p>Any suggestion?</p>
<p>Thanks!</p>

username: mathematics
cooked: <p>Yeah, I was looking for LRN which <a href="https://pytorch.org/docs/master/generated/torch.nn.LocalResponseNorm.html" rel="nofollow noopener">https://pytorch.org/docs/master/generated/torch.nn.LocalResponseNorm.html</a> solved .</p>
Difference between parameters in, out_channels:, kernel_size:, stride in torch.nn.Conv1d

username: granth_jain
cooked: <p>Hi,</p>
<p>I would like to know the difference between kernel_size and stride in conv1d class in torch.nn and how does they both work.</p>
<p>As i understand my input of size in_channels gets converted to out_channels but I can not get how out_channels, kernel_size and stride works out.</p>
<p>If out_channels is the output size then shouldn’t it be dependent on kernel_size and stride</p>
<p>Can someone please help to clarify this…as I am trying in code examples I am getting the input and output channels is fixed as we provide in object init…but the third parameter is changing with kernel_Size and stride.</p>
<p>My problem is to apply a 1D convolution over a 50 step time series data.<br>
Can someone please help to clarify these parameters.</p>

username: albanD
cooked: <p>Hi,</p>
<p>I would recommend reading this blogpost: <a href="https://towardsdatascience.com/intuitively-understanding-convolutions-for-deep-learning-1f6f42faee1">https://towardsdatascience.com/intuitively-understanding-convolutions-for-deep-learning-1f6f42faee1</a> that explains this with very nice animations.<br>
Let me know if you have any further question afterwards!</p>
Operation that created tensor

username: soklyras
cooked: <p>Hello.<br>
Is it possible to assign an attribute to the add operation?<br>
More specifically, after e.g. training a neural network, I want something like the following code to work :</p>
<blockquote>
<p>Blockquote<br>
if output.attribute == ‘addition’:</p>
</blockquote>

username: albanD
cooked: <p>Hi,</p>
<p>We don’t provide anything like that.<br>
But you can set any attribute you want on Tensors (that are not already used). So you can do this yourself if you want.</p>
Does Pytorch provide official preact-ResNet implementation?

username: Konpat_Ta_Preechakul
cooked: <p>ResNet was proposed in “Deep Residual Learning for Image Recognition.”<br>
There is a following paper named “Identity mappings in deep residual networks” which suggests a “pre-act” version of the ResNet, usually known as ResNetV2.</p>
<p>I have yet to find an official implementation of ResNetV2, is there any?</p>

username: albanD
cooked: <p>From what I can find in the doc here: <a href="https://pytorch.org/docs/stable/torchvision/models.html">https://pytorch.org/docs/stable/torchvision/models.html</a> we have mostly resnet and wide resnet from there.</p>
<p>Maybe <a class="mention" href="/u/fmassa">@fmassa</a> has a better idea about this?</p>

username: Konpat_Ta_Preechakul
cooked: <p>Sadly, none is preact-ResNet <img src="https://discuss.pytorch.org/images/emoji/apple/cry.png?v=9" title=":cry:" class="emoji" alt=":cry:">. The network is quite well-known, and outperforms the original ResNet. Would it be a good idea to add into the catalog?</p>

username: albanD
cooked: <p>I am not the best person to judge this <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"> You can try to open a feature request on the torchvision repo and see what they say!</p>
Model running on different steps

username: alameer
cooked: <p>I have two models both run using the same parameters (Epoch=50, Batch size=16) If I compare both models one seem to run for 965 steps, and the other runs for 395. I would expect them both to run similar steps. Any idea why would this be the case. I have attached a copy of the graph?<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/e/eee5470e860ad1a008441d947ca93898e099369d.png" data-download-href="https://discuss.pytorch.org/uploads/default/eee5470e860ad1a008441d947ca93898e099369d" title="Screenshot 2020-08-07 at 11.21.03"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/eee5470e860ad1a008441d947ca93898e099369d_2_690x486.png" alt="Screenshot 2020-08-07 at 11.21.03" data-base62-sha1="y5mRuVH2klORadkVdG0jz3Hapgp" width="690" height="486" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/eee5470e860ad1a008441d947ca93898e099369d_2_690x486.png, https://discuss.pytorch.org/uploads/default/original/3X/e/e/eee5470e860ad1a008441d947ca93898e099369d.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/e/eee5470e860ad1a008441d947ca93898e099369d.png 2x" data-dominant-color="F9F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2020-08-07 at 11.21.03</span><span class="informations">831×586 35.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> Is the model clever enough to stop as training value isn’t changing any more?</p>

username: ptrblck
cooked: <p>The runtime difference might come from:</p>
<ul>
<li>different datasets and thus a different number of samples</li>
<li>different print logic in your code</li>
<li>different training (including early stopping)</li>
<li>…</li>
</ul>
<p>Without seeing code I can just speculate. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: alameer
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Many Thanks once more for your help. The points you mentioned help to pin point the issue. Thanks a lot.</p>
Why passing data to subprocess caused model block?

username: chhuang
cooked: <p>Hi, when I want to get data from dataloader and  pass it to subprocess, my model and subprocess will block.  But in subprocess create dataloader and get data, model will work normally.</p>
<p>Following code will block</p>
<pre><code class="lang-auto">def func(net, d):
    out = net(d)

if __name__ == '__main__':
    net = Net(input_w=28*28, width=64, n_layer=3, output_w=10) #dense network
    trainloader = get_data() #get trainloader
    data, label = iter(trainloader).next()
    data = data.view(data.size(0), -1)
    with Pool() as pool:      
        pool.starmap(func, [(net, data)])
</code></pre>
<p>Following code works normally</p>
<pre><code class="lang-auto">def func(net):
    trainloader = get_data() #get trainloader
    data, label = iter(trainloader).next()
    data = data.view(data.size(0), -1)
    out = net(data)

if __name__ == '__main__':
    net = Net(input_w=28*28, width=64, n_layer=3, output_w=10) #dense network
    with Pool() as pool:      
        pool.starmap(func, [(net, )])
</code></pre>
<p>I don’t know what reason caused this problem. Thanks everyone.</p>

username: mrshenli
cooked: <p>Does it work if the <code>data</code> tensor is not from <code>DataLoader</code>. Say what if you create the tensor using <code>torch.zeros</code>? Does it still hang?</p>

username: chhuang
cooked: <p>Not working. If I create torch.zeros in func(), it can work. But create in “if <strong>name</strong> == ‘<strong>main</strong>’:” then pass to func(), it hangs.</p>

username: mrshenli
cooked: <p>hmm, the following code works for me locally. Can you try this in your dev env?</p>
<pre><code class="lang-python">import torch
from torch.multiprocessing import Pool

def func(net, d):
    out = net(d)
    print(out)

if __name__ == '__main__':
    net = torch.nn.Linear(2, 2)
    data = torch.zeros(2, 2)
    with Pool() as pool:
        pool.starmap(func, [(net, data)])
</code></pre>

username: chhuang
cooked: <p>I tried your code then I found a strange problem!! If I pass <code>torch.zeros(32, 28*28)</code>, it can work. But!! If passing <code>(64, 28*28)</code>, it hangs. This problem doesn’t happen on my macbook, But happened on Linux PC.</p>

username: mrshenli
cooked: <p>Could it be the machine/container has been configured to use a very small shm size?</p>

username: chhuang
cooked: <p>Your suggestion is helpful for me! I’ll try to adjust shm size. If succed, I’ll reply. Thank you very much.</p>

username: chhuang
cooked: <p>I typed <code>df -h</code> in terminal and content showed as following :<br>
`Filesystem  Size  Used Avail Use% Mounted on’</p>
<p>tmpfs           7.9G   30M  7.9G   1% /dev/shm`</p>
<p>The <code>/dev/shm</code> has 7.9G, this space is big enough for using.</p>

username: mrshenli
cooked: <p>cc <a class="mention" href="/u/colesbury">@colesbury</a> do you know what could cause this error?</p>
How to use pretrained ResNet 2+1D to do video classification

username: SimonZYC
cooked: <p>Hi,</p>
<p>I am working on a demo about video classification. Could anyone share with me example codes of reading a video, doing the transformation, predicting and outputting the label with pretrained <a href="https://pytorch.org/docs/stable/torchvision/models.html#resnet-2-1-d" rel="nofollow noopener"><code>torchvision.models.video.r2plus1d_18</code></a> ?</p>
<p>Thanks a lot!</p>

username: XY_Z
cooked: <p>I found an example in <a href="https://github.com/HHTseng/video-classification/blob/master/ResNetCRNN/UCF101_ResNetCRNN.py" rel="nofollow noopener">https://github.com/HHTseng/video-classification/blob/master/ResNetCRNN/UCF101_ResNetCRNN.py</a>, that may be helpful.</p>

username: SimonZYC
cooked: <p>Thanks a lot! I got some inspiration from it.</p>

username: Istvan_Kecseti
cooked: <p>Hi,</p>
<p>Did you managed to find some example code?<br>
I also need that, please share if you have any <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Thanks</p>
Datasets: change y values of existing dataset

username: Roy_R
cooked: <p>Hi all,<br>
I would love to get some assistance regarding Dataset handling.</p>
<p>My data:</p>
<ol>
<li>a single folder of images [~3K small images]</li>
<li>a pandas dataframe containing a numeric value for each image [range 0 - 10000]</li>
</ol>
<p>My Task:<br>
to “predict” the numeric value of an unseen image - using convolution layers and fc layers.</p>
<p>Status:<br>
since i am relatively new to pytorch and to this task, I am quiet struggling with creating the dataset.<br>
i read some examples online - and in most of them the dataset [like cifar-10] is loaded already with the classes attached for each image.</p>
<p>Questions:</p>
<ol>
<li>I tried to create a dataset using the ImageFolder class - which in turn makes all images class value 0. is there a way to change the class value with my actual values ?</li>
<li>is my approach wrong ? should i try to create a new dataset ? and if so, how do i attach “the y values” to the “x values” ?</li>
<li>since my computer isnt very strong - how can i divide my data into batches efficiently in order to train my entire dataset without the computer yelling “not enough memory space” ?</li>
</ol>
<p>thanks !</p>
<p>P.S - if you think there is a category for this post - let me know<br>
P.S - if you think the title of this question should be different - let me know<br>
P.S - i tried reading this post : <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/data_loading_tutorial.html</a><br>
but didnt understand how to apply it to my situation</p>

username: user_123454321
cooked: <p>You can define your custom dataset class extending the torch.utils.data.Dataset class. You have to implement the <strong>init</strong>(), <strong>getitem</strong>() and <strong>len</strong>() classes. Here is a good tutorial for writing custom datasets <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">Pytorch Custom Dataset</a></p>
Saved on libtorch, load on libtorch

username: yessyw
cooked: <p>I try to load my trained model on libtorch CUDA.<br>
This is my model (How to save and Load)</p>
<pre><code class="lang-auto">struct NetImpl : torch::nn::Module{
  NetImpl() :
               ///////////Model////////////////
}
TORCH_MODULE(Net);

Net model;
device = torch::Device(torch::kCUDA, numDev);
model-&gt;to(device);
torch::load(model, saveAndLoad + "/model_checkpoint.pt");
torch::load(optimizer, saveAndLoad + "/optimizer_checkpoint.pt");
</code></pre>
<pre><code class="lang-auto">torch::save(model, saveAndLoad + "/model_checkpoint.pt");
torch::save(optimizer, saveAndLoad + "/optimizer_checkpoint.pt");
</code></pre>
<p>It is work on same device number GPU.<br>
If load the model on across device there is error.</p>
<p>I found document about “map_location”.<br>
It is a function that allocate device number when load model on python.<br>
But, I can’t find how to set map_location on C++. please help me.</p>
Strange classification accuracy drop

username: Dian-Shan
cooked: <p>Hello,<br>
I’m training an incremental classification model (ResNet-32) on the CIFAR-100 dataset.</p>
<p>Using cross-entropy loss, I can get the final 43.2% accuracy. I fixed random seed for each random module (eg Python built-in random module, np.random, torch, etc.), so I can get the same accuracy each time if I trained the model many times.</p>
<p>However, if I put some other torch.tensor on GPU, it causes the accuracy drop (42.6%). And indeed this torch.tensor is not used in training.</p>
<p>(torch.tensor example, <code>self.class_sizes = torch.tensor([20, 20, 500, 500,.....]).view(-1, 1).to(self._device)</code>, and <code>self.class_sizes</code> doesn’t be used in training process. I just put this variable there.)</p>
<p>Does anyone have any ideas or encountered similar situations? Thanks!</p>
<p>conda environment</p>
<pre><code class="lang-auto">pytorch=1.3.0=py3.7_cuda10.0.130_cudnn7.6.3_0
torchvision=0.2.2
cudatoolkit=10.0.130=0
</code></pre>
<p>0809 Update:<br>
Another similar situation<br>
If I have a Python class (<code>BigModel</code> below), which includes the training/test procedure.</p>
<pre><code class="lang-auto">class AuxModel(nn.Module):
    def __init__(self):
        super(AuxModel, self).__init__()

        self.align_FC = nn.Linear(64, 1)

        for m in self.modules():
            if isinstance(m, nn.Linear):
                nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')
                if m.bias is not None:
                    m.bias.data.zero_()

    def forward(self, x):
        return self.align_FC(x)

class BigModel():
    def __init__(self, args):
           ....
           self._network = ResNet32(...)

           self.SP = AuxModel()

    def train(self, ...):
            ....

    def test(self, ...):
            ....
</code></pre>
<p>If I train the model (ResNet32), and totally not used <code>self.SP</code>, I will get lower accuracy (74.2%).</p>
<p>But if I directly comment out the <code>#self.SP</code>, I can get higher accuracy (76.6%). Another thing I can know is that I do not use the <code>self.SP</code> in my code, because I directly commented out that, I did not get any error message.</p>

username: ptrblck
cooked: <p>Since <code>AuxModel</code> initialized modules randomly, the <a href="https://en.wikipedia.org/wiki/Pseudorandom_number_generator">pseudorandom number generator</a> will be called and will thus change the random values of all following operations. The accuracy decrease seems to be a symptom of the changed seed and you should see the same effect, if you remove <code>self.SP</code> and rerun the code with different seeds.</p>

username: Dian-Shan
cooked: <p>Many thanks for your reply!</p>
<p>I will try to use different seeds and <code>self.SP</code> architectures to do some experiments, and report the results later.</p>
<p>Not mentioned above, different <code>self.SP</code> architectures cause different final accuracy drops! I think this observation also results from the symptom that you had mentioned, which mainly related to pseudorandom number generator)</p>
How to add 2 regressor block for last layer in densnet

username: Varsha_Raj
cooked: <p>i want to replace last layer of densnet161 with 2 regressor block each containing ReLU-activated hidden layer of 256 units.</p>
<p>Can anyone suggest me how to replace</p>

username: user_123454321
cooked: <p>You can try maybe…</p>
<pre><code class="lang-auto">class MyModel(nn.Module):
   def __init__(self):
      super(MyModel, self).__init__()
      self.densenet = nn.Sequential(*list(torchvision.models.densenet169(pretrained=True).children())[:-1])
      self.avg_pool = nn.AdaptiveAvgPool2d((1, 1))
      self.res_blocks = nn.Sequential(
                                        nn.Linear(1664, 256),
                                        nn.ReLU(inplace=True),
                                        nn.Linear(256, 256),
                                        nn.ReLU(inplace=True))
   def forward(self, x):
      # suppose shape of x is (batch, 3, 224, 224)
      x = self.densenet(x) # shape is (batch, 1664, 7, 7)
      x = self.avg_pool(x) # shape is (batch, 1664, 1, 1)
      x = x.reshape(x.shape[0], -1) # shape is (batch, 1664)
      return self.res_blocks(x)
</code></pre>

username: Varsha_Raj
cooked: <p>but each unit is directed to output<br>
ie 2 regressor block each containing ReLU-activated hidden layer of 256 units will be continued to output unit<br>
ie 2 outputs are expected</p>

username: user_123454321
cooked: <p>Okay…do you mean like this ?</p>
<pre><code class="lang-auto">class MyModel(nn.Module):
   def __init__(self):
      super(MyModel, self).__init__()
      self.densenet = nn.Sequential(*list(torchvision.models.densenet169(pretrained=True).children())[:-1])
      self.avg_pool = nn.AdaptiveAvgPool2d((1, 1))
      self.reg_block1 = nn.Sequential(
                                        nn.Linear(1664, 256),
                                        nn.ReLU(inplace=True))
      self.reg_block2 = nn.Sequential(
                                        nn.Linear(1664, 256),
                                        nn.ReLU(inplace=True))
   def forward(self, x):
      # suppose shape of x is (batch, 3, 224, 224)
      x = self.densenet(x) # shape is (batch, 1664, 7, 7)
      x = self.avg_pool(x) # shape is (batch, 1664, 1, 1)
      x = x.reshape(x.shape[0], -1) # shape is (batch, 1664)
      out1 = self.reg_block1(x)
      out2 = self.reg_block2(x)
      return out1, out2
</code></pre>

username: Varsha_Raj
cooked: <p>maybe same , thank you .<br>
i will check with this</p>
PyTorch evaluation metrics

username: alameer
cooked: <p>I would like to know what is the best evaluation metric to use for evaluating a model, please? The training code as below calculates the train count error train loss, test error, and test error. Could anyone please help? One way, I thought about doing is to get the average value for each of these values? rain count error train loss, test error, and test error. Many Thanks!</p>
<pre><code class="lang-auto"># main training loop
global_step = 0
best_test_error = 10000
for epoch in range(15):
    print("Epoch %d" % epoch)
    model.train()
    for images, paths in tqdm(loader_train):
        images = images.to(device)
        targets = torch.tensor([metadata['count'][os.path.split(path)[-1]] for path in paths]) # B
        targets = targets.float().to(device)

        # forward pass:
        output = model(images) # B x 1 x 9 x 9 (analogous to a heatmap)
        preds = output.sum(dim=[1,2,3]) # predicted cell counts (vector of length B)
        
        # backward pass:
        loss = torch.mean((preds - targets)**2)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        # logging:
        count_error = torch.abs(preds - targets).mean()
        writer.add_scalar('train_loss', loss.item(), global_step=global_step)
        writer.add_scalar('train_count_error', count_error.item(), global_step=global_step)

        print("Step %d, loss=%f, count error=%f" % (global_step,loss.item(),count_error.item()))

        global_step += 1
    
    mean_test_error = 0
    model.eval()
    for images, paths in tqdm(loader_test):
        images = images.to(device)
        targets = torch.tensor([metadata['count'][os.path.split(path)[-1]] for path in paths]) # B
        targets = targets.float().to(device)

        # forward pass:
        output = model(images) # B x 1 x 9 x 9 (analogous to a heatmap)
        preds = output.sum(dim=[1,2,3]) # predicted cell counts (vector of length B)

        # logging:
        loss = torch.mean((preds - targets)**2)
        count_error = torch.abs(preds - targets).mean()
        mean_test_error += count_error
        writer.add_scalar('test_loss', loss.item(), global_step=global_step)
        writer.add_scalar('test_count_error', count_error.item(), global_step=global_step)
        
        global_step += 1
    
    mean_test_error = mean_test_error / len(loader_test)
    print("Test count error: %f" % mean_test_error)
    if mean_test_error &lt; best_test_error:
        best_test_error = mean_test_error
        torch.save({'state_dict':model.state_dict(),
                    'optimizer_state_dict':optimizer.state_dict(),
                    'globalStep':global_step,
                    'train_paths':dataset_train.files,
                    'test_paths':dataset_test.files},checkpoint_path)</code></pre>
How to get a new tensor via repeating index for element times?

username: darknight
cooked: <p>If I have a tensor [1,2,3,4], how can I get tensor [0, 1, 1, 2, 2, 2, 3, 3, 3, 3] as fast as possible?</p>
<p>A for-loop is very slow, but I haven’t find any relevant Pytorch API.</p>
<p>Can anyone provide a good way?</p>

username: ptrblck
cooked: <p>Generally for this kind of problem, you could use <code>torch.repeat_interleave</code>:</p>
<pre><code class="lang-python">x = torch.arange(5)
torch.repeat_interleave(x, torch.tensor([1, 2, 3, 4, 5]), dim=0)
</code></pre>
<p>However, in your example, somehow a <code>0</code> is added at the beginning and the <code>4</code> is missing, so you might need to use <code>torch.cat</code> and a slicing operation.</p>

username: darknight
cooked: <p>Thank you very much! And I tested the efficiency of three methods:<br>
                        <a href="https://imgur.com/4VcyLRH" target="_blank" rel="nofollow noopener" class="onebox">
              <img src="https://i.imgur.com/4VcyLRH.jpg?fb" title="Imgur" alt="Imgur" height="315" width="600">
            </a>

</p>
When I export DeepLab V3 using torch.onnx.export, I find that the onnx model has two outputs. Why？

username: xuruizhi
cooked: <p>model = models.segmentation.deeplabv3_resnet101(pretrained=True)<br>
model.eval()<br>
print(model)<br>
torch.save(model, “deeplabv3_resnet101.pth”)</p>
<p>dummy_input = torch.randn(1, 3, 224, 224)<br>
input_names = [“actual_input_1”] + [“learned_%d” % i for i in range(16)]<br>
torch.onnx.export(model,<br>
dummy_input,<br>
“deeplabv3_resnet101.onnx”,<br>
verbose=True,<br>
input_names=input_names,<br>
opset_version=11,<br>
)</p>
<p>deeplabv3_resnet101.onnx, have two outputs: “1094” “1078”</p>
Mixed precision for validation?

username: Martin_cb
cooked: <h1>following on from the pytorch tutorials for amp here:</h1>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://pytorch.org/favicon.ico?" class="site-icon" width="16" height="16">
      <a href="https://pytorch.org/blog/accelerating-training-on-nvidia-gpus-with-pytorch-automatic-mixed-precision/" target="_blank" rel="nofollow noopener">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="https://pytorch.org/assets/images/pytorch-logo.png" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://pytorch.org/blog/accelerating-training-on-nvidia-gpus-with-pytorch-automatic-mixed-precision/" target="_blank" rel="nofollow noopener">PyTorch</a></h3>

<p>An open source deep learning platform that provides a seamless path from research prototyping to production deployment.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Here is how I apply the amp</p>
<pre><code class="lang-auto">scaler = GradScaler()
for data, label in data_iter: 
   optimizer.zero_grad() 
   # Casts operations to mixed precision 
   with autocast(): 
      loss = model(data) 
 
   scaler.scale(loss).backward() 
   scaler.step(optimizer) 
   scaler.update() 


for data, label in data_iter_valid: 
    # ??
</code></pre>
<p>I was wondering if/when to apply mixed precision onto the validation data??  Do you need to use scaler on it?<br>
Or is that even necessary? Wouldn’t it be needed if you wanted to keep the peak memory requirements the same during training/validation?</p>

username: ptrblck
cooked: <p>The <code>scaler</code> is not necessary during evaluation, as you won’t call <code>backward</code> in this step and thus there are not gradients to scale.<br>
You can still use <code>autocast</code> and also wrap the code in a <code>with torch.no_grad()</code> block to save more memory.</p>