akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Correct order of .backward()'s for multiple losses, multiple model runs in training step

JeremySMorgan: <p>Hi There,</p>
<p>I’m working on training an Invertible Neural Network (see <a href="https://arxiv.org/abs/1808.04730" rel="nofollow noopener">https://arxiv.org/abs/1808.04730</a> / <a href="https://github.com/VLL-HD/FrEIA" rel="nofollow noopener">https://github.com/VLL-HD/FrEIA</a>), which basically is a neural network that you can run both in the forward and in the reverse direction. For each training batch, I run the network in the forward direction, then calculate multiple losses and then run the network in reverse, and calculate multiple losses.</p>
<p>I’m posting to get help with <em>when to call .backward() on my losses</em>.</p>
<p>I want to only run one parameter update per batch, so not after the forward, and reverse step. Simplified down, my current program runs like:</p>
<pre><code class="lang-auto">def training_step(x, y):

    y_hat = f(x)
    forward_loss1 = floss1(y_hat, y)
    forward_loss2 = floss2(y_hat, y)
    l_forward = forward_loss1 + forward_loss2
    l_forward.backward()
    ...

    x_hat = f(y, reverse=True)
    reverse_loss1 = rloss1(x_hat, x)
    reverse_loss2 = rloss2(x_hat, x)
    l_rev = reverse_loss1 + reverse_loss2
    l_rev.backward()

    optimizer.step()
</code></pre>
<p>Questions:</p>
<ol>
<li>Is this a correct way to train? would it instead be better to run <code>l_total = reverse_loss1 + reverse_loss2 + forward_loss1 + forward_loss2</code> and then <code>l_total.backward()</code>?</li>
<li>Does running the network the second time clear the gradient saved from the l_forward.backward()?</li>
</ol>
<p>Thanks so much!</p>

mariosasko: <ol>
<li>It seems fine. The second approach consumes more memory (it has to store two graphs from the forward and the reverse pass at the same time), but the end result should be the same.</li>
<li>No, the gradients get accumulated (summed with the previous value).</li>
</ol>
Loss consistently between 0.7 and 0.8 with around 50% accuracy

brenth: <p>Hi all, my losses for training and validation are constantly between 0.7 and 0.8 while accuracy is around 50%.<br>
Am using transfer learning so in the training phase: source’s classification loss, together with the distance loss from difference between source dataset and target dataset are used.</p>
<p>Does anyone have any advice what to do with the stagnating loss?  Newbie here so would appreciate any advice, however obvious it may seem. Thanks.</p>
<blockquote>
<pre><code>        with torch.set_grad_enabled(phase=='source_train'):  
</code></pre>
</blockquote>
<pre><code>            features=model(inputs)
            classifier_layer=nn.Linear(8000,2)
            outputs=classifier_layer(features)
            if phase!='source_train':
                logits=outputs
                loss=loss_func(logits,labels)
            else:
                logits=outputs.narrow(0,0,labels.size(0))
                classifier_loss=loss_func(logits,labels)
                distance_loss=DAN(features.narrow(0, 0, labels.size(0)),features.narrow(0, labels.size(0), inputs.size(0)-labels.size(0)))
                loss=classifier_loss+distance_loss
</code></pre>

Aryaman_Sriram: <p>Try printing out the labels and see if they are changing every 10/20 steps depending upon the number of steps per epoch.</p>
Which activation function to use in segmentation?

arjun_pukale: <p>which activation function to be used at last layer of segmentation models like segnet, unet?<br>
should I use <code>F.sigmoid</code> while defining the model’s last layer itself?</p>

KFrank: <p>Hi Arjun!</p>
<aside class="quote no-group" data-username="arjun_pukale" data-post="1" data-topic="92993" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/arjun_pukale/48/22716_2.png" class="avatar"> arjun_pukale:</div>
<blockquote>
<p>which activation function to be used at last layer of segmentation models like segnet, unet?<br>
should I use <code>F.sigmoid</code> while defining the model’s last layer itself?</p>
</blockquote>
</aside>
<p>The short answer is that you should just use the output of your last<br>
linear layer, with no activation function, as input to your loss function<br>
(for plain-vanilla use cases).</p>
<p>I assume that you are asking about training a network to segment<br>
2d images, that is, the output of your network will be an “image”<br>
whose pixel values tell you the segment that the original, input pixels<br>
are in.</p>
<p>If you are doing binary segmentation, e.g., background / foreground,<br>
the output of your last linear layer should have shape<br>
<code>[nBatch, height, width]</code> (as should your target), and you should<br>
use <code>BCEWithLogitsLoss</code> as your loss function.</p>
<p>For multi-class segmentation, e.g., background / normal tissue / benign<br>
tumor / malignant tumor, the shape of your network’s output should be<br>
<code>[nBatch, nClass, height, width]</code>, your target should have shape<br>
<code>[nBatch, height, width]</code>, and you should use <code>CrossEntropyLoss</code><br>
as your loss function.</p>
<p>Again, in neither case should you feed the output of your last linear<br>
layer through an activation function.</p>
<p>Best.</p>
<p>K. Frank</p>
Computing matrix derivatives using autograd

elkop: <p>I am running into few issues while trying to compute matrix derivatives. Here is a minimal working code to reproduce the error.</p>
<pre><code class="lang-auto">theta = torch.tensor(np.random.uniform(low=-np.pi, high=np.pi), requires_grad=True)
rot_mat = torch.tensor([[torch.cos(theta), torch.sin(theta), 0], 
                        [-torch.sin(theta), torch.cos(theta), 0]], 
                        dtype=torch.float, requires_grad=True)
torch.autograd.grad(outputs=rot_mat, 
                    inputs=theta, grad_outputs=torch.ones_like(rot_mat), 
                    create_graph=True, retain_graph=True)
</code></pre>
<p>The above code results in this error “One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.”</p>
<p>I tried using allow_unused=True but the gradients are returned as None. I am not sure what is causing the graph to be disconnected here. Any help is appreciated.</p>

googlebot: <p>torch.tensor does not backpropagate, you should use cat/stack instead</p>
How do I find the optimal learning rate and weight decay to be used in conjunction with OneCycleLr

Peter_Featherstone: <p>How do I find the optimal learning rate and weight decay to be used in conjunction with OneCycleLr?<br>
If I use something like:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; data_loader = torch.utils.data.DataLoader(...)
&gt;&gt;&gt; optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
&gt;&gt;&gt; scheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr=0.01, steps_per_epoch=len(data_loader), epochs=10)
&gt;&gt;&gt; for epoch in range(10):
&gt;&gt;&gt;     for batch in data_loader:
&gt;&gt;&gt;         train_batch(...)
&gt;&gt;&gt;         scheduler.step()
</code></pre>
<p>How can I find good values for <code>lr</code>, <code>momentum</code>, <code>max_lr</code>, <code>weight_decay</code>?<br>
This repository  <a href="https://github.com/davidtvs/pytorch-lr-finder" rel="nofollow noopener">https://github.com/davidtvs/pytorch-lr-finder</a> shows you how to get a learning rate curve but how do i find an optimal value for weight decay? Surely the optimal values for both have to be found at the same time, i.e. we need to find a minimum on a hyperplane as opposed to a curve…</p>
Dataset Inheritance does not require super?

n4tman: <p>I have seen in the Dataset pytorch tutorial (<a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Writing Custom Datasets, DataLoaders and Transforms — PyTorch Tutorials 2.2.0+cu121 documentation</a>) that super is not called when inheriting from Dataset.</p>
<p>Does anyone know why this has not been done?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/6/963b7e800a02c60657037b91e8e6a55767aa8805.png" data-download-href="https://discuss.pytorch.org/uploads/default/963b7e800a02c60657037b91e8e6a55767aa8805" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/963b7e800a02c60657037b91e8e6a55767aa8805_2_690x327.png" alt="image" data-base62-sha1="lr15lgpckc7Z2voTmCAUDVTQpxP" width="690" height="327" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/963b7e800a02c60657037b91e8e6a55767aa8805_2_690x327.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/963b7e800a02c60657037b91e8e6a55767aa8805_2_1035x490.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/6/963b7e800a02c60657037b91e8e6a55767aa8805.png 2x" data-dominant-color="EEEBEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1314×624 60.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

mariosasko: <p><code>torch.utils.data.Dataset</code> doesn’t define its own <code>__init__</code> so by calling <code>super().__init__()</code> you are not performing any meaningful actions. In comparison, calling <code>super().__init__()</code> when subclassing <code>torch.nn.Module</code> is required to initialize the internal data structures for storing buffers, modules, etc.</p>

n4tman: <p>So when extending the Dataset class we should not use <code>super().__init__()</code> in our init function of the new class that extends Dataset?</p>

mariosasko: <p>You can, but it’s not necessary. The end-result is the same.</p>
Why `torch.max` has expected device "cpu"?

kshitijpatil09: <p>I’ve two tensors namely <code>pred_boxes</code> and <code>gt_boxes</code> and both are on the ‘cuda’. When I pass these tensors to <a href="https://github.com/pytorch/vision/blob/c547e5cbbf25e46870abb2e925b42c67c3367afc/torchvision/ops/boxes.py#L153" rel="nofollow noopener"><code>box_iou</code></a> op from the torchvision, I get the following error:</p>
<pre><code class="lang-auto">/opt/conda/lib/python3.7/site-packages/torchvision/ops/boxes.py in box_iou(boxes1, boxes2)
    160     area2 = box_area(boxes2)
    161 
--&gt; 162     lt = torch.max(boxes1[:, None, :2], boxes2[:, :2])  # [N,M,2]
    163     rb = torch.min(boxes1[:, None, 2:], boxes2[:, 2:])  # [N,M,2]
    164 

RuntimeError: expected device cpu but got device cuda:0

</code></pre>
<p>So I tried calling the <code>torch.max</code> for the bounding box tensors to verify the same</p>
<pre><code class="lang-python">torch.max(gt_boxes[:, None, :2], pred_boxes[:, :2])
</code></pre>
<p>Which indeed is causing this issue</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-42-2742a63bf57b&gt; in &lt;module&gt;
----&gt; 1 torch.max(gt_boxes[:, None, :2], pred_boxes[:, :2])

RuntimeError: expected device cpu but got device cuda:0
</code></pre>
<p>I want to know why do we need the tensors to be on the cpu make use of max operation?</p>

KFrank: <p>Hi Kshitij!</p>
<aside class="quote no-group" data-username="kshitijpatil09" data-post="1" data-topic="92986" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kshitijpatil09/48/19552_2.png" class="avatar"> kshitijpatil09:</div>
<blockquote>
<p>I’ve two tensors namely <code>pred_boxes</code> and <code>gt_boxes</code> and both are on the ‘cuda’.</p>
</blockquote>
</aside>
<p>Please double check this.  I can’t reproduce this (on an older version<br>
of pytorch).</p>
<aside class="quote no-group">
<blockquote>
<p>So I tried calling the <code>torch.max</code> for the bounding box tensors to verify the same</p>
<pre data-code-wrap="python"><code class="lang-python">torch.max(gt_boxes[:, None, :2], pred_boxes[:, :2])
</code></pre>
<p>Which indeed is causing this issue</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-42-2742a63bf57b&gt; in &lt;module&gt;
----&gt; 1 torch.max(gt_boxes[:, None, :2], pred_boxes[:, :2])

RuntimeError: expected device cpu but got device cuda:0
</code></pre>
</blockquote>
</aside>
<p>Could you add</p>
<pre data-code-wrap="python"><code class="lang-python">print (torch.max)   # to make sure you didn't overwrite torch.max
print (gt_boxes.device, pred_boxes.device)   # verify tensor locations
</code></pre>
<p>before your call to <code>torch.max()</code>?</p>
<p>Best.</p>
<p>K. Frank</p>
What losses can be used to predict a scalar value?

oasjd7: <p>The Ground Truth label is [0.0,1.0]. And I want to predict this scalar value.</p>
<p>In this case, What losses can be used ?</p>
<p>Can I use <code>MSE</code> loss with <code>nn.Linear(input_dim, 1)</code>?</p>
<pre><code class="lang-auto">networks = nn.Sequential(... , nn.Linear(input_dim, 1)
predictions = networks(input)
loss = MSE(predictions, GT)
</code></pre>
<p>Thanks  in advance !</p>

KFrank: <p>Hello oasjd7!</p>
<aside class="quote no-group" data-username="oasjd7" data-post="1" data-topic="92899" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oasjd7/48/5053_2.png" class="avatar"> oasjd7:</div>
<blockquote>
<p>The Ground Truth label is [0.0,1.0]. And I want to predict this scalar value.</p>
</blockquote>
</aside>
<p>If by this you mean that each individual training sample is assigned a<br>
ground-truth target of <em>either</em> <code>0.0</code> <em>or</em> <code>1.0</code>, I would call this a <em>binary</em><br>
value (although still technically a scalar).</p>
<aside class="quote no-group">
<blockquote>
<p>In this case, What losses can be used ?</p>
</blockquote>
</aside>
<p>For a <em>binary classification</em> problem <code>BCEWithLogitsLoss</code> will be your<br>
best choice for the loss function.</p>
<aside class="quote no-group">
<blockquote>
<p>Can I use <code>MSE</code> loss with <code>nn.Linear(input_dim, 1)</code>?</p>
</blockquote>
</aside>
<p>You could, but general experience shows that for binary classification<br>
<code>MSELoss</code> won’t work nearly as well as <code>BCEWithLogitsLoss</code>.</p>
<p>If your ground-truth target value were a (morally speaking) continuous<br>
variable, then <code>MSELoss</code> would be the sensible choice.</p>
<p>For example, if you were training a network to predict the price of used<br>
cars (given, say, model, miles, age, rust, etc.) and your training data<br>
included known prices, the <code>MSELoss</code> would be the natural choice.</p>
<p>Best.</p>
<p>K. Frank</p>
Save dictionary into state_dict

Ilkin: <p>So, I wrote a custom extension of BatchNorm where I keep track of multiple running_mean and running_vars in a dictionary.  The number of such mean and vars is not specified beforehand (e.g. can be 3 or 100), so I can’t replace them with a fixed number of parameters. The dictionary method works well for me except I cannot register my running_means and running_vars dictionary into the state dict for later reloading. Can someone suggest a workaround?</p>

mariosasko: <p>You can use <code>self.register_buffer(param_name, initial_value)</code> to add these tensors in the model’s state dict.</p>
<p>This snippet shows how to do it:</p>
<pre><code class="lang-python">def __init__(self):
    super().__init__()
    ...
    self.running_means = {}
        for i in range(100):
            value = torch.randn(5)
            self.running_means[f'running_mean{i}'] = value
            self.register_buffer(f'running_mean{i}', value)
    ...
</code></pre>

Ilkin: <p>I tried this but the <code>running_mean{i}</code> in the buffer does not get updated for some reason when I update the <code>self.running_means[i]</code>. In fact, I registered <code>self.running_means[i]</code> into the buffer directly to make sure the buffer points to the dictionary element directly, but that didn’t help either.</p>

mariosasko: <p>If you update them by reassigning the value like this <code>self.running_means[i] = torch.tensor(...)</code>, the buffers won’t get updated, but if you change these tensors in-place then it should work, e.g. <code>self.running_means[i].add_(...)</code>.</p>

Ilkin: <p>So, I should do:</p>
<p><code>self.running_means[i].add_(-self.running_means[i] + new_val)</code></p>
<p>to set <code>self.running_means[i] = new_val</code>.</p>

mariosasko: <p>To set <code>self.running_means[i]</code> to <code>new_val</code>, you can do it in-place like this: <code>self.running_means[i].data = new_val</code>.</p>
<p>If you reassign the value with <code>self.runnng_means[i] = new_val</code>, the dict key points to the new value that is not stored in the buffer (the buffer has the old value).</p>

Ilkin: <p>Worked! Thanks a lot.</p>
Single weight-sharing across a network

Chris_Culley: <p>I’d like to force a single weight to be the same in two different linear layers and have them optimise jointly.</p>
<p>I’ve tried the following, but now I can’t optimise since the ‘hidden_layer’ is no longer a leaf node.</p>
<pre><code class="lang-auto">class Net(nn.Module):
  def __init__(self, dim_in, dim_out, dim_hidden):
    super(Net, self).__init__()
    self.input_layer = nn.Linear(dim_in, dim_hidden)
    self.hidden_layer = nn.Linear(dim_hidden, dim_hidden)
    self.hidden_layer.weight[0,0] = self.input_layer.weight[0,0] # &lt;- this is my attempt at weight-sharing
    self.out_layer = nn.Linear(dim_hidden, dim_out)
</code></pre>
<p>Is there anyway to overcome this? I’d like to have build up clusters of shared weights if this is possible?</p>

adamr: <p>I am not sure if you can share <em>part</em> of the tensor. Try <a href="https://discuss.pytorch.org/t/how-to-use-shared-weights-in-different-layers-of-a-model/71263" class="inline-onebox">How to use shared weights in different layers of a model</a> and see if that works if you apply it to your case. I am curious too.</p>
Torch::from_blob from mutiple cv::Mat files

vic_rain: <p>I have two cv:Mat files: image1, image2, with same resolution.<br>
if copy separately, it is easy to do as below:<br>
torch::Tensor tensor1 = torch::from_blob (image1.data, { 1, 1, image1.rows, image1.cols });</p>
<p>However, how can I load the two images such that tensor1 has size { 1, 2, image1.rows, image1.cols }?</p>
Examples for object detection,pose estimation,segmentation

zeus: <p>Recently I used image classification in mobile and it works good.Will there be documents to deploy object detection,pose estimation, segmentation in mobile as in tensorflow in future?</p>

Ludwig_Friborg: <p>I was able to get a segmentation model work by using the classification tutorial from the pytorch homepage. I used <code>&lt;tensor&gt;.getDataAsFloatArray()</code> to get the output values.</p>
<p>In any case, currently the amount of documentation content for pytorch-mobile is ridiculously low. Like zero tutorials…</p>

zeus: <p>Yeah. But I saw in discussion saying that they are looking into it. Maybe in near future we can except those.</p>

Redney_Quartz: <p>did you get it out as a single dimensional array and have to put it back into it proper structure?</p>

zeus: <p>I only tried image classification. But wanted to do for other tasks also. If torch has these tutorials then I can fully move into torch</p>

Redney_Quartz: <p>I have been using a mobilenet-ssd successfully.<br>
I use the below function to restructure the array. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<pre><code class="lang-auto">    public float[][] floatArray1dto2d(final float[] array, final int rows, final int cols) {
        if (array.length != (rows * cols))
            throw new IllegalArgumentException("Invalid array length");

        float[][] outArray = new float[rows][cols];
        for (int i = 0; i &lt; rows; i++)
            System.arraycopy(array, (i * cols), outArray[i], 0, cols);

        return outArray;
    }
</code></pre>

kshitijpatil09: <p><a class="mention" href="/u/redney_quartz">@Redney_Quartz</a> what’s the size of your model? and on which platform you was able to use the model (ios/android) ?</p>
Different output tensor size for torch distributions

mamadpierre: <p>I’m wondering why there is <strong>a difference in the size</strong> of the output tensors for different torch.distributions?</p>
<pre><code class="lang-auto">import torch

probabilities = [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
dist1 = torch.distributions.Categorical(torch.tensor(probabilities))

dist2 = torch.distributions.beta.Beta(torch.tensor([2.0]), torch.tensor([2.0]))

distGen1 = lambda: dist1.sample([2])
distGen2 = lambda: dist2.sample([2])


print("size: ", distGen1().size())   #torch.Size([2])
print("size: ", distGen2().size())   #torch.Size([2, 1])

</code></pre>
<p>One can use <code>torch.squeezed</code> for instance to make them similar size-wise.</p>
I have two datasets but different labels alignments. How do I allign labels?

TanmDL: <p>I have two datasets like A dataset:{feature, label_A} and B dataset: {data, labels_B} where label_A and label_B are not aligned.</p>
<p>I want to write a custom dataset from these two datasets which will pass to dataloader and give outputs like feature[index], data[index], align labels (A), and index.</p>
<p>Please help me.</p>

TanmDL: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> please look into this problem.</p>

ptrblck: <p>Could you explain a bit, how these labels could be aligned?<br>
If you have a lookup table for the corresponding indices between <code>label_A</code> and <code>label_B</code>, you could use this map to load the samples from both <code>Dataset</code>s inside your custom <code>Dataset</code>.</p>

TanmDL: <p>Thank you for your kind reply…For demo purpose I am adding a demo code.</p>
<p>class my_custom_dataset(torch.utils.data.Dataset):</p>
<pre><code>  def __init__(self,data=None):
    self.data = data
    
    self.features_data = torch.from_numpy(np.random.randn(70000, 2048)).float()
    self.labels_data = torch.from_numpy(np.random.randint(0,10,(70000,))).long()

    
  def __getitem__(self, index):
    data = self.features_data[index]
    target = self.labels_data[index]
    tr_data, tr_target = self.data[index]
        
    return data,  target, tr_data, tr_target

  def __len__(self):
      
    return len(self.features_data)    
</code></pre>
<p>train_dataset = torchvision.datasets.CIFAR10(root=’./data’, train=True, download=True, transform=transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))]))<br>
test_dataset = torchvision.datasets.CIFAR10(root=’./data’, train=False, download=True, transform=transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))]))</p>
<p>total_dataset = train_dataset + test_dataset</p>
<p>custom_set = my_custom_dataset(data= total_dataset)</p>
<p>custom_loader = torch.utils.data.DataLoader(custom_set, batch_size=128, shuffle=False, num_workers=2)</p>
<p>data_, label, tr_data, tr_target = iter(custom_loader).next()<br>
print(label[:20])<br>
print(tr_target[:20])</p>
<p>print(data_.size())<br>
print(tr_data.size())</p>
<p>I have sent a demo code above. Please check the code.</p>

ptrblck: <p>Thanks for the code. I still don’t understand where exactly the issue is.<br>
Are you seeing unexpected results using the posted approach?</p>
How to do RUS and ROS on imbalanced datasets

piyush2003m: <p>I am unable to find code on how I can perform random over sampling and random under sampling on imbalanced dataset. Is there a pre-existing package or some code that is always used. I am relatively new to PyTorch. Any help would be appreciated!</p>

ptrblck: <p>You could use the <code>WeightedRandomSampler</code> to over- or undersample the dataset.<br>
<a href="https://discuss.pytorch.org/t/how-to-handle-imbalanced-classes/11264/2">Here</a> is an example showing how to use it. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

piyush2003m: <p>Thanks for the reply! Weightedrandomsampler changes the weights of class or actually oversamples/ undersamples data? Will there be any difference in the two approaches?</p>

ptrblck: <p><code>WeightedRandomSampler</code> is a custom sampler, which draws the samples from your <code>Dataset</code> given the specified weights and can be used to over- or undersample the dataset.<br>
A class weighting can be enabled via the <code>weight</code> argument in the criterion (e.g. <code>nn.CrossEntropyLoss</code>) and would apply the class weights to the loss calculation.<br>
Both approaches can be used to counter the effects of an imbalanced dataset, but are different in their workflow.</p>
How to keep hidden state of the last layer of packed GRU

Jerome_MASSOT: <p>Hi everybody,</p>
<p>I am playing with seq2seq for NMT and I was trying to add several layers to my working GRU model. Unfortunately, I see that the hidden state vectors dimensions is impacted by the number of layers.</p>
<p>If I want to keep only the last GRU layer’s hidden state, I need to truncate my h_nn tensor… but I am lost how to do it … <img src="https://discuss.pytorch.org/images/emoji/apple/face_with_hand_over_mouth.png?v=9" title=":face_with_hand_over_mouth:" class="emoji" alt=":face_with_hand_over_mouth:"></p>
<p>Or how to how to concatenate the multiple layers to get a final hidden state vectors with the same shape as I was using a monolayer GRU?</p>
<p>Thanks in advance</p>
<p>Best regards</p>
<p>Jerome</p>
<p>def forward(self, x_source, x_lengths):<br>
“”"<br>
Forward pass<br>
:param x_source: input batch of sequences<br>
:param x_lengths: length of each sequence<br>
:return: x_unpacked, x_birnn_h<br>
“”"</p>
<pre><code>    # apply the embedding on the input sequences
    x_embedded = self._source_embedding(x_source)

    # create the packed sequences structure
    x_packed = pack_padded_sequence(
        x_embedded, 
        x_lengths.detach().cpu().numpy(),
        batch_first=True
    )

    # apply the rnn
    x_birnn_out, x_birnn_h = self._birnn(x_packed)
    if self._num_layers&gt;1:
        x_birnn_h = ## TO DO : keep hiddzn for last layer only...

    # permute the dimensions of the hidden state and flatten it
    x_birnn_h = x_birnn_h.permute(1, 0, 2)
    x_birnn_h = x_birnn_h.contiguous().view(x_birnn_h.size(0), -1)

    # unpacked the sequences before to return them
    x_unpacked, _ = pad_packed_sequence(x_birnn_out, batch_first=True)

    return x_unpacked, x_birnn_h</code></pre>

vdw: <p>According to the docs, the shape of <code>x_birnn_h</code> is <code>(num_layers * num_directions, batch, hidden_size)</code>. A common way to to work with the last hidden states is:</p>
<pre><code class="lang-auto">output, hidden = self._birnn(x)
# hidden.shape = (n_layers * n_directions, batch_size, hidden_dim)
hidden = hidden.view(n_layers, n_directions, batch_size, hidden_dim)
# This view() comes directly from the PyTorch docs
# hidden.shape = (n_layers, n_directions, batch_size, hidden_dim)
hidden = hidden[-1]
# hidden.shape = (n_directions, batch_size, hidden_dim)
hidden_forward, hidden_backward = hidden[0], hidden[1]
# Both shapes (batch_size, hidden_dim)
hidden = torch.cat((hidden_forward, hidden_backward), dim=1)
</code></pre>
<p><strong>Edit:</strong> If you want the last <code>hidden</code> have the same shape as the unidirectional GRU, you can also add <code>hidden_forward</code> and <code>hidden_backward</code> (instead of concatinating). I’ve seen it done, but don’t really know what’s the best practice here.</p>

Jerome_MASSOT: <p>Thanks Chris, it helps a lot.</p>
Input to LSTM as a matrix or similar

ygl: <p>Hi, Im working on gender inference. have the data like this,<br>
userid, text,<br>
1, hello, world<br>
1, how are you<br>
…(up to 3000)   gender:Female<br>
2, whats up<br>
2, Im here<br>
…(up to 3000) gender:Male</p>
<p>The problem is that if I concat all of the texts for each user, it would be too long for LSTM to process. What I am thinking is to map each sentence into sentence embedding using bert and average them. In this way, each user is represented as a vector. However, this will generate a vector. How do I feed it into a LSTM model which takes a 3d tensor as input?</p>

vdw: <p>Well, when you use, for example, BERT to create a sentence embedding – with or with out avareging the embeddings for the same user – you do no longer have sequences. That’s the goal of a sentence embedding, to convert se sequence of words/tokens into a vector. Hence, there’s no need or even a meaningful way to feed this into an LSTM. Once you have the sentence embedding, you can feed it into any non-RNN network of linear or CNN layers.</p>
<p>On the other side, I don’t see any particular reason to treat all sentence of the same user as one “block”? For starters, you can simply treat each (sentence, gender) pair as an individual data item . Then you can use an RNN (LSTM or GRU) to train a binary classifier. It’s the most straightforward way to do, and would give if nothing else a good baseline for any more advanced methods you want to try.</p>
<p>Even if you create BERT to create sentence embeddings, why do you think it’s important to average the embeddings for all sentences of the same users. Again, I would start with using each (sentence_embedding, gender) as individual data item first.</p>

ygl: <p>Thanks Chris! I guess the reason why I wanna make a block is because for testing, using a text, gender pair, if a user has 1500 tweets, say, we could have maybe 800 males and 700 females. How do I deal with a situation like this? Just count?</p>
MAE and MSE implementations

alameer: <p>I’m trying to work out the MAE and MSE values for my model. Please can someone confirm that i am using this the correct formula?</p>
<pre><code class="lang-auto">        mae = torch.abs(preds - targets).sum().data
        mse = ((preds - targets)*(preds - targets)).sum().data

        mae = mae.detach() / len(loader_test)
        MAE_for_all_epochs.append(mae.item())
  
        mse = math.sqrt(loss / len(loader_test))
        MSE_for_all_epochs.append(mse)

        print("MAE: %f" % mae)
        print("MSE: %f" % mse)

MSE_for_all_epochs = sum(MSE_for_all_epochs)
MAE_for_all_epochs = sum(MAE_for_all_epochs)
print(MAE_for_all_epochs)
print(MSE_for_all_epochs)

model_mae= MSE_for_all_epochs / (epoch + 1)
model_mse= MAE_for_all_epochs / (epoch + 1)
print("Model MAE: %f" % model_mae)
print("Model MSE: %f" % model_mse)

</code></pre>
<p>Many Thanks</p>
Spliting folders containing images

CS.Enthu: <p>Hello, I have a folder containing n number of patient folders. Each patient folder has subfolders as dates and contains two types of subclasses CT and PET. I want to create an image data loader and apply preprocessing and  patch extraction and saving them as numpy arrays for the GANS network in a similar data structure.<br>
Is there any simple way to acheive this using pytorch. Such that the data structure is also preserved(since it contains datewise and patientwise dicom images)</p>
<p>patient_01–&gt;db10102014–&gt;CT, PET</p>
<pre><code>         &gt;db11102014--&gt;CT, PET
  .                               
  .
  .
</code></pre>
<p>patient_N</p>
<p>In the end I want  three folders train ,test and valid that will have same subfolders as  above<br>
train – patient_01&gt;db_xxxx&gt;CT, PET(each PET,CT will contain patches,instead of raw images)<br>
patient_02<br>
and so on<br>
Similarly  for test and valid folder.</p>
<p>Thank you in advance.</p>
Pytorch built in debug mode

r2d2bol: <p>Hello there, I built pytorch from source and noticed some of my training scripts were running slower than I would expect them to.  The output of a pytorch.bottleneck run has:</p>
<hr>
<h2>Environment Summary</h2>
<p>PyTorch 1.7.0a0+8864148 DEBUG compiled w/ CUDA 11.0</p>
<p>This seems to suggest pytorch is built in debug mode.  However, in a python shell:</p>
<p>In [3]: torch.version.debug<br>
Out[3]: False</p>
<p>How would I tell for sure if pytorch is built in debug mode?</p>
Generating montonically increasing sequence

praneethsv: <p>Hello,</p>
<p>I have a requirement that my neural network output should produce a floating-point sequence that is monotonically increasing i.e. something like below:</p>
<p>output = [0.01, 0.05, 0.1, 0.2, 0.3, 0.7, 0.9, 1.0]</p>
<p>Is there any way that I can implement this requirement in PyTorch? Assuming I am using RNN based neural network.</p>

vdw: <p>This requirement in itself doesn’t make much sense. What is the input of your network and what are the predictions?</p>
<p>There’s no need for a neural network to create a sequence of monotonically increasing floats <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

googlebot: <p>map outputs to positive domain (e.g. softplus(rnn_output)) and treat them as deltas: outputs=cumsum(deltas, dim=timedim)</p>

praneethsv: <p>Hi Alex,</p>
<p>Thanks for your kind reply, Can you explain, What do you mean by treat them as deltas? Please.</p>

googlebot: <p>differencing, y[t] = y[t-1] + delta, with cumsum you “integrate” deltas back</p>
Solved - Dataloader iteration

Andi: <p>Why does this basic Dataloader deliver twice the first value in the data, and not the next (first and second)? I need just a simple yield funcionality for the next series value.<br>
What is to change?<br>
Thanks in advance!</p>
<p>class myDataset(Dataset):<br>
&gt;def <strong>init</strong>(self, csv_file):<br>
&gt;&gt;self.datenwerte = pd.read_csv(csv_file)<br>
&gt;def <strong>len</strong>(self):<br>
&gt;&gt; return len(self.datenwerte)<br>
&gt;def <strong>getitem</strong>(self, idx):<br>
&gt;&gt;sample = self.datenwerte.iloc[idx,1]<br>
&gt;&gt;return sample</p>
<p>dataset1=myDataset(‘airline-passengers.csv’)<br>
data_iter1=torch.utils.data.DataLoader(dataset1, batch_size=1, shuffle=False)<br>
inhalte1 = iter(data_iter1).next()<br>
inhalte2 = iter(data_iter1).next()<br>
print(inhalte1,inhalte2)</p>
<p>-&gt; delivers tensor([112]) tensor([112])</p>
<p>-&gt; should: tensor([112]) tensor([118])</p>
<p>data (‘airline-passengers.csv’)<br>
Month  Passengers<br>
0  1949-01         112<br>
1  1949-02         118<br>
2  1949-03         132<br>
3  1949-04         129…</p>

Andi: <p>got it, I may call iter(…) only once, that´s it:<br>
it = iter(data_iter1)<br>
first = next(it)<br>
second = next(it)<br>
print(first,second)</p>
<p>and this one makes it iterable over the length:</p>
<pre><code class="lang-auto">from itertools import cycle,islice

class myAirlineDataset(IterableDataset):
    def __init__(self, data):
        self.data = data
    def process_data(self,data):
        for x in data:
          yield x
    def get_stream(self,data):
        return cycle(self.process_data(data))
    def __len__(self):
        return len(data)
    def __iter__(self):
       return (self.get_stream(self.data.iloc[:,1]))

data=pd.read_csv('airline-passengers.csv')
dataset1=myAirlineDataset(data) 
data_iter1=torch.utils.data.DataLoader(dataset1, batch_size=1)

#for batch in islice(data_iter1,144,146):
#  print(batch)

it = iter(data_iter1)
for n in range(145):
  a=next(it)
print(a)
</code></pre>
<p>br</p>
How to load .pth file which contains only 'model' and 'opt'

Mudassar: <p>I have .pth file which when loaded with torch.load() gives only ‘model’ and ‘opt’.<br>
File is actually build with fastai unet with resnet34.<br>
I don’t know the complete architecture of the model to re-create the model instance.<br>
I tried with<br>
model = torchvision.models.resnet34()<br>
state_dict = torch.load(‘file.pth’)<br>
model.load_state_dict(state_dict)</p>
<p>It’s not working, pth file is not matching with model</p>
<p>links:<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://cfl.dropboxstatic.com/static/images/favicon-vflUeLeeY.ico" class="site-icon" width="32" height="32">
      <a href="https://www.dropbox.com/s/zkehq1uwahhbc2o/ColorizeArtistic_gen.pth?dl=0" target="_blank" rel="nofollow noopener">Dropbox</a>
  </header>
  <article class="onebox-body">
    <img src="https://www.dropbox.com/static/images/spectrum-icons/generated/content/content-unknown-large.png" class="thumbnail onebox-avatar" width="160" height="160">

<h3><a href="https://www.dropbox.com/s/zkehq1uwahhbc2o/ColorizeArtistic_gen.pth?dl=0" target="_blank" rel="nofollow noopener">ColorizeArtistic_gen.pth</a></h3>

<p>Shared with Dropbox</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">
      <a href="https://github.com/jantic/DeOldify" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/495;"><img src="https://repository-images.githubusercontent.com/155636246/2da2db00-6ebe-11e9-8c46-17ef077c5639" class="thumbnail" width="690" height="495"></div>

<h3><a href="https://github.com/jantic/DeOldify" target="_blank" rel="nofollow noopener">jantic/DeOldify</a></h3>

<p>A Deep Learning based project for colorizing and restoring old images (and video!) - jantic/DeOldify</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


mariosasko: <p>If the dict returned by <code>torch.load('file.pth')</code> has two keys (you mention <code>model</code> and <code>opt</code>), you have to extract the value of the key <code>model</code> and then pass it to the model’s <code>load_state_dict</code> like that:</p>
<pre><code class="lang-python">model = torchvision.models.resnet34()
state_dict = torch.load('file.pth')
model.load_state_dict(state_dict['model'])
</code></pre>
<p>Alternatively, you can pass the whole <code>state_dict</code> to <code>load_state_dict</code> and set the flag <code>strict</code> to <code>False</code>:</p>
<pre><code class="lang-python">model.load_state_dict(state_dict['model'], strict=False)
</code></pre>
<p>It should work unless the structures of the models differ (the saved model is not an instance of resnet).</p>
Native implementation for pdist, Pearson correlation

Scott_Hoang: <p>I’m trying to calculate the similarity between two activation matrix of two different models following the Teacher Guided Architecture Search paper. My question is, does python has a native implementation of pdist similar to Scipy.spatial.distance.pdist? And a native application for Pearson correlation?</p>

mariosasko: <p>I assume you meant PyTorch, not Python. For pdist see <code>torch.nn.functional.pdist</code> and for Pearson correlation check out this <a href="https://github.com/pytorch/pytorch/issues/1254" rel="nofollow noopener">issue</a>.</p>

Scott_Hoang: <p>Thank you!<br>
do you know what metric pytorch for pdist? for example in Scipy.spatial.distance.pdist, you can select between cosine, euclidean, …etc</p>

mariosasko: <p>Check out the <a href="https://pytorch.org/docs/stable/nn.functional.html#pdist" rel="nofollow noopener">docs</a> for available metrics (in that regard pdist in PyTorch is very limited compared to the scipy’s counterpart).</p>
Questions after following transfer learning tutorial

Mark_McPherson: <p>I followed the tutorial at <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html#further-learning" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html#further-learning</a>. Everything ran ok, but I have some questions I hope the community can help me with. If this belongs in a different topic, apologies!</p>
<ol>
<li>
<p>The tutorial states that on GPU it takes less than a minute to train (15-25 mins on CPU). I ran it on an nVidia GTX Strix 1060 6GB. Not the fastest card of course, but I expected it to still run it in a matter of 2-3 minutes. However, it took between around 8.5 - 11.5 minutes, depending on the batch size I set.<br>
I had a feeling this may be due to my old HDD that was threatening to fail at any minute! So I replaced it with an SSD, which is much faster. This knocked the training time down by 40-50%, so around 5m on average. It that a training time that would be expected with my GPU?</p>
</li>
<li>
<p>Using either the old HDD or a new SSD, I observed the GPU to be very under-utilised. Is this to be expected with the example in the tutorial? If so, how might one go about increasing the utilisation? If someone can give or link to some code that would be great.</p>
</li>
<li>
<p>I tried out different batch sizes, both to observe the running time and also to see how it affected the convergence. I had thought that the bigger the batch size, the faster a model would converge and also each epoch would run faster. Is that usually true? In this case I found that increasing the batch size in steps all the way up to all the data (it is a fairly small dataset, so it did fit on my GPU) resulted in significantly worse convergence and actually took longer. The batch size used in the tutorial was 4. I tried using 2 and found it to perform worse again. So it seems there is something special about using batch size of 4, tho there should not be AFAIK. Anyone able to give some insight into why this would be?</p>
</li>
</ol>

ptrblck: <ol>
<li>
<p>Depending where the bottleneck in your current system is, ~5 minutes might be expected. I just reran the script on a V100 server (which should be of course a bit more powerful) and it finished in 34s.</p>
</li>
<li>
<p><a href="https://discuss.pytorch.org/t/how-to-prefetch-data-when-processing-with-gpu/548/19">This post</a> gives you a good overview about potential bottlenecks and their workarounds.</p>
</li>
<li>
<p>This effect is explained e.g. in <a href="https://arxiv.org/abs/1804.07612">Revisiting small batch training for deep neural networks</a>, which claims that the best performance is consistently obtained for batch sizes between 2 and 32.</p>
</li>
</ol>
Extremely high output values from the network (1e34) with no training with normalization

dvirginz: <p>Hello, after experimenting with multiple off-the-shelf and written from scratch networks I am starting to feel there is something wrong with my network without being able to understand what:<br>
My network</p>
<pre><code class="lang-auto">class MySubPixelCNN(nn.Module):
    def __init__(self, upscale_factor,num_features):
        super(MySubPixelCNN, self).__init__()

        self.relu = nn.ReLU()
        self.conv1 = nn.Conv2d(num_features, 64, kernel_size=5, stride=1, padding=2)
        self.bb1 = nn.BatchNorm2d(64)
        self.conv2 = nn.Conv2d(64, 64, kernel_size=3, stride=1, padding=1)
        self.bb2 = nn.BatchNorm2d(64)
        self.conv3 = nn.Conv2d(64, 32, kernel_size=3, stride=1, padding=1)
        self.bb3 = nn.BatchNorm2d(32)
        self.conv4 = nn.Conv2d(32, num_features*upscale_factor ** 2, kernel_size=3, stride=1, padding=1)
        self.pixel_shuffle = nn.PixelShuffle(upscale_factor)

    def forward(self, x):
        x = self.conv1(x)
        x = self.bb1(self.relu(x))
        x = self.conv2(x)
        x = self.bb2(self.relu(x))
        x = self.conv3(x)
        x = self.bb3(self.relu(x))
        x = self.conv4(x)
        x = self.pixel_shuffle(x)
        return x
</code></pre>
<p>My input images are textures (I.e not the ordinary Images), with a simply preprocessing to have them in <code>[0,1]</code>.<br>
Yet, sometimes (which makes the network immediately diverge)<br>
<code>model(input).max() &gt; 1e30</code><br>
As mentioned, I always add batch norm, and as the network is not that deep, I simply cannot understand what could’ve gone wrong.</p>

ZdsAlpha: <p>Shouldn’t it be num_features * 2 ** upscale_factor?</p>

dvirginz: <p>No, why? Is that might be the reason why the results vary from 0 to 1e34?</p>

dvirginz: <p>I am now 100% certain that it is a problem with cuda 11, When using another GPU I have with cuda10.1 it works fine. I cannot prove it unfortunately, but I now understand that it started when I installed cuda11 and <code>conda install pytorch=10.2</code> as previous posts here suggested it is fine. I assume it is considered a bug, how can I nail the problem and report it?</p>

ptrblck: <p>Could you post some information for this setup, i.e.:</p>
<ul>
<li>which GPU are you using</li>
<li>how did you install CUDA11 and which version exactly</li>
<li>which cudnn version are you using</li>
<li>which PyTorch commit are you using</li>
<li>are you seeing the extremely high loss after a certain number of iterations for random inputs in <code>[0, 1]</code>?</li>
</ul>
Pytorch rename labels of a trained model

Praveen_K: <p>I have a trained model. During training there was some problem with string characters. So i converted my labels into numbers like:</p>
<p><code>red : 0</code></p>
<p><code>blue: 1</code></p>
<p><code>green: 2</code></p>
<p>Now is it possible to rename my label back to actual label names.<br>
Hours of training. Would be  helpful if anyone has an idea.</p>
<p>Train and validate the model</p>
<pre><code>for epoch in range(1, epoch_num + 1):
    loss_train, acc_train = train(train_loader, model, criterion, optimizer, epoch)
    loss_val, acc_val = validate(val_loader, model, criterion, epoch)
    total_loss_val.append(loss_val)
    total_acc_val.append(acc_val)
</code></pre>
<p>Test Single Image:</p>
<pre><code>def eval_image(file_path):
    model = torch.load(file_path)
    model.eval()
    device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
    X = Image.open('red.jpeg')
    test_transforms = transforms.Compose([transforms.ToTensor()])
    image_tensor = test_transforms(X).float()
    image_tensor = image_tensor.unsqueeze_(0)
    input = Variable(image_tensor)
    input = input.to(device)
    output = model(input)
    index = output.data.cpu().numpy().argmax()
    print(index)
</code></pre>
<p>Once the training is done, The evaluation script cannot be made generic. I will have to pass following code</p>
<pre><code>idx_to_class = {
        0: "red",
        1: "blue",
        2: "green",
    }
    class_name = idx_to_class[index]
</code></pre>
<p>But I do not want to pass the above code. As my evaluation script needs to generic.</p>

ptrblck: <p>Could you explain a bit more what “generic” means in this case and what is not working for you using this mapping?</p>