akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Is that overfitting? IMDB dataset 50K

username: Ximing_Dong11104
cooked: <p>Hi everyone, when I train my model, the result looks like in this picture.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/a/3a456f4fc167968d7237a65434593bfff826822a.png" alt="Screen Shot 2021-06-02 at 2.26.52 PM" data-base62-sha1="8jusju5BT8hAmSRhlg4G8M92KF4" width="466" height="439"></p>
<p>I think this is not right, since the best epoch is 2 and but in epoch 3  valid loss is increase.  I think it is overfitting.<br>
here is my code<br>
‘’’<br>
class BERTGRUSentiment(nn.Module):<br>
def <strong>init</strong>(self,bert,hidden_dim,output_dim,n_layers,bidirectional,dropout):</p>
<pre><code>    super().__init__()

    self.bert = bert
    self.bert_drop = nn.Dropout(0.3)
    self.out = nn.Linear(768, 1)

def forward(self, ids, mask, token_type_ids):
    o2 = self.bert(ids, attention_mask=mask, token_type_ids=token_type_ids)[1]
    bo = self.bert_drop(o2)
    output = self.out(bo)
    return output
</code></pre>
<p>model.to(device)<br>
criterion = nn.BCEWithLogitsLoss()<br>
criterion = criterion.to(device)<br>
# optimizer = optim.Adam(model.parameters())<br>
param_optimizer = list(model.named_parameters())<br>
no_decay = [“bias”, “LayerNorm.bias”, “LayerNorm.weight”]<br>
optimizer_parameters = [<br>
{<br>
“params”: [<br>
p for n, p in param_optimizer if not any(nd in n for nd in no_decay)<br>
],<br>
“weight_decay”: 0.001,<br>
},<br>
{<br>
“params”: [<br>
p for n, p in param_optimizer if any(nd in n for nd in no_decay)<br>
],<br>
“weight_decay”: 0.0,<br>
},<br>
]</p>
<pre><code>num_train_steps = int(len(train_dataset) / batch_size * config.EPOCHS)
optimizer = AdamW(optimizer_parameters, lr=3e-5)
scheduler = get_linear_schedule_with_warmup(
    optimizer, num_warmup_steps=0, num_training_steps=num_train_steps
)
</code></pre>
<p>‘’‘’<br>
can you give me some ideas about it?</p>

username: ptrblck
cooked: <p>Yes, it seems your model is overfitting and the general recommendation would be to add more regularization to the model e.g. via dropout etc.</p>
<p>PS: you might want to change the title to get better support, as it’s a bit ambiguous at the moment <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: Ximing_Dong11104
cooked: <p>haha, thanks very much, <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>
How does pytorch faster rcnn model share features for RPN and fast R-CNN?

username: yichong96
cooked: <p>In the original faster rcnn paper, the authors describe object detection as a 2 stage module, whereby in the first stage, region proposals are generated by a CNN. In the second stage, the fast r-cnn model is used that takes as input these region proposals, having another CNN network by itself to carry out the object detection from the proposals.</p>
<p>In their paper, they describe a few approaches (Alternating training, Approximate joint training, Non-approximate joint training, 4-Step Alternating Training) that lead to a single shared CNN network. However, most of the approaches described involve training two separate CNN’s before sharing the CNN layers eventually.</p>
<p>Which approach described in the paper does the faster r-cnn model use to carry out it’s training? I would like to have an intuition of what happens under the hood during the training.</p>
Creating my own dataset in pytorch-geometric

username: spring
cooked: <p>Hi,</p>
<p>I am using the pytorch-geomeric package to implement GCN in pytorch.<br>
The simplest example on the pytorch-geomeric official homepage is a bit confusing to me, so I’m asking a question.<br>
(<a href="https://pytorch-geometric.readthedocs.io/en/latest/notes/introduction.html#data-handling-of-graphs" class="inline-onebox" rel="noopener nofollow ugc">Introduction by Example — pytorch_geometric documentation</a>)</p>
<blockquote>
<p>import torch<br>
from torch_geometric.data import Data</p>
<p>edge_index = torch.tensor([[0, 1, 1, 2],<br>
[1, 0, 2, 1]], dtype=torch.long)<br>
x = torch.tensor([[-1], [0], [1]], dtype=torch.float)</p>
<p>data = Data(x=x, edge_index=edge_index)<br>
print(data)</p>
</blockquote>
<p> → Data(edge_index=[2, 4], x=[3, 1])</p>
<p>In the code above, the [[-1], [0], [1]] list is considered to be embedded feature vectors of each node, but it seems that it is not specified which node’s feature vector each is. Since that is a list, the only information we can know is the value and index information. So, does the index become the node name?</p>
<p>That is, since ‘0’, ‘1’, ‘2’ appearing in edge_index are node names, is the feature vector of node ‘0’=[-1], the feature vector of node ‘1’=[0], and the feature vector of node ‘2’=[1]?</p>
Need help with multilabel, multiclass

username: Govert
cooked: <p>Hi,</p>
<p>I need some help advice for the following. I have a text classification problem, where i’m training for 2 labels (let’s call them A and B). The classes for A and B differ: A can be 0-4 and B can be 0-8 (A can be cast to 0-8 i guess).</p>
<p>How would i go about this problem? I can use a one-hot for each label, but then i don’t know how to calculate the loss.</p>

username: tom
cooked: <p>If I understand you correctly, you have two classification tasks (so it would be multi-task not multilabel in the usual lingo).</p>
<p>In that case, you can just output 14 logits and then group them for the loss if you want.<br>
More concretely, you could output a batch x 14 vector of scores (so no final activations) and organize your labels into two batch-sized tensors.<br>
Then for <code>F.cross_entropy_loss(output[:, :9], target_a) + F.cross_entropy_loss(output[:, 9:], target_b)</code>.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Govert
cooked: <p>Thanks, i’ll give it a try:</p>
<ul>
<li>concat the one-hots as input</li>
<li>aggregate the losses via the grouped outputs</li>
</ul>
Short-circuit multiplication by 1 and addition by 0

username: miaotianyi
cooked: <p>When a tensor is multiplied/divided by 1 or added/subtracted by 0, we expect it to remain the same. So the most intuitive implementation appears to be:</p>
<pre><code class="lang-auto">def my_add(a, b):
    if is_scalar(b) and b == 0:
        return a
    if is_scalar(a) and a == 0:
        return b
    return a + b
</code></pre>
<p>for some type-checking function <code>is_scalar</code>.</p>
<p>In other words, we can short-circuit these operations instead of performing element-wise addition by 0 on every element of a tensor. However, this is not the case in Numpy or PyTorch, which can be verified by the following script:</p>
<pre><code class="lang-auto">from timeit import timeit
n, c, h, w = 64, 3, 128, 128
arr = torch.rand(n, c, h, w, device="cuda")
t0 = timeit("arr", globals=globals(), number=1000)
t1 = timeit("arr.clone()", globals=globals(), number=1000)
t2 = timeit("arr + 0", globals=globals(), number=1000)
# t0 &lt; t1 &lt; t2
</code></pre>
<p>This can be especially time-consuming when a user inadvertently uses the following code as a basic building block in a network that deals with really large data batches, thinking it’s just an innocent summation:</p>
<pre><code class="lang-auto">def func(x: torch.Tensor):
    y = 0
    for i in range(n):
        y += do_something(x)
    return y
</code></pre>
<p>I see no reason against such short-circuiting (operators, autograd graphs, etc.) but I couldn’t find any discussion on this topic at all. Perhaps the improvement is too small for most users? I’m personally working on edge computing so every bit of time matters a lot.</p>
How is reduction performed in `F.cross_entropy` when `weight` parameter is provided?

username: Dazitu616
cooked: <p>I try to use rescaling <code>weight</code> in <code>torch.nn.functional.cross_entropy</code>, and find the result very hard to understand. Here is the code:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; import torch.nn.functional as F
&gt;&gt;&gt; pred = torch.tensor([[[0.8054, 0.6918],
         [0.8704, 0.1927],
         [0.4033, 0.3574],
         [0.6289, 0.2227],
         [0.0425, 0.8065]],

        [[0.4279, 0.4677],
         [0.4958, 0.3767],
         [0.3411, 0.9530],
         [0.4712, 0.7330],
         [0.9196, 0.8033]]]).float()  # [2, 5, 2], 5-way classification
&gt;&gt;&gt; label = torch.tensor([[2, 4], [1, 3]]).long()  # [2, 2]
&gt;&gt;&gt; weight = torch.tensor([0.0886, 0.2397, 0.1851, 0.2225, 0.2640]).float()  # weight.sum() == 1
&gt;&gt;&gt; loss1 = F.cross_entropy(pred, label, reduction='mean', weight=weight)
&gt;&gt;&gt; loss1
tensor(1.5594)
&gt;&gt;&gt; loss2 = F.cross_entropy(pred, label, reduction='none', weight=weight).sum() / label.numel()
&gt;&gt;&gt; loss2
tensor(0.3553)
&gt;&gt;&gt; loss3 = F.cross_entropy(pred, label, reduction='sum', weight=weight) / label.numel()
&gt;&gt;&gt; loss3
tensor(0.3553)
</code></pre>
<p>If I understand correctly, <code>loss1</code> should be the same as <code>loss2</code>. Obviously the reduction method is not what I suppose. So I wonder how <code>F.cross_entropy</code> performs mean reduction when <code>weight</code> is provided? Thanks!</p>

username: ptrblck
cooked: <p><code>nn.CrossEntropyLoss</code> normalizes with the used weights, so you would have to change the <code>loss2</code> calculation to:</p>
<pre><code class="lang-python">loss2 = F.cross_entropy(pred, label, reduction='none', weight=weight).sum() / weight[label].sum()
loss2
&gt; tensor(1.5594)
</code></pre>
<p><a href="https://discuss.pytorch.org/t/passing-the-weights-to-crossentropyloss-correctly/14731/10">This post</a> also describes it using another example.</p>

username: Dazitu616
cooked: <p>Thank you for your reply! That solves my problem.</p>
Question about lane detection

username: ysleer
cooked: <p>Hi.</p>
<p>I am developing on lane detection using CULane dataset.</p>
<p>I have a one question about lane detection.</p>
<p>In general, 5 classes(background + 4 lanes) are used for lane detection.</p>
<p>Some open source only use 4 lane classes without background class.</p>
<p>When learning, is there a big difference in performance between classifying the background and not doing it?</p>
Why DataParallel replicate the module during forward?

username: Luoshang_Lowson_Pan
cooked: <p>I’m reading the code about DataParallel but I don’t quite understand why module replicate happens during call to <code>forward</code>.<br>
I thought we should replicate module once and execute forward for multiple times? Or is this class design for training phase because the backward prop would require parameter averaging (synchronization) after each forward call?</p>
<ul>
<li>Code: <a href="https://github.com/pytorch/pytorch/blob/5b4c3a9da11120f60d732af505bc65f79df14637/torch/nn/parallel/data_parallel.py#L167" class="inline-onebox" rel="noopener nofollow ugc">pytorch/data_parallel.py at 5b4c3a9da11120f60d732af505bc65f79df14637 · pytorch/pytorch · GitHub</a>
</li>
</ul>

username: ptrblck
cooked: <p>The scatter and gather operations are used, as a “simple” data parallel implementation via <code>nn.DataParallel</code>. <a href="https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255">This blog post</a> explains the overall workflow in more details.<br>
This overhead is also why we recommend to use <code>DistributedDataParallel</code> with a single process per GPU, which would avoid these copies, and would thus yield the best performance.</p>
How to convert batch of graphs into batch tensor?

username: math_dude_2022
cooked: <p>I’m using PyTorch Geometric, and I’m interested in converting a batch of graphs into separate batches of node features and edge features. In particular, I’m interested in being able to separate the nodes along the dimension by which graph they belong to, but I’m only given the batches in <a href="https://pytorch-geometric.readthedocs.io/en/latest/notes/introduction.html#mini-batches" rel="noopener nofollow ugc">this format</a>.</p>
<p>Concretely, I have tensors <code>batch = Tensor([0, 0, 0, 0, 1, 1, 2])</code>, <code>nodes = Tensor([1, 2, 3, 4, 5, 6, 7])</code>, and I want to convert it to a tensor that looks something like <code>nodes = Tensor([[1,2,3,4], [5,6, pad, pad], [7, pad, pad, pad])</code> where I use <code>pad</code> to denote some padding tokens. In this case, I have turned a tensor of shape <code>(7,)</code> (number of nodes) into a tensor of shape <code>(3, 4)</code> (number of graphs in batch, maximum number of nodes per graph).</p>

username: stsan
cooked: <p>Is this what you’re looking for?<br>
<a href="https://pytorch-geometric.readthedocs.io/en/latest/modules/utils.html#torch_geometric.utils.to_dense_batch" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch-geometric.readthedocs.io/en/latest/modules/utils.html#torch_geometric.utils.to_dense_batch</a></p>
HDF keys and DatasetHDF5

username: saran
cooked: <p>how I can check the dimension of an HDF file  and is it still an hdf file  if its again passed  another dataset  HDF?<br>
I have an hdf file  which is again divided into trainset and testset  as shown below but when I am trying to check the dimension of hdf it says HDF5 has no attribute “keys”</p>
<p>trainset =          Dataset4DFromHDF5(args.data,</p>
<pre><code>                                 labels=(ref_type,),
                                                          
                                 device=loader_device,

                                 start=args.intervals[0], 
                                 end=args.intervals[1],
                                 crop=args.crop,
                                 augment=args.img_augm,
                                 augment_freq=args.freq_augm)

testset = Dataset4DFromHDF5(args.data,
                                labels=(ref_type,),
                                device=loader_device,
                                start=args.intervals[2], 
                                end=args.intervals[3],
                                crop=args.crop,
                                augment=False,
                                augment_freq=False)
</code></pre>
<p>list(trainset.keys())</p>
<p>AttributeError: ‘DatasetDeepPhysHDF5’ object has no attribute ‘keys’</p>

username: ptrblck
cooked: <p>I don’t know how <code>DatasetDeepPhysHDF5</code> is implemented, but would assume that it might be a custom <code>torch.utils.data.Dataset</code>, which wraps the HFD dataset internally, and is thus not implementing the <code>keys()</code> operation. You could check it via <code>print(type(trainset))</code> and check which methods are implemented for this class.<br>
If my guess is correct, you might be able to access the internal HDF dataset and call <code>keys()</code> on it directly.</p>

username: saran
cooked: <p>thanks for feedback i will check it</p>
Some basic hyper parameter ques

username: banikr
cooked: <p>What is the difference between epoch and iterations?<br>
In this 3D multi-class segmentation <a href="https://arxiv.org/pdf/1806.02237v1.pdf" rel="noopener nofollow ugc">paper</a> in section <strong>3 Experiments &amp; Results</strong> the authors mention <em>“…Training of 20,000 iterations.”</em> <br> Does it mean they ran 20,000 subvolume patches?</p>

username: ptrblck
cooked: <p>An epoch usually refers to using the entire dataset once, while an iteration usually refers to a training step using a batch.<br>
In case you know the batch size and number of sample in the <code>Dataset</code>, you could transform “Training of 20,000 iterations” into epochs.</p>
Image normalization transfer learning on greyscale images

username: yichong96
cooked: <p>I want to use the pre-trained fasterrcnn_resnet50 model for my own custom object detector. My own images are greyscale and I have about 100+ images of those. Using the pre-trained network, I have to normalise the images by the mean and std of image net. However, with greyscale images, I am not sure whether I should use the same normalisation values of image net. Since all my channels for greyscale all have the same values?</p>
How To Check Tensorboard Version when installed via pytorch?

username: seyeeet
cooked: <p>I have some issue with my tensorboard in pytorch, it does not update the updates as it goes on to further epochs.<br>
So I decided to check its version, but <code>from torch.utils.tensorboard import version</code> does not exist in pytorch, so I was wondeing how I can check the tensorboard that is available there.</p>

username: Williams
cooked: <p>Try typing which tensorboard in your terminal. It should exist if you installed with pip as mentioned in the tensorboard README <a href="https://www.mypascoconnect.website/" rel="noopener nofollow ugc">visit this site</a> ( although the documentation doesn’t tell you that you can now launch “tensorboard” without doing anything else.)</p>
Train Deeplabv3 on 4 channels images

username: Doha_Bouallal
cooked: <p>Hi there, i want to train deeplabV3 on my own Dataset with 4 channels images. but i didn’t find any PyTorch implementation of deeplabV3 where i could change parameters and input channels number of the model to fit my (4channels) images .<br>
How can i modify deeplabV3 to adapt it to my dataset?</p>

username: ptrblck
cooked: <p><code>torchvision</code> provides <code>deeplabv3</code> implementations <a href="https://pytorch.org/vision/stable/models.html#torchvision.models.segmentation.deeplabv3_resnet50">here</a> and you could manipulate the first conv layer as seen here:</p>
<pre><code class="lang-python">model = models.segmentation.deeplabv3_resnet50(pretrained=False, progress=True, num_classes=21, aux_loss=None)

x = torch.randn(2, 3, 224, 224)
out = model(x)

model.backbone.conv1 = nn.Conv2d(4, 64, 7, 2, 3, bias=False)
x = torch.randn(2, 4, 224, 224)
out = model(x)
</code></pre>
How to create a clean copy of a pretrained sub-network?

username: 111429
cooked: <p>Let’s say I have a pretrained autoencoder, and I just need the pretrained encoder as a part of a new model.</p>
<pre><code class="lang-auto">class autoencoder(nn.Module):
    def __init__(self):
        super(autoencoder, self).__init__()
        self.encoder = nn.Sequential(
            nn.Linear(28 * 28, 128),
            nn.ReLU(True),
            nn.Linear(128, 64),
            nn.ReLU(True), nn.Linear(64, 12), nn.ReLU(True), nn.Linear(12, 3))
        self.decoder = nn.Sequential(
            nn.Linear(3, 12),
            nn.ReLU(True),
            nn.Linear(12, 64),
            nn.ReLU(True),
            nn.Linear(64, 128),
            nn.ReLU(True), nn.Linear(128, 28 * 28), nn.Tanh())

    def forward(self, x):
        x = self.encoder(x)
        x = self.decoder(x)
        return x

    def get_encoder(self):
        return self.encoder

class autoencoder2(nn.Module):
    def __init__(self, encoder):
        super(autoencoder, self).__init__()
        self.encoder = encoder
        self.decoder = nn.Sequential(
            nn.Linear(3, 12),
            nn.ReLU(True),
            nn.Linear(12, 64),
            nn.ReLU(True),
            nn.Linear(64, 128),
            nn.ReLU(True), nn.Linear(128, 28 * 28), nn.Tanh())

    def forward(self, x):
        x = self.encoder(x)
        x = self.decoder(x)
        return x

    def get_encoder(self):
        return self.encoder


</code></pre>
<pre><code class="lang-auto">pretrn_AE = autoencoder()
checkpoint = torch.load('AE.pt', map_location='cpu')
pretrn_AE.load_state_dict(checkpoint['model'])

pretrn_encoder = pretrn_AE.get_encoder()

new_AE = autoencoder2(pretrn_encoder)
</code></pre>
<p>While training <code>new_AE</code>, will this cause warning or error saying that some trainable weights are not used?(the decoder in pretrn_AE).<br>
To play safe, I want to have a clean copy of the pretrained encoder to feed into <code>new_AE</code>.<br>
And while training the <code>new_AE</code>, it has nothing to do with the <code>pretrn_AE</code>.<br>
Any advice?<br>
Thanks.</p>

username: ptrblck
cooked: <p>Your workflow looks generally alright, but you could also use <code>copy.deepcopy</code> to create a new instance of the internal <code>encoder</code>.</p>
Colab issues when using mount drive

username: Ximing_Dong11104
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/f/3f8b4326eb21978269fcc1b7da278c9a8f14bc56.png" data-download-href="https://discuss.pytorch.org/uploads/default/3f8b4326eb21978269fcc1b7da278c9a8f14bc56" title="Screen Shot 2021-06-01 at 2.59.08 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3f8b4326eb21978269fcc1b7da278c9a8f14bc56_2_690x387.png" alt="Screen Shot 2021-06-01 at 2.59.08 PM" data-base62-sha1="948rP3JBbFTODWghMz9hHkNfyLk" width="690" height="387" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3f8b4326eb21978269fcc1b7da278c9a8f14bc56_2_690x387.png, https://discuss.pytorch.org/uploads/default/original/3X/3/f/3f8b4326eb21978269fcc1b7da278c9a8f14bc56.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/f/3f8b4326eb21978269fcc1b7da278c9a8f14bc56.png 2x" data-dominant-color="FDFDFD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-06-01 at 2.59.08 PM</span><span class="informations">731×411 9.01 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Hi everyone<br>
I am using the mount drive, but sometimes the screen will have this problem. how can I avoid it? thanks</p>

username: ptrblck
cooked: <p>This issue doesn’t seem to be PyTorch-related, so I would recommend to post this question in a Google/Colab discussion board or raise an issue in their GitHub repository.</p>

username: Ximing_Dong11104
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="123010">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Colab discussion board</p>
</blockquote>
</aside>
<p>could you give me a link ? thanks</p>

username: ptrblck
cooked: <p>Unfortunately not, as I don’t know where Colab questions are discussed. Maybe you could take a look at StackOverflow or check other Colab references.</p>
DCGAN issue with training with input size/shape

username: Power_Rangers
cooked: <p>Hello I am working on learning how to build GANS and was trying Pytroch DCGAN from both the training material and from a GitRepo I found.</p>
<p>I am using Kaggle to run this and this is my code right now<br>
<a href="https://www.kaggle.com/matthewmillar/gan-anime-face-generation-test" class="onebox" target="_blank" rel="noopener nofollow ugc">https://www.kaggle.com/matthewmillar/gan-anime-face-generation-test</a></p>
<p>Everything seems to be fine with my dataset and my dataloaders but not with the training. I copied the code as I was running into errors thinking it was my implementation of it, but still had the same issue.</p>
<pre><code class="lang-auto">RuntimeError: Expected 4-dimensional input for 4-dimensional weight [64, 3, 4, 4], but got 3-dimensional input of size [3, 64, 64] instead
</code></pre>
<p>I know what this means but when i look at what my dataloader is producing it is fine [batch_size, 3, 64, 64]<br>
So that should work out fine I would think.</p>
<p>I not quite sure what is going wrong with this.</p>
<p>Here are the place where I am working from for the code<br>
<a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html</a></p>
<p>Thank you for any help with this.</p>

username: ptrblck
cooked: <p>It seems the posted Kaggle Kernel runs without any issues or am I missing the error message?<br>
In any case, as you’ve already described, the batch dimension is missing, so you could check, if you are indexing the batch somewhere before passing it to the conv layer.</p>

username: Power_Rangers
cooked: <p>I was working on this for a bit, and found out what the error was and it was two things I was doing wrong.<br>
The first was the label i made needed to be a float like this</p>
<pre><code class="lang-auto">real_label = 1.0
fake_label = 0.0
</code></pre>
<p>And then by following the other code lead me down a wrong road<br>
The error came from this line</p>
<pre><code class="lang-auto">real_cpu = data[0].to(device)
</code></pre>
<p>And I fixed it by simple doing this</p>
<pre><code class="lang-auto">real_cpu = data.to(device)
</code></pre>
<p>I still dont know how to tutorial and the github code  I looked at worked then.<br>
I appreciate you looking at this.</p>
Permute/reshape, view difference

username: leftkappa48
cooked: <p>Hello, so in my code I have a tensor of size[1,8,64,1024].<br>
Let’s say I want to reshape it to its original size, that is [1,512,1024].</p>
<p>So I want to “integrate” (this is not exactly the word) 8x64 dimensions to one dim of 512.<br>
I used <code>view(*(1,512,1024))</code> to get from [1,8,64,1024] back to [1,512,1024].<br>
But then I was experimenting to understand torch functions and then with<br>
<code>permute(0, 2, 1, 3)</code> followed by <code>reshape(1, 512, 1024)</code> I had the same result.</p>
<p>The results I get are equal, checking with torch.eq(). But what is better to use for less complexity ?</p>
<p>Thanks a lot</p>

username: eqy
cooked: <p>I’m confused that you get the same results here. (e.g., in this code snippet the results are clearly not the same)</p>
<pre><code class="lang-auto">$ cat temp.py
import torch

a = torch.randn(1, 8, 64, 1024)
b = a.reshape(1, 512, 1024)
c = a.permute(0, 2, 1, 3).reshape(1, 512, 1024)
a = a.view(1, 512, 1024)

print(torch.allclose(a,b))
print(torch.allclose(b,c))
$ python3 temp.py
True
False
$
</code></pre>
<p>In summary permute is very different from view and reshape in that it actually changes the data layout or <em>ordering</em> of elements (e.g., consider what happens as you access each element by incrementing the last index by 1).</p>
<p>This post <a href="https://discuss.pytorch.org/t/for-beginners-do-not-use-view-or-reshape-to-swap-dimensions-of-tensors/75524">For beginners: Do not use view() or reshape() to swap dimensions of tensors! - PyTorch Forums</a> is a great intro to the pitfalls of using <code>view</code> or <code>reshape</code> when the intent is to change the <em>ordering</em> of elements.</p>
Distributed data parallel slowed by cpu

username: Mox_P
cooked: <p>Hi, I’m new to distributed data parallel (DDP) module.</p>
<p>I have some questions one my model in which I use syncBatchNorm start to train on 2GPU in a single node, it seems to use a lot of CPU cores, and seems limited by CPU.</p>
<p>I tried to profile my code and the two most CPU time consuming operations are “to” (which is called around 4000times ) and “syncBatchNorm”.</p>
<p>So, I don’t know if it’s normal ? And if any optimization are possible ?</p>
<p>Thanks for your help !</p>

username: wanchaol
cooked: <p>Do you have a code snippet that reproduces the issue? From the description I guess most of the profiler results are from moving your model and parameters from CPU to GPU, instead of doing the actual training. Can you profile it after move the model to GPU? The model should be converted to GPU before wrapping it with DDP, see the <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" rel="noopener nofollow ugc">DDP tutorial</a></p>
F.relu and nn.ReLU

username: Shawn_Zhuang
cooked: <p>Hi,<br>
in pytorch lightning, recently I found that if I use F.Dropout in forward step, even when I set mode to model.val, the dropout still work, then I realize I should replace it with nn.Dropout as module attribute. After that, everything performs normally.</p>
<ol>
<li>Then my concern would be like, if I use F.relu in forward instead of nn.ReLU as attribute, can the model performs normally in backward, i.e. considering there is a relu or relu just work in forward?</li>
<li>and if I want to use twice nn.ReLu, should I define one self.relu=nn.ReLU and use self.relu in 2 different position or should I define 2, i.e. self.relu1 = nn.ReLU, self.relu2 = nn.ReLU</li>
</ol>
<p>So basically I am not very clear how the backpropagation find which module( or function) it should consider.</p>

username: KFrank
cooked: <p>Hi Shawn!</p>
<aside class="quote no-group" data-username="Shawn_Zhuang" data-post="1" data-topic="123071" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shawn_zhuang/48/30094_2.png" class="avatar"> Shawn_Zhuang:</div>
<blockquote>
<p>if I use F.relu in forward instead of nn.ReLU as attribute, can the model performs normally in backward</p>
</blockquote>
</aside>
<p><code>F.relu()</code> (which is to say <code>torch.nn.functional.relu()</code>) is a<br>
function.  <code>nn.ReLU</code> (<code>torch.nn.ReLU</code>) is a class that simply calls<br>
<code>F.relu()</code>.  These two ways of packaging the function do the same<br>
thing, including when calling <code>.backward()</code>.</p>
<aside class="quote no-group">
<blockquote>
<p>if I want to use twice nn.ReLu, should I define one self.relu=nn.ReLU and use self.relu in 2 different position or should I define 2, i.e. self.relu1 = nn.ReLU, self.relu2 = nn.ReLU</p>
</blockquote>
</aside>
<p>There is no need to instantiate two instances of the <code>nn.ReLU</code> function<br>
object (but you can if you want).  An instance of <code>nn.ReLU</code> doesn’t<br>
contain any state, so whether you have two instances or only one,<br>
they all do the same thing, simply calling <code>F.relu()</code>.  (My preference<br>
would be to instantiate only one <code>nn.Relu</code> function object, or if I didn’t<br>
need an object instance, simple call <code>F.relu()</code>.  But it really doesn’t<br>
matter.)</p>
<p>Best.</p>
<p>K. Frank</p>

username: Shawn_Zhuang
cooked: <p>Thank you! It is very clear.</p>
Using a per-channel mean and std to create tensor

username: lnair
cooked: <p>Is there a way to create a tensor using a per-channel means and std? For instance, I have two tensors <code>mean</code> and <code>std</code> with <code>k</code> values each, and I would like to generate an <code>output</code> tensor which is a stack of <code>k</code> tensors generated from normal distributions using each mean and std value. What would be an efficient way to do it? Essentially, I would like to do the following:</p>
<pre><code class="lang-auto">output = torch.empty(shape)
for i in range(k):
    output[:, i, ...] = output[:, i, ...].normal_(mean=mean[i], std=std[i])
</code></pre>
<p>Is there a better way of achieving this? Thanks in advance!</p>
Custom grad_fn when operation in forward pass detaches from computation graph

username: rainbow_soup
cooked: <p>Hi, I’m trying to make a custom backward function for argsort. My actual code is a bit more complex, but this is basically what I want to run:</p>
<pre><code class="lang-auto">import torch

class TestSorter(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x):
        y = torch.argsort(x)
        ctx.save_for_backward(x, y)
        return y

    @staticmethod
    def backward(ctx, grad_output):
        x, y = ctx.saved_tensors
        return x * y


testsort = TestSorter.apply


if __name__ == '__main__':
    x = torch.tensor([5., 4, 3, 2, 1], requires_grad=True)
    y = testsort(x)
    y.sum().backward()
    print(y, x.grad)
</code></pre>
<p>This throws <code>RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</code>. If I change <code>y = torch.argsort(x)</code> to <code>y = 0 * x + torch.argsort(x)</code> in the forward pass, it works, but this can hardly be the solution. How do I properly do this?</p>
<p>I know the fix is probably really simple, but I just started using PyTorch, was basically thrown in cold water. Any help would be appreciated! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: soulitzer
cooked: <p>Hmm wouldn’t the derivative of argsort be zero everywhere (except the points where two elements are tied, in which it wouldn’t be differentiable)?</p>
<p>What is your use case for something like this? Why not just perform argsort on a tensor that does not require grad?</p>

username: rainbow_soup
cooked: <p>That is true. I’m trying to implement the result of a paper that describes a method to reasonably approximate a backward pass, so even if you have a discrete function like argsort I can define an alternative derivative through which it makes sense to backpropagate. I know the example doesn’t make a lot of sense like this, though. However, I figured there has to be a way to get this working. I don’t really care if this is the proper derivative of argsort.</p>

username: rainbow_soup
cooked: <p><strong>Solution:</strong> change x = torch.tensor([5., 4, 3, 2, 1], requires_grad=True) to x = torch.tensor([5., 4, 3, 2, 1], requires_grad=True).float()</p>
Issue with torchvision.io.video.read_video

username: ragnaarok
cooked: <p>Hi, I’m trying to use torchvision.io.video.read_video but I’m noticing some weird behavior and I wonder if there’s an issue with this method or if I just don’t understand something. Torchvision version is up to date (0.9.1).</p>
<p>Situation: I want to read a 30 seconds clip (audio + video), 25 fps, audio at 44100Hz</p>
<p>I use read_video(pth), everything works fine, outputs shape [750, …], [2, 1323008]<br>
– maths: 25frame * 30sec=750, 44100samples*30sec=1323000<br>
I use read_video(pth, pts_units=‘sec’), same<br>
I use read_video(pth, 0, 10, pts_units=‘sec’), shapes are [251, …], [2, 12]<br>
I use read_video(pth, 0, 12500, pts_units=‘pts’), shapes are [25, …], [2, 13312]</p>
<p>I don’t really understand the two last outputs. For me it should be something like [251, …], [2, 441000], and [25, …], [2, 44100].<br>
I would like to be able to only read parts of the video.<br>
Thanks for any help</p>

username: ragnaarok
cooked: <p>Ok, using</p>
<p><code>torchvision.set_video_backend('video_reader')</code></p>
<p>solved my issue for the “sec” unit. I now have [251, …], [2, 441344].<br>
And the “pts” unit seems to be depreciated so it’s not important.<br>
Seems to be solved.<br>
Though it says in the documentation I should compile torchvision from source but I didn’t and it still works, maybe something will be wrong later</p>
When I load a .pth model, why it retrain the model?

username: forcefulowl
cooked: <p>after training the model, I use <code>torch.save</code> to save the model with <code>.pth</code> format.<br>
And in another python file, I try to use <code>torch.load</code> to load the model and do the prediction, but seems like it retrain the model and then do the prediction,<br>
Can someone explains that for me ?</p>

username: ptrblck
cooked: <p>What do you mean by “but seems like it retrain the model and then do the prediction”?<br>
After loading your model, you usually set it to evaluation mode using <code>model.eval()</code>, since some layers behave differently.<br>
<code>BatchNorm</code> and <code>Dropout</code> are examples of the different behavior between training and evaluation.</p>
<p>Could you explain a bit more?</p>

username: forcefulowl
cooked: <p>thanks for your reply, for e.g., I trained a model with dataset mnist, while training, I set 5 epoch, and print something after each epoch, and the end of the code, I use <code>torch. save(cnn,'path')</code> to save the model with <code>.pth</code> format. And in another python file, when I use <code>model = torch.load('path')</code> to load the model, it prints from the first epoch (train again?). So that’s the point I’m confused with.<br>
Thanks again for your reply.</p>

username: ptrblck
cooked: <p>Could you try to save and load the <code>state_dict</code>?<br>
You can find some information <a href="http://pytorch.org/docs/master/notes/serialization.html">here</a>.</p>

username: forcefulowl
cooked: <p>I’ll try now, thanks for the information.</p>

username: Schrodinger
cooked: <p>Hi, I know it’s been years, but do u find a solution for this? I’m stuck with the same issue although I did load the state_dict as the documentation stated.</p>
Incorrect gradient for combined network

username: axolotl
cooked: <p>Hi there. I am using combined network made of pre-trained ESM transformer by FAIR and my own classifier. I should mention that I changed first layer of transformer to identity layer because i want to calculate embeddings with embedd. layer of this transformer separately in order to be able to manipulate them later.<br>
I want to calculate gradient for the input however, when i use torch autograd I get a wrong one. I checked it by manualy calculating grad for one position using f(x+e)-f(x-e))/2e where f- is my combined network, x - specific position of input, e - small increment.  I am not sure what exactly was wrong with my automatic diff.<br>
here is my code:</p>
<pre><code class="lang-auto"># ======== split the transformer model in two, getting access to the embedding layer=====
splitted_model = []
for name, module in transformer.named_children():
    splitted_model.append(module)

# take the first layer
embedding_layer = splitted_model[0]
embedding = embedding_layer(input_tokens)
#replace the embedding layer with an Identity layer
identity_layer = torch.nn.Identity()
transformer.embed_tokens = identity_layer
#set token dropout to False
transformer.args.token_dropout=False

# this will be an input to my combined network
input_embedding = embedding_layer(input_tokens)

# ========. combine transformer and my classifier. ==============
# create class
class FullModel(nn.Module):
    def __init__(self, transformer, classifier_nn):
        super(FullModel, self).__init__()
        self.transformer = transformer
        self.classifier_nn = classifier_nn
        
    def forward(self, x):
# calculate representation matrix of a shape L, E. L - length on input sequence, E - length of feature verctor. token 0 is a start-of-sequence token, so the first symbol of input is token 1
        x1 = self.transformer(x, repr_layers=[34])["representations"][34][0,1 : len(x[0]) + 1]
# average over L to get representation vector of length E
        x2 = torch.mean(x1, dim=0)
# use it for the classification
        x3 = self.classifier_nn(x2)
        return x3
# combine
transformer_model = transformer
classifier_nn = my_classifier
final_model = FullModel(transformer_model, classifier_nn)

# ======== compute gradient =========
# define for which output class i want to get gradient
external = torch.tensor([1,0,0]) #1st out of 3 possible

x = Variable(input_embedding, requires_grad=True)
pred = final_model(x)
pred.backward(gradient=external)
input_gradient = x.grad
</code></pre>

username: soulitzer
cooked: <p>Have you checked that the gradient is correct for the non-combined ESM transformer on its own using your finite differencing method? Its possible that for large models the input is unstable at the point you sampled.</p>
Writing PyTorch style pseudocode

username: ashwani-bhat
cooked: <p>Hi all. In few research papers I have found that pseudocode for an algorithm is written in PyTorch style. Is there any format for writing these PyTorch style pseudocode. If yes, kindly post the related link.</p>
<p>PS: It would be beneficial if you also post some more research papers with PyTorch style pseudocode.</p>
<p>Thank you</p>

username: TinfoilHat0
cooked: <p>can you share some of these papers you mentioned so we can see what the style looks like?</p>

username: KamWoh_Ng
cooked: <pre><code class="lang-auto">\usepackage[ruled,vlined]{algorithm2e}

\definecolor{commentcolor}{RGB}{110,154,155}   % define comment color
\newcommand{\PyComment}[1]{\ttfamily\textcolor{commentcolor}{\# #1}}  % add a "#" before the input text "#1"
\newcommand{\PyCode}[1]{\ttfamily\textcolor{black}{#1}} % \ttfamily is the code font

...

\begin{algorithm}[h]
\SetAlgoLined
    \PyComment{this is a comment} \\
    \PyComment{this is a comment} \\
    \PyComment{} \\
    \PyComment{going to have indentation} \\
    \PyCode{for i in range(N):} \\
    \Indp   % start indent
        \PyComment{your comment} \\
        \PyCode{your code} \PyComment{inline comment} \\ 
    \Indm % end indent, must end with this, else all the below text will be indented
    \PyComment{this is a comment} \\
    \PyCode{your code}
\caption{PyTorch-style pseudocode for your-algo}
\label{algo:your-algo}
\end{algorithm}

</code></pre>
<p>I use the above latex code to construct a Pytorch-Style (more to a Python-Style) pseudocode algorithm table.</p>