akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Net stops to learn when number of classes is increased

mmisiur: <p>I’m kind of stuck, and instead of trying to randomly shoot the net with my ideas maybe I can consult it with you (one epoch takes 7h, so I can’t test my random ideas). Here’s the crime scene:</p>
<ol>
<li>My objective is to train a VGG-family net on specific custom moderately-large dataset (4.3 mln images, 7205 classes).</li>
<li>Since 1 epoch takes 7h to calculate (on whole dataset), I’ve tuned the hyperparameters on 300 classes (approx. 200 000 images). The net gets about 50% top-1 accuracy after 40 epochs, which is ok for me (learning curve attached), and also does pretty good on 1000 (and 2000) classes (on the pic).</li>
<li>Now I’m prepared for the big heist and I’m training the net on whole dataset. But now the net doesn’t really learn anything, with the accuracy oscillating slightly above random, even after 44 epochs (yep, 13 days of training). Predicted lables of the net are always the same (i.e. 5494, 5494, 5494, 5494, 5494, …) , sometimes after few epochs the prediction change but always for one lable.</li>
<li>Specs:
<ul>
<li>batch size: 64</li>
<li>learning rate: epochs [0-3]: 0.01, [4-7]: 0.001, [8- …]: 0.0001</li>
<li>CrossEntrpyLoss with class weightening to prevent overrepresented classes to mess with weights</li>
<li>optimizer: SGD, momentum=0.9, w/o weight decay</li>
<li>net trained from scratch<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/4/47d71a2d4ab3a93a626585369931b31eed070c81.png" alt="Pm00m" data-base62-sha1="afwKNF1QaPCsIYsBcEs3XrVWfDP" width="420" height="315"></li>
</ul>
</li>
</ol>
<p>The net architecture and training is exactly the same (except ofc last FC layer which size I’ve changed from 300 to 7205). Do you have any ideas on this? To small FC layers? Wrong learning rate? What Am I missing?</p>

RuFAI: <p>Maybe your learning rate is too high. I had some similar issues in relation to learning progress and more classes. If this is the solution, the reason for that is that gradient descent does not work properly an ignores local minima. Please let me know.</p>

mmisiur: <p>If the problem is local minima, decreasing learning rate won’t help i think (correct me if I’m wrong).</p>
<p>Also it would be strange if the problem would be local minima -the loss is not decreasing at all, so I would be v. unlucky if I started training with random weigths from local minima.</p>
<p>I don’t think this is the problem, but I’ll try messing with learning rate both dircetions.</p>
How to implement multiple loss

Rojin: <p>Let’s say that I have two MLP networks with one hidden layer each and size 100 that I would like to train simultaneously. Then I would like to implement 3 loss functions for this model, one for MLP1, one for MLP2, and one for 2 specific nodes from MLP1 and MLP2.<br>
Any help is appreciated</p>

Oli: <p>Hello!</p>
<p>I wrote a little code that <em>might</em> work. Try it out and report back <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:">.</p>
<pre><code class="lang-auto">model1 = Net(...)
model2 = Net(...)

all_params = list(model1.parameters()) + list(model2.parameters())
optimizer = torch.optim.Adam(all_params)

loss = torch.nn.CrossEntropyLoss() # or other loss
loss_for_specific_nodes = None # whatever loss you want

outp1 = model1(inputs)
outp2 = model2(inputs)

loss1 = loss(outp1, labels)
loss2 = loss(outp2, labels)

special_node_output = ...
special_node_labels = ...
special_loss = loss_for_specific_nodes(special_node_output, special_node_labels)

final_loss = loss1 + loss2 + special_loss
final_loss.backward()
optimizer.step()
</code></pre>

Rojin: <aside class="quote no-group" data-username="Oli" data-post="2" data-topic="45749">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oli/48/9773_2.png" class="avatar"> Oli:</div>
<blockquote>
<p>parameters</p>
</blockquote>
</aside>
<p>Thanks for the code, but my question is how I can choose the nodes?</p>

Oli: <p>Well that’s hard to answer since you never said which nodes you want to choose. Please elaborate on this <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Rojin: <p>Sure! sorry about that, let’s say that nodes are ordered as 0,1,2,3,…100 in mlp1 and 0,1,2,3,100 in mlp2. Then I choose nodes 0,1,2,3 from mlp1 and 0,1,2,3 from mlp2, and I would like to have a separate loss for those specific nodes, so I want to see if there is a way to select those nodes?</p>

Oli: <p>There sure is <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
<pre><code class="lang-auto">outp1 = model1(inputs) # is of shape (batch_size, 100)
outp2 = model2(inputs)

indexes = torch.tensor([0,1,2,3])

spec1 = outp1.index_select(dim=1, index=indexes) # Selects nodes at indexes over the whole batch
spec2 = outp2.index_select(dim=1, index=indexes)
</code></pre>
<p>Here is isolated code that you can run to experiment.</p>
<pre><code class="lang-auto">import torch

outp1 = torch.randn(2, 5)
outp2 = torch.randn(2, 5)
indexes = torch.tensor([0,1,2])

spec1 = outp1.index_select(dim=1, index=indexes)
spec2 = outp2.index_select(dim=1, index=indexes)


print("Output1: \n%s\n" % outp1)
print("Output2: \n%s\n" % outp2)
print("Selected1: \n%s\n" % spec1)
print("Selected2: \n%s\n" % spec2)

</code></pre>

Rojin: <p>thank you so much!!!</p>
Test time Augmentation Method?

wormschu: <p>In test time augmentation, is it simply to convert a tensor to a PIL image, then convert it (Flip, Rotation) and then convert it to a tensor and put it into the model?</p>

JuanFMontesinos: <p>You shouldn’t apply any transformation  but normalization at the time of testing</p>

wormschu: <p>Thank you for answering my question.<br>
What is mean ‘shouldn’t apply any transformation’?<br>
Could you tell me a little more specifically?</p>
[resolved] Synchronize BatchNorm mean and variance across gpu

HANG_ZHANG: <p>Is there a way to synchronize the BatchNorm layer across different GPU to calculate mean and variance during the training?</p>

HANG_ZHANG: <p>I figured it out.</p>
<p>Sync mean and variance for forward pass.<br>
Sync gradMean and Variance during the backward pass.</p>

zhanghang1989: <p>Forgot to update. The code is available here <a href="http://hangzh.com/PyTorch-Encoding/syncbn.html" rel="nofollow noopener">http://hangzh.com/PyTorch-Encoding/syncbn.html</a></p>

JayanthRR: <p>I am facing the same issue and would like to know your solution, but the code is not found on the link you mentioned. Could you please update the link with the code to solve this issue?</p>
What does batch_size argument in PyTorch mean?

scarecrow21: <p>For example, If I have a dataset with 10 rows. I want to train an MLP/RNN/CNN on this using mini batches.</p>
<p>So, let’s say, I take 2 rows at a time to train. 2 x 5 = 10. So, I train my model with batches where each batch contains 2 rows. So, number of batches = 5 and number of rows per batch is 2.</p>
<p>Is my batch_size 2? or is it 5?</p>
<p>In the dataloader, should my batch_size = 2 or batch_size = 5?</p>

InnovArul: <p>In your example, <code>batch_size=2</code>.<br>
Once you give the <code>batch_size</code>, <code>Dataloader</code> will take care of splitting the dataset into batches (5 in your case) and provide it to you.</p>

scarecrow21: <p>Thank you for clarifying! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

scarecrow21: <p>The same definition of batch_size holds in case of an RNN as well?</p>

InnovArul: <p>Yes. The same definition of <code>batch_size</code> applies to the RNN as well.<br>
But the addition of <code>time steps</code> might make things a bit tricky (RNNs take input as <code>batch x time x dim</code> as input, assuming all the data instances in the batch are padded to have same number of time steps).</p>
<p>Also, take care of <code>batch_first=True/False</code> option in RNNs.</p>
How to do Modulo transform on an image stored as a tensor

Deeply: <p>I need to do a modulo transform on each value of an image that is stored as a <code>tensor</code>. For an image stored in PIL format, this can be done on the red channel using:</p>
<pre><code class="lang-auto">Cr = 50; maxV = 255
red, green, blue = img.split()    
red = red.point(lambda p: (p+Cr) % max_V)
</code></pre>
<p>Not sure if there is a way to do the same in <code>torch</code>, without converting the image back to <code>PIL</code> and forth to <code>torch tensor</code>. The problem is that the <code>tensor</code> values are float and usually have -ve and +ve values. I can assume, for example, that  <code>maxV = 1</code>  and <code>Cr = 0.2</code>.</p>
How to forward dynamic layers properly?

abeytheo: <p>Hi, so I am working on a custom CNN module which needs to forward some inputs through several dynamic layers. This module has three loss functions which need to be back-propagated.</p>
<p>The following is a snippet of my network module:</p>
<pre><code class="lang-auto">class ProtoModule(nn.Module):
    def __init__(self,num_classes,num_proto):
        super(ProtoModule,self).__init__()
        ...
        self.proto_layers = nn.ModuleList()

        for c in range(self.num_classes):
            for p in range(self.num_proto):
                self.proto_layers.append(
                    nn.Sequential(
                        nn.Conv2d(chan,chan,kernel_size=(3,3), stride=(1, 1)),
                        nn.ReLU(inplace=True)
                    )
                )

        self.fc = nn.Linear(self.num_classes * self.num_proto, self.num_classes)
    
    def forward(self, x):
        ...
        proto = Variable(torch.Tensor(),requires_grad=True)
        
        for proto_layer in self.proto_layers:
            res = proto_layer(x)
            res = F.max_pool2d(res, kernel_size=res.size()[2:])
            res = max_pool_channel(res,res.size()[1])
            proto = torch.cat((proto,res))

        proto = proto.reshape(curr_batch_size, self.num_classes * self.num_proto)
        x = self.fc(proto)

        return x, proto
</code></pre>
<p>And here is how I currently back-propagate the loss:</p>
<pre><code class="lang-auto">...
with torch.set_grad_enabled(phase == 'train'):
                output, proto_output = model(input)
                _, preds = torch.max(output, 1)
                
                loss1 = cross_entropy_loss(output, labels)
                loss2 = cluster_loss(proto_output, labels)
                loss3 = separation_loss(proto_output, labels)
                
                total_loss = loss1 + 0.5 * loss2 + 0.5 * loss3
                
                if phase == 'train':
                    total_loss.backward()
                    optimizer.step()
...
</code></pre>
<p>The problem is my network won’t converge and I think this has something to do with the forward through the dynamic layers. Did I do the forwarding correctly?</p>
<p>Please advise. Thanks!</p>
ImageFolder rotates images by 90 degrees to the left

cmplx96: <p>Hi all,</p>
<p>I found out that when loading images with the ImageFolder dataset, the images are rotated by 90 degrees to the left.</p>
<p>The following code opens a single images from the specified folder, permutes the axis such that the format is H x W x C and then displays the image.</p>
<pre><code class="lang-auto">data = torchvision.datasets.ImageFolder('images', torchvision.transforms.Compose([torchvision.transforms.ToTensor()]))
data_loader = torch.utils.data.DataLoader(data, batch_size=1, shuffle=True, num_workers=2)

x, _ = next(iter(data_loader))
x = x.permute(0, 3, 2, 1)

x = x[0].numpy()
im = Image.fromarray(np.uint8(x * 255))
im.show()
</code></pre>
<p>The image is rotated by 90 degrees to the left. I am wondering why this is the case. Is this on purpose? Or did I use ‘permute’ incorrectly?</p>
<p>Cheers!</p>

ptrblck: <p>Image tensors are usually loaded as <code>[batch_size, c, h, w]</code>.<br>
Could you try to permute them as <code>.permute(0, 2, 3, 1)</code> to get <code>[batch_size, h, w, c]</code> and check your code again?</p>
tf.nn.embedding_lookup_sparse C++

songziqin: <p>I have two questions:<br>
question one : I decide to use libtorch for my training problems, and I want to use the function that have implemented in tensorflow,that is sparse tensor embedding. and I want to know if libtorch have the same interface.</p>
<p>question two: tensorflow have the function tf.concat<br>
,and I want to know if libtorch have the similar interface.</p>
Global Average Pooling Implementation

J_Na: <p>Any differences between <code>nn.AvgPool2d(kernel_size = 8)</code> and <code>nn.AvgPool2d(kernel_size = 8, stride = 1)</code> ?</p>

vabh: <p>The default <code>stride</code> value is equal to the <code>kernel_size</code>. So, in general, yes there is a difference.<br>
<a href="https://pytorch.org/docs/stable/nn.html#torch.nn.AvgPool2d" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html#torch.nn.AvgPool2d</a></p>
<p>However, for global average pooling it probably doesnt make a difference because you’re averaging values over the entire input feature map to produce a single value.</p>
Install TorchText using Conda

Trix: <p>Is it possible to install torchtext using conda on a linux machine?</p>
<p>I’m having major issues installing SpaCy on a remote machine given that it tries to use root permissions for some tasks. To fix this I installed Spacy and Torch using conda however I also require torchtext which does not seem to have a conda package.</p>
<p>I’m using spacy as the tokenizer in torchtext.</p>
<p>Is there a solution for this?</p>

rcshubhadeep: <p>I am presently trying to do the same. I only way I found was this -</p>
<pre><code class="lang-auto">conda install -c derickl torchtext
</code></pre>
<p>I am hoping that this channel is maintained and the package we get is updated.</p>
Tensorboard plugins on pytorch

mahsa: <p>I know there are some versions of tensorboard to use with pytorch (e.g. <a href="https://github.com/torrvision/crayon" rel="nofollow noopener">this</a>, <a href="https://github.com/lanpa/tensorboardX" rel="nofollow noopener">this</a>), but is there any way to use tensorboard plugins with pytorch?<br>
Thanks</p>

eplu: <p>I’d be interested to know if any plugins are available for the built in tensorboard integration (torch.utils.tensorboard). Particularly, is there anyway to use the HParams tensorboard plugin with Pytorch?</p>
Understanding Feature Pyramid networks

Jaideep_Valani: <p>Hi<br>
Can some one help understand the what are Feature pyramid networks introduced in the RetinaNet for object detection .<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://blog.zenggyu.com/img/icon.png" class="site-icon" width="32" height="32">
      <a href="https://blog.zenggyu.com/en/post/2018-12-05/retinanet-explained-and-demystified/" target="_blank" title="12:00AM - 05 December 2018" rel="nofollow noopener">NickZeng|曾广宇 – 5 Dec 18</a>
  </header>
  <article class="onebox-body">
    

<h3><a href="https://blog.zenggyu.com/en/post/2018-12-05/retinanet-explained-and-demystified/" target="_blank" rel="nofollow noopener">RetinaNet Explained and Demystified | NickZeng|曾广宇</a></h3>

<p>Introduction Recently I have been doing some research on object detection, trying to find a state-of-the-art detector for a project. I found several popular detectors including: OverFeat (Sermanet et al. 2013), R-CNN (Girshick et al. 2013), Fast...</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://cdn-static-1.medium.com/_/fp/icons/favicon-rebrand-medium.3Y6xpZ-0FSdWDnPM3hSBIA.ico" class="site-icon" width="32" height="32">
      <a href="https://medium.com/@14prakash/the-intuition-behind-retinanet-eb636755607d" target="_blank" title="06:30PM - 23 March 2018" rel="nofollow noopener">Medium – 23 Mar 18</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:404/500;"><img src="https://cdn-images-1.medium.com/max/1200/1*eljlGZ8X6HERYWumuCtRVg.png" class="thumbnail"></div>

<h3><a href="https://medium.com/@14prakash/the-intuition-behind-retinanet-eb636755607d" target="_blank" rel="nofollow noopener">The intuition behind RetinaNet</a></h3>

<p>The end goal of this blog post is to make readers intuitively understand the deep working of RetinaNet.</p>

  <p><span class="label1">Reading time: 10 min read</span>
    </p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
Feeding several datasets(with different lengths) seperately into your model

mina: <p>Hello all<br>
I am new to Pytorch.<br>
I have three datasets,each has different lengths. I have a model with three input pipelines (similar to Siamese, but three input pipelines) and therefore, I have to separately feed the data and I cannot concatenate them. The first one has around 600 ,second one has 3000 and last one has 40000 samples. In each training loop, I want to go through them and feed them into my model.</p>
<p>I defined three datasets and three dataloaders in pytorch; however, I am not sure how to iterate over them correctly? Since each has different length</p>
<p>In the case of two datasets, I have tried cycle over the smaller dataset and worked. But it is not working for three.</p>
<pre><code class="lang-auto">for step, ((images_Di, labels_Di),(images_Dj, labels_Dj)) in enumerate(zip(loader_D2, cycle(loader_D1))):
</code></pre>
<p>I appreciate if any one help me how is best way to handle them.</p>

MariosOreo: <p>I think you could zip the three dataloaders in the same way.</p>
<p>How does <code>cycle</code> implement the cycling operation?</p>
How to read the LibTorch API?

RuFAI: <p>Hi, does anyone have a tip how to read the LibTorch API? I am very confused with the structure. If there was no tutorial, I would not have found the basic types like e.g. torch::jit::IValue.Where is it? I can’t even find it, when I examine the namespace TORCH::JIT namespace. How is this possible?</p>
Time cost suddenly increases

malofleur: <p>When I train my model, there is always some time that each epoch will cost much time (longer than 5 mins), and another period only 4 seconds.<br>
I’m sure the running environment doesn’t change, is there any reason that will cause this phenomenon?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/d/d53047dab87a4b025403b570e53758b0e6b82585.png" data-download-href="https://discuss.pytorch.org/uploads/default/d53047dab87a4b025403b570e53758b0e6b82585" title="time%20cost"><img src="https://discuss.pytorch.org/uploads/default/original/2X/d/d53047dab87a4b025403b570e53758b0e6b82585.png" alt="time%20cost" data-base62-sha1="upX8ObxD9N0nUqZY1ZKUvoLQVvf" width="442" height="500" data-dominant-color="0C0C0C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">time%20cost</span><span class="informations">467×528 6.19 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <img src="https://discuss.pytorch.org/uploads/default/original/2X/2/2e6026d79ed822fcdebb1d9591ef9c16e65d7e0d.png" alt="time%20cost1" data-base62-sha1="6CfY05XZe6ECDE0978MjGcWYhmR" width="290" height="303"><br>
Any suggestion would be appreciated!</p>
Layer normalization outputs all zeros

mcstarioni: <p>I have a transformer-like model and it gave the same result for any inputs. I’ve figured out that it was a bias from linear layer and that there were zeroes as inputs. I’ve tracked this zeroing to Layer normalization within PositionWiseFF network:</p>
<pre><code class="lang-auto">class PositionWiseFeedForward(nn.Module):
    def __init__(self, spectral: SpectralWrapper, d_in, d_hid, d_out, dropout=0.1):
        super().__init__()
        self.layer1 = spectral.wrap(nn.Conv1d(d_in, d_hid, 1))
        self.layer2 = spectral.wrap(nn.Conv1d(d_hid, d_out, 1))
        self.res_layer = spectral.wrap(nn.Conv1d(d_in, d_out, 1))

        self.dropout = nn.Dropout(dropout)
        self.layer_norm = nn.LayerNorm(d_out)

    def forward(self, x):
        x = x.transpose(1, 2)

        output = self.layer2(F.relu(self.layer1(x)))
        output = output.transpose(1, 2)
        output = self.dropout(output)

        residual = self.res_layer(x)
        residual = residual.transpose(1, 2)

        output = output + residual
        output = self.layer_norm(output)
        return output
</code></pre>
<p>I’ve added some hooks:</p>
<pre><code class="lang-auto">activation = {}


def get_activation(name):
    def hook(model, input, output):
        activation[name] = (input[0].detach(), output.detach())

    return hook


from src.models import TransformerEncoder
model = TransformerEncoder((6, 3))
model.enc_block2.ff.layer_norm.register_forward_hook(get_activation('x'))
a = torch.rand(1, 6, 3)
b = model(a)
print("inputs:", activation["x"][0])
print("outputs:", activation["x"][1])
</code></pre>
<p>and this gave me this result:</p>
<pre><code class="lang-auto">inputs:  
tensor([[[ 0.1303883195],
         [ 1.3866363764],
         [ 0.5093938112],
         [-1.3120404482],
         [-0.4072887599],
         [ 1.0759189129]]])
outputs: 
 tensor([[[0.],
         [0.],
         [0.],
         [0.],
         [0.],
         [0.]]])
</code></pre>
<p>Also it worth mentioning that it happens only in second encoder block. The first one output real numbers.</p>

mcstarioni: <p>Layer Normalization i used was normalizing features along axis which had a size of 1 in the last layer, so all of the output was normalized to be 0.</p>
Building libtorch from source copies the headers from some libraries to the unexpected location

KohMat: <p>I build the libtorch from the source of 1.1.0 with turning off the USE_DISTRIBUTED=OFF and executing the following command.</p>
<pre><code>cmake .. -DCMAKE_INSTALL_PREFIX=/home/noname/test/
make -j 12
make install
</code></pre>
<p>it has been succeeded… However, I found the “torch” directory in the “home”(not home/test) after install.<br>
In the “torch” directory, there are the headers:</p>
<p>test<br>
│<br>
torch<br>
└── csrc<br>
└── jit<br>
├── script<br>
│   ├── error_report.h<br>
│   ├── function_schema_parser.h<br>
│   ├── lexer.h<br>
│   ├── parse_string_literal.h<br>
│   ├── schema_type_parser.h<br>
│   ├── strtod.h<br>
│   └── tree.h<br>
├── source_location.h<br>
└── source_range.h</p>
<p>This headers comes from the line 421 in the CMakeList.txt <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/CMakeLists.txt#L416" rel="nofollow noopener">here</a>.<br>
I know that the current design only assumes that the library is installed into the “pytorch/torch” directory.<br>
However,  is there any plans to install a particular directory where the user want without those files?</p>
The detail of shuffle in DataLoader

igreen: <p>In Doc of DataLoader, shuffle (bool, optional): set to <code>True</code> to have the data reshuffled at every epoch (default: <code>False</code>). So,  how to know the stop of one epoch, and then shuffle the training data. If I use this DataLoader with shuffle to load testing data, for example,</p>
<p>test_data = DataLoader(test_data_path)<br>
model.eval()<br>
test_model(test_data, model)<br>
test_model(test_data, model)</p>
<p>I run test twice, is the order of test data same? In other words, how shuffle work for this case. There is no “epoch” concept.</p>

InnovArul: <p>When you iterate on the <code>Dataloader</code> fully, it is considered as one epoch.</p>
<pre><code class="lang-auto">for data in test_data:
    do_test(data)

# one epoch is completed/all data points have been tested
</code></pre>
<p>In case of testing, <code>shuffle</code> is normally set to <code>False</code>. So, the order of <code>data</code> remains same.</p>

LW-Ricarido: <p>Now I only load part of my train set. I want to get the same part of my train set at every epoch. Is there any chance that I can set epoch concept.<br>
My code like this:<br>
I used this to get my dataloader</p>
<pre><code class="lang-python">return DataLoader(
        dataset,
        batch_size=args.batch_size,
        shuffle=args.shuffle,
        num_workers=args.workers,
        pin_memory=True
    )
</code></pre>
<p>use this to get part of train set:</p>
<pre><code class="lang-python"> for (input_tensor, target) in islice(train_loader,self.data_size):
    #my train code
</code></pre>

InnovArul: <p>You can use SubsetRandomSampler to do so.</p>
<p><a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.SubsetRandomSampler" class="onebox" target="_blank">https://pytorch.org/docs/stable/data.html#torch.utils.data.SubsetRandomSampler</a></p>

basingse: <p>I am facing some problem with shuffle = True in this thread : <a href="https://discuss.pytorch.org/t/dataloader-problem-problem-arises-when-shuffle-true/45631/13" class="inline-onebox">[DataLoader Problem] Problem arises when shuffle = True</a></p>
<p>Can I know where I am doing wrong ?</p>
Why does it no learn?

ollini42: <p>Hi,<br>
why is the following code not learning? The loss is always at 0.25 and the result looks random…<br>
Changing the network topology, optimizer, etc. is not changing a bit…<br>
Any help appreciated, Thanks, regards Oliver</p>
<p>import torch<br>
import torch.nn as nns<br>
import numpy as np<br>
from sklearn.datasets.samples_generator import make_blobs</p>
<p>x, y = make_blobs(n_samples=1000, centers=2, n_features=2)<br>
x_data = torch.Tensor(x)<br>
y_data = torch.Tensor(y)<br>
x_data.requires_grad_(True)<br>
y_data.requires_grad_(True)</p>
<p>model = torch.nn.Sequential(<br>
torch.nn.Linear(2, 10),<br>
torch.nn.Sigmoid(),<br>
torch.nn.Linear(10, 10),<br>
torch.nn.Sigmoid(),<br>
torch.nn.Linear(10, 1),<br>
)</p>
<p>optimizer = torch.optim.Adam(model.parameters())</p>
<p>loss_fn = torch.nn.MSELoss()</p>
<p>for t in range(1000):<br>
y_pred = model(x_data)</p>
<pre><code>  loss = loss_fn(y_data, y_pred)
  if t%100==0:
        print(t, loss.item())  

  model.zero_grad()
  loss.backward()

  optimizer.step()</code></pre>

ptrblck: <p>Could you print the shape of <code>y_pred</code> and <code>y_data</code>?<br>
Both should have the same shape, i.e. <code>[batch_size, 1]</code> in your case.<br>
In the past we’ve had similar issues where dim1 was missing in the target tensor, so that an unwanted broadcasting was performed. The latest PyTorch version will throw a warning, if this shape mismatch is detected.</p>

ollini42: <p>print(y_data.shape, y_pred.shape )<br>
gives<br>
torch.Size([10]) torch.Size([10, 1])</p>

ptrblck: <p>Add dim1 to your target and run your code again:</p>
<pre><code class="lang-python">y_data = y_data.unsqueeze(1)
loss = loss_fn(y_pred, y_data)
</code></pre>

ollini42: <p>Thanks! I really appreciate the help</p>
Question of Imagefolder using custom image dataset

Rayh: <p>It’s the simple Question.<br>
I got a 5 class folder and each have many images.<br>
So I use the Imagefolder, but list type output using the many CPU and RAM.<br>
I want to change this format to tensor like MNIST dset.<br>
How can I change this?<br>
Please help me T.T</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/f/f9b936be4651d92391c060fc8c5ebb1bbb09d0b9.png" data-download-href="https://discuss.pytorch.org/uploads/default/f9b936be4651d92391c060fc8c5ebb1bbb09d0b9" title="2"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/f/f9b936be4651d92391c060fc8c5ebb1bbb09d0b9_2_690x244.png" alt="2" data-base62-sha1="zD9HXgo90rmqkpkRKaJOJar7tQR" width="690" height="244" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/f/f9b936be4651d92391c060fc8c5ebb1bbb09d0b9_2_690x244.png, https://discuss.pytorch.org/uploads/default/original/2X/f/f9b936be4651d92391c060fc8c5ebb1bbb09d0b9.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/f/f9b936be4651d92391c060fc8c5ebb1bbb09d0b9.png 2x" data-dominant-color="73777C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2</span><span class="informations">837×296 85.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Up : My dataset<br>
Down : MNIST dataset</p>

ptrblck: <p>The advantage of using an <code>ImageFolder</code> is that you are loading <code>PIL.Images</code> and can apply all <code>torchvision</code> transformations on them directly (without transforming the binary data back to an image).</p>
<p>Although the MNIST dataset is saved as binary images, each image is converted back to a <code>PIL.Image</code> as seen in <a href="https://github.com/pytorch/vision/blob/cffa4e001a08054eb67c5550367fb03e771fed33/torchvision/datasets/mnist.py#L92" rel="nofollow noopener">this line of code</a> so that the output is consistent throughout all datasets.</p>
<p>Do you see any unusual CPU and RAM usage using <code>ImageFolder</code>?</p>
How can nn.Embedding output Tensor with dtype 'float64'?

w55100: <p>I found the output of nn.Embedding is default to be ‘float32’<br>
but I need it to be ‘float64’</p>
<pre><code class="lang-auto">CLASS torch.nn.Embedding(num_embeddings, embedding_dim, padding_idx=None, max_norm=None, norm_type=2.0, scale_grad_by_freq=False, sparse=False, _weight=None)
</code></pre>
<p>and torch documents don’t provide the way to set attr as ‘dtype=’</p>
<p>how can I change that?</p>

ptrblck: <p>Try calling <code>double()</code> on your layer:</p>
<pre><code class="lang-python">emb = nn.Embedding(...)
emb = emb.double()
</code></pre>
<p>This should change the type of internal parameters to <code>float64</code>.</p>

w55100: <p>thanks a lot!<br>
this really works!</p>
Training a model when dataset size is too big to fit the hard drive

torayeff: <p>I have faced this situation: I have a large dataset of size 1 TB, but my local drive has the capacity of 500GB. I can train a model only on a smaller subset of the original dataset, but I would like to make use of the whole dataset. So I wonder if there is a method or paper about the training of a model in these kinds of situations.</p>

ptrblck: <p>Where is the dataset stored currently?<br>
If it’s stored on a network drive, you could load the data lazily from there, which will most likely slow down your overall training due to the latency.<br>
Otherwise, you would somehow have to swap the old data for the new one (in each epoch), which might also be too slow.</p>
<p>Do you have an external SSD (with USB 3) which might be big enough?</p>

torayeff: <p>My data is exactly on a network drive. So it seems the only choice now is to purchase an external SSD <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>

ptrblck: <p>Internal drives would be better of course, if that’s possible. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
How can I recalculate gradient according to a manipulated gradient?

HelpMe: <p>This is the question about Autograd and Pytorch’s gradient management system.</p>
<ol>
<li>
<p>Basically, my neural network is like the encoder part of VAE - it gets images as its input, and returns a vector R.</p>
</li>
<li>
<p>With R, compute loss function L= SomeLossFunction( R ).</p>
</li>
<li>
<p>After performing L.backward(), R will have its own gradient. I mean, there must be something like R.grad.</p>
</li>
<li>
<p><strong>What I want to do : adjust the R.grad manually, and re-compute all the gradient of all the weight parameters in NN according to this adjusted gradient.</strong></p>
</li>
</ol>
<p>Is this computation possible without fixing autograd system too much?</p>

sonsus: <p>Not really sure what you want to do, but you could circumvent modifying autograd itself since you can directly access to gradient informations by <code>sometensor.grad</code> (or maybe there would be some proper way to do this) for Tensors of <strong>leaf nodes</strong> with <strong>requires_grad==True</strong>. In my case, when I faced a wall with autograd problem, this youtube <a href="https://youtu.be/MswxJw-8PvE" rel="nofollow noopener">vid</a> really helped me a lot (13 min)</p>
<p>btw is it proper to link a youtube here? it is educational one about pytorch. <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>

HelpMe: <p>I’m really sorry. I noticed what I really wanted, thanks to your vid and tutorial - just modifying backward function like Relu solves my problem. Thanks for your help.</p>

sonsus: <p>Glad to hear that you found your own solution!</p>
Problem about nn.BatchNorm2d's output

kanbaiou: <p>Hi , I have just met a paroblem with nn.BatchNorm2d layer</p>
<p>due some reasons, I have to copy one BN-layer’s weight to an other . when I do this I just do like</p>
<pre><code class="lang-auto">layer1.weight.data = layer2.weight.data
layer1.bais.data = layer2.bias.data

</code></pre>
<p>and after doing this , I check these two layer’s weight and bias , they are same.</p>
<p>and also, when print this two layers , they are both shown as</p>
<blockquote>
<p>BatchNorm2d(16,eps=1e-05,momentum =0.1,affine = True,track_running_state = True)</p>
</blockquote>
<p>However, when I put the same input to these two layers, the outputs of them are different!</p>
<p>I  am not clear about  what’s wrong, what other parts should I check about these two layers?</p>
<p>thank you very much.</p>

kanbaiou: <p>ok ,</p>
<p>I think I should consider about running_mean and running_var<br>
when I copy these two as well , the problem has been solved</p>
Can I mark the gradient?

HelpMe: <p>Hello, Pytorch forum.<br>
My neural network(with weight w) takes some input, and its output is two scalar variables, namely x and y.<br>
The loss function is simply L(x,y). Now, what I want to do is calculating two vectors dx/dw, dy/dw(it is sad that I cannot use LateX here - I mean \nabla x, \nabla y).<br>
Doing L.backward() will calculate dL/dw, which is dL/dx* dx/dw + dL/dy*dy/dw by chain rule. For my research, I just need dx/dw, dy/dw. Is there any convenient method or function that I can use?</p>
<p>Plus, if you know any papers which deal with this topic, please let me know… it would be a great help for me… I am now looking for the natural gradient descent things on VAE.</p>
The most common question on `optimizer.zero_grad()`. Just re-confirming my understanding

iArunava: <p>Is it</p>
<pre><code class="lang-auto">optimizer.zero_grad()
out = model(batch_X)
loss = loss_fn(out, batch_y)
loss.backward()
optimizer.step()
</code></pre>
<p>or</p>
<pre><code class="lang-auto">out = model(batch_X)
loss = loss_fn(out, batch_y)
optimizer.zero_grad()
loss.backward()
optimizer.step()
</code></pre>
<p>I use the second one, as I belive just before the gradients are calculated, I need to zero out the existing ones.</p>
<p>But, my project reviewer on Udacity says to use the first one.<br>
The model is LSTM.</p>
<p>Also, I think both are the same, isn’t it?</p>

MariosOreo: <p>Hi bro,</p>
<p>I also think they are the same.</p>
<p>I have read a thread about correct order in this forums (I cannot find it now). From developers’ comment, they recommeded to use the first way you mentioned above.</p>

iArunava: <p>Thanks a ton man. Also, in case you stumble upon that thread. Just let me know here <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

MariosOreo: <p>Sorry for misremembering it.</p>
<p>In this <a href="https://discuss.pytorch.org/t/when-is-the-better-time-to-zero-the-gradients/39930/2">comment</a>, he just recommended calling <code>optimizer.zero_grad()</code> before <code>.backward()</code>.<img src="https://discuss.pytorch.org/images/emoji/apple/disappointed_relieved.png?v=9" title=":disappointed_relieved:" class="emoji" alt=":disappointed_relieved:"></p>

alex.veuthey: <p>I also agree that both should be the same, and this makes sense, as the gradients are only computed when <code>backward()</code> is called…</p>

ptrblck: <p>I would argue it depends on your “workflow” as both approaches yield the same result as others already said.</p>
<p>I personally prefer the first approach due to my mindset of<br>
“new iteration -&gt; new gradients -&gt; get rid of the old ones”.<br>
Otherwise I’ve sometimes forgotten to zero out the gradients. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

iArunava: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> for your answer <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
On using 'with torch.no_grad' and 'detach' during testing

Deeply: <p>I noticed that during testing the <code>grad</code> would still be true for the tensors we got from the models as outputs. Would using <code>detach()</code> reduce the memory usage, or give any benefit?<br>
The code during testing can be like:</p>
<pre><code class="lang-auto">my_model.eval()
with torch.no_grad():
       for batch_idx, batch in enumerate(test_dataloader):            
           imgs =get_imgs(batch)                      
           output  =  my_model(imgs).detach()
</code></pre>
<p>Any ideas?</p>

ptrblck: <p>The usage of <code>detach()</code> won’t save additional memory, since <code>output</code> is not attached to anything.<br>
If you try to print the output, you’ll see that the <code>grad_fn</code> is missing, so that <code>detach()</code> won’t change anything.<br>
The gradients (from the previous run) might still be allocated as <code>torch.no_grad()</code> avoids storing intermediate activations, which would be needed for the backward pass.</p>
Save optimizer states on multi-gpu?

fermat97: <p>Hi, I have a question about saving the optimizer states on multi-gpu training scenario:</p>
<pre><code class="lang-auto">checkpoint = {
         'model': model.module.state_dict(),
         'optimizer': optimizer.module.state_dict()}
</code></pre>
<p>The code above does not work. If I just save like single gpu:</p>
<pre><code class="lang-auto">checkpoint = {
         'model': model.module.state_dict(),
         'optimizer': optimizer.state_dict()}
</code></pre>
<p>Is that accurate?</p>

ptrblck: <p>The second method should work fine, as the optimizer should get the parameters of the non-parallelized model.</p>
Can pytorch track gradient like this?

Cho: <p>In my forward function, I resize and concatenate 2 output from 2 LSTM, then I put this concatenated tensor into a  connected network with Relu and Sigmoid. I am wondering whether Pytotch is able to track gradient after I resize and concatenate the output from 2 LSTM. Since even the loss is decreasing after back propagation, but the accuracy on the test set is not improving.</p>
<pre><code class="lang-auto">        output_allhiden_1,(final_hidden_state_1,final_cell_state_1)=self.LSTM_1(input_1)
        output_allhiden_2,(final_hidden_state_2,final_cell_state_2)=self.LSTM_2(input_2)
        final_hidden_state_1=final_hidden_state_1.view(num_layers,batchsize,-1)
        final_hidden_state_2=final_hidden_state_2.view(num_layers,batchsize,-1)
        concat=torch.cat((final_hidden_state_1[num_layers-1],final_hidden_state_2[num_layers-1]),dim=1)
        output_p=self.output1(concat)
        tanhoutput=self.relu(output_p)
        output_p2=self.output2(tanhoutput)
        sigput_p2=self.sig(output_p2)
</code></pre>

leowalkling: <p>torch.cat and index_select are definitely not blocking your gradients, what you’re doing in the methods not shown could, however.<br>
Btw, output_allhiden_1 will be identical to the hidden state of the last layer, and the layout of the hidden state should rather be treated as implementation detail of LSTM).</p>
<p>You can easily test this yourself:</p>
<pre><code class="lang-auto">import torch
from torch import nn
from torch import autograd as ta

self = type("Net", (torch.nn.Module,), {})()
self.LSTM_1 = nn.LSTM(3, 40, 2)
self.LSTM_2 = nn.LSTM(3, 40, 2)
input_1 = torch.randn(5, 1, 3)
input_2 = torch.randn(5, 1, 3)
output_allhiden_1, (final_hidden_state_1, final_cell_state_1) = self.LSTM_1(input_1)
output_allhiden_2, (final_hidden_state_2, final_cell_state_2) = self.LSTM_2(input_2)
# No need to reshape, shape should already be (num_layers, 1, hidden_size)
# final_hidden_state_1 = final_hidden_state_1.view(num_layers, 1, -1)
# final_hidden_state_2 = final_hidden_state_2.view(num_layers, 1, -1)
assert (output_allhiden_1[-1] == final_hidden_state_1[-1]).all().item()
concat = torch.cat((output_allhiden_1[-1], output_allhiden_2[-1]), dim=1)
concat.sum().backward()
for n, t in self.named_parameters():
    if t.grad is not None:
        print(n, ": ", t.grad.var().item())
</code></pre>
Delete references between variable and .backward

Mdhvince: <p>Hi,</p>
<p>I’m trying to reproduce a code from chainer to PyTorch. I would like to know if there is a way to perform the chainer method <code>unchain_backward()</code> in PyTorch. This method deletes references between variable nodes and functions backward.</p>
<p>Thanks,</p>
<p>Medhy</p>

ptrblck: <p>I’m not completely sure, but I think <code>.detach()</code> should be the equivalent method.</p>

Mdhvince: <p>I was thinking to use <code>.detach</code> too, but not sure. I’ll try. Thank you for your reply <a class="mention" href="/u/ptrblck">@ptrblck</a>.</p>

leowalkling: <p>I’m not sure which references you want deleted, but if you want to make sure you’re not keeping from deallocation anything but the result you’re returning, input.detach() is right.</p>

Mdhvince: <p>Yes it is the idea. Thanks</p>
Unclear about relation between TorchScript and backprop

leowalkling: <p>Hi,<br>
I am trying to write a TorchScript operator to replace a custom torch.autograd.Function to influence the gradient computation, since TorchScript cannot serialize calls to custom Functions.<br>
Is there an equivalent of torch.autograd.Function in ATen or Torch?</p>
<p>Best regards<br>
Leopold</p>

ailzhang: <p><a class="mention" href="/u/leowalkling">@leowalkling</a>,  could you elaborate more on what you want to replace here with a example? In case it’s helpful, in JIT we currently put ops to 2 categories:</p>
<ol>
<li>Ops have autodiff formulas in autodiff.cpp: their backward graph is replaced by <a href="https://github.com/pytorch/pytorch/blob/0cb24098c74f8ebed81ec08b83bf6cb5ab3903f5/torch/csrc/jit/graph_executor.cpp#L76" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/0cb24098c74f8ebed81ec08b83bf6cb5ab3903f5/torch/csrc/jit/graph_executor.cpp#L76</a>
</li>
<li>Ops don’t have autodiff formulas: their backwards are handled by eager mode autograd directly.<br>
Please let us know the context of problem so that we can help more on this.</li>
</ol>
<p>Thanks,<br>
Ailing</p>

leowalkling: <p>For reference, I’ve come up with a solution by imitating the generated kernels and the code generation<br>
from /tools/autograd.<br>
But it seems to me that it is a very intricate task, given that there is little documentation, so I made sure to follow the original code of builtin operations closely.</p>
<p>The purpose I had in mind was to perform in-place modification of a variable without incrementing invalidating its former gradient, but instead creating a new variable depending on the old one.<br>
I’m using this in my implementations of IAF, MADE, etc. to save RAM (by using fewer clones of the same data).</p>
<p>The following is what wasn’t obvious from the docs (to me):</p>
<ol>
<li>Use ta::make_variable or ta::make_variable_view to create the output Variable_s</li>
<li>Implement a new subclass of ta::TraceableFunction in your module.<br>
With Pytorch 1.0 from Conda, the headers lack generated code, such as the xxxBackward classes, so there was no good way to use even the existing subclasses of ta::TraceableFunction.</li>
<li>Instantiate your custom xxxBackward and register it in the graph using the methods set_next_edges and add_input_metadata of TraceableFunction and set_gradient_edge of Variable. Helper functions available in “torch/csrc/autograd/functions/utils.h”</li>
<li>TorchScript-compatibility requires registering the op as described in the tutorials on extending TorchScript.</li>
</ol>
Torch.cuba.is_available() returns false in jupyter notebook

AvrilGZC: <p>I used Jupiter notebook (azure notebook&amp;ibm Jupiter lab),it both returned FALSE when I tried <code>torch.cuba.is_available()</code>.<br>
My torch version is 1.0.1, python version is 3.6.<br>
And I both tried pip and conda to install torch.<br>
Is that due to Jupiter notebook cannot dispatch GPU?<br>
Thanks</p>

Nikronic: <p>Are you using free version of these clouds?<br>
The free version of jupyter on those clouds are without GPUs so <code>torch.cuda.is_available()</code> is returning <code>False</code>.</p>
<p>It is not true which jupyter cannot dispatch GPU, because it is just a interface on Ipython which is python itself.</p>
Problem loading C++ torch script oeprator from python

jviereck: <p>Hi folks,</p>
<p>following the tutorial at <a href="https://pytorch.org/tutorials/advanced/torch_script_custom_ops.html" rel="nofollow noopener">1</a>, I am trying to implement a simple C++ torch script operator. I’ve installed libtorch <a href="https://pytorch.org/cppdocs/installing.html" rel="nofollow noopener">2</a> and managed to compile my module. However, when I am trying to load the operator via</p>
<pre><code class="lang-python">import torch
torch.ops.load_library("/home/jviereck/dev/tools/torch_pinocchio/build/libtorch_pinocchio.so")
</code></pre>
<p>this is the error I am getting:</p>
<pre><code class="lang-auto">$ python2 ../test/basic.py 
Traceback (most recent call last):
  File "../test/basic.py", line 2, in &lt;module&gt;
    torch.ops.load_library("/home/jviereck/dev/tools/torch_pinocchio/build/libtorch_pinocchio.so")
  File "/home/jviereck/.local/lib/python2.7/site-packages/torch/_ops.py", line 102, in load_library
    ctypes.CDLL(path)
  File "/usr/lib/python2.7/ctypes/__init__.py", line 362, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /home/jviereck/dev/tools/torch_pinocchio/build/libtorch_pinocchio.so: undefined symbol: _ZN6caffe26detail37_typeMetaDataInstance_preallocated_29E
</code></pre>
<p>Note that I’ve installed the torch via <code>pip install torch</code>.</p>
<p>My best guess is that the python torch is not aware about the libraries in the libtorch folder. Does someone has an idea how to resolve this?</p>
<p>Best,<br>
Julian</p>

ptrblck: <p>Not sure if that’s related to your problem, but similar issues originated in multiple PyTorch installatons (or some old, unused installs).<br>
Could you check, if your current environment has some old PyTorch installs (in conda, pip, and build from source)?</p>

leowalkling: <p>My guess is that either</p>
<ul>
<li>Your compiler version is not the same as the one that compiled the caffe2 library, in particular differing in its name mangling (mapping C++ namespace+symbol+signature to a unique name in an address table)</li>
<li>The function you were using somewhere in your extension (caffe2::typeMetaDataInstance_preallocated) is not exported by the libraries you linked against.</li>
</ul>
<p>What you can do is check whether you are getting the same error when compiling with the same compiler as was used to compile Caffe2.</p>