akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Seq2seq raining- why is (len) timesteps used with nn.GRU?

sid-ls: <p>I’m going through the very useful tutorial by <a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html" rel="nofollow noopener">PyTorch</a> on Seq2Seq training.</p>
<p>Given that <code>nn.GRU</code> takes in input of shape <em>(seq_len, batch, input_size)</em>, why does the encoder have N (seq_len) number of timesteps during training?</p>
<pre><code class="lang-auto">    for ei in range(input_length):
        encoder_output, encoder_hidden = encoder(
            input_tensor[ei], encoder_hidden)
        encoder_outputs[ei] = encoder_output[0, 0]
</code></pre>
<p>From the docs GRU can take in a batch input of the above shape, shouldn’t it work without a N timesteps?</p>
<pre><code class="lang-auto">encoder_output, encoder_hidden = encoder(
            input_tensor)
</code></pre>
<p>Doesn’t calling GRU(input) take care of all the timesteps? If it was one just GRU cell (<code>nn.GRUCell</code>) then I can sort of undertsand why the above method is used, but it’s using a GRU layer instead</p>

vdw: <p>Hm, I also do not think that the loop for the encoder here is necessary (and probably would wouldn’t work correctly in case the GRU layer would be bidirectional). I would almost argue that this loop wasn’t in the first version of the tutorial since I used it as baseline for my own code, and I give the whole sequence at once to the GRU layer.</p>
<p>Doing it using a loop obviously allows you to modify the hidden state after each time step (if there are meaningful ways to do it), but here it’s not needed.</p>

asdf11: <p>A question regarding the input length to the used encoder:</p>
<p>The encoder consists of an amount (hidden_dim) of GRU cells, which means that the maximum length of the source sentence cant be higher than the hidden_dim, right? What happens if the source sentence length is longer than that?</p>

sid-ls: <p>The encoder doesn’t consist of a <code>hidden_dim</code> number of GRU cells. The input length is not related to <code>hidden_dim</code></p>
<p><code>hidden_dim</code> is the size of the hidden state you desire, and is a result of matrix multiplications on your input. Your input of shape <code>(batch_size, seq_len, num_features</code>) will result in GRU output shape of <code>(batch_size, seq_len, hidden_dim)</code></p>

asdf11: <p>Thanks for reply. Im not really sure if I got the idea right then:</p>
<p>I thought each (embedded) word is read in bei a single cell of an encoder? E.g. There is a sentence of 10 words, then Id need an Encoder of at least 10 GRU cells? So every GRU cell holds the information for one word?</p>
<p>Or is the input of an single GRU cell (which I thought is the hidden_dim) all 10 words?</p>
<p>Same for the decoder: I thought the max length of a predicted sentence is the length of a Decoder?</p>

sid-ls: <p>Yes, if your encoder input sentence is of 10 words then you must have at least 10 GRU cells. 1 GRU cell there will take 2 inputs- the current word and a hidden state. This hidden state is either got from the encoder of the previous timestep or can be initialized</p>
ResNet unexpected output

Bruno_Oliveira: <p>I’m working on a project and I need the output of a truncated ResNet. Taking ResNet18 as an example, if I do the following changes</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torchvision.models as models

resnet = models.resnet18(pretrained=False)
resnet.avgpool = nn.Identity()
resnet.fc = nn.Identity()
</code></pre>
<p>The output gets flattened and I don’t know why. The curious thing is that if I forward a (16, 3, 256, 256) tensor through it, I get an output of size (16, 32768), while if I manually forward the input layer by layer, I get the expected output (16, 512, 8, 8). It seems like a bug, but I’m not sure. Can anyone take a look into it?</p>

ptrblck: <p>It’s expected, as ResNet uses a functional call to <code>torch.flatten</code> in the forward pass in <a href="https://github.com/pytorch/vision/blob/bd27e942683099dc9aa3d13600c2a9272108eed9/torchvision/models/resnet.py#L214">this line of code</a>.<br>
You could either derive a custom ResNet class and reimplement the <code>forward</code> method or use forward hooks to get the desired activation.</p>
<p>Alternatively, you could also create an <code>nn.Sequential</code> container and add the layers in the same order as they are called in the original <code>forward</code> method.</p>

Bruno_Oliveira: <p>That’s true, I should have checked the ResNet code beforehand.<br>
And yes, I did work it around by calling the layers in a nn.Sequential container to avoid the flattened tensor.<br>
Thanks ptrblck.</p>
GPU computation of large array

Yongqiang_Chen: <p>Hi Everyone,</p>
<p>I converting a 10GB CT scanning images to an array and then transformed to torch tensor. I intend to do some mathematical operation with GPU parallelization on this big tensor. It is too large to move to the GPU memory. Is there any method to do this operation in Pytorch?</p>
<p>Thanks, everyone.</p>

ptrblck: <p>It depends on the available memory of your GPU.<br>
If you have a newer GPU with 16 or 32GB, some operations might be possible on this tensor.<br>
Depending on the operation, you might need to create an output tensor, which could take the same amount of memory.</p>
<p>Which operations are you interested in?</p>

Yongqiang_Chen: <p>Thanks for the kind reply. My workstation only has 4GB GPU memory. That is the problem.<br>
I wonder if the data can be read and processed in GPU piece by piece. So not all the data is needed to be imported into the GPU memory. Is this function available in pytorch?</p>
<p>Thanks a lot.</p>

ptrblck: <p>If you could split the operation into chunks of data, it’s possible.<br>
E.g. if you would like to sum a tensor, you could slice the original tensor, push this slice to the device, sum the values, and transfer the result back to the host.<br>
Once all slices are done, you could sum all temporary results.</p>
<p>Note that you add overhead due to the host2device and device2host copies and depending on your operation and workload, it might be faster on the CPU, so you should profile both approaches.</p>

Yongqiang_Chen: <p>Thanks for the advice. I’ll give it a try.</p>
When is DispatchStub called

almeetb: <p>I am trying to create a new backend and on tensor creation with a new device type I am getting error: <code>DispatchStub: unsupported device</code></p>
<p>How is DispatchStub called?</p>

ezyang: <p>It’s going to be hard to answer this question without some code. Could you post it somewhere?</p>

ezyang: <p>Using my psychic debugging skills, I am guessing that what is happening is that you’re hitting some preexisting code that is registered to work for all backends, which is attempting to a redispatch in DispatchStub. Since you probably have not registered a real implementation to DispatchStub, it is saying the device is unsupported.</p>
<p>However, what the right solution for this is, depends a lot on what you are actually trying to do.</p>

almeetb: <p>I figured what I was missing, thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
What were the problems with GAN's , How did they solved?

mathematics: <p>As this is deep learning forum. I thought to ask this question anyway.<br>
As Ian talked about GAN’s problems. About that weird structure GAN were making given in the Nips lecture.<br>
As GAN today progress much much better.</p>
<p>Problems he mentioned was</p>
<ul>
<li>Problems with counting</li>
<li>cherry picked</li>
<li>global_structure</li>
<li>perspective</li>
</ul>
<p>as these problems gave some images like<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/9/890ea663ecdfb11804c6c88d6d453a14b3cc3b95.jpeg" alt="counting" data-base62-sha1="jysNEtbokrjIpS6Y0Di1zO9RguF" width="535" height="395"> <img src="https://discuss.pytorch.org/uploads/default/original/3X/0/c/0c43ef2bad9d70837de84b3bb40f22680003b159.jpeg" alt="global structure" data-base62-sha1="1Kvhf7mZsJftYYt9ufNlV5MJdLb" width="552" height="423"> <img src="https://discuss.pytorch.org/uploads/default/original/3X/2/b/2b55179e2ff5c2ffd4da16efa0b9d0d9f0bda302.jpeg" alt="Perspective" data-base62-sha1="6bkQ74E7dMpRXXhsuJkk0qmSK7E" width="539" height="381"> <img src="https://discuss.pytorch.org/uploads/default/original/3X/6/2/6290bac05cf345be76556f58c95cfdec27fd456d.jpeg" alt="cherry_picked" data-base62-sha1="e3WSacM9toJt13O02ku3IKMxZbn" width="560" height="388"></p>
<p>What conditions because of that they got this problem, how was it solved ?<br>
If you know , please tell each solution of problem for mentioned problem.<br>
Thank you for help.</p>
How to compute RNN Hidden-to-Hidden Jacobian in Pytorch?

RylanSchaeffer: <p>I want to compute the hidden state to hidden state Jacobian of a RNN with a single layer, evaluated at a number of points (i.e. over a batch), and I want to confirm that I’m doing this correctly.</p>
<p>The hidden state has shape <code>(batch size, num layers=1, hidden state size)</code>. I first compute the unit basis vectors as</p>
<pre><code class="lang-auto">    num_basis_vectors = num_layers * hidden_size
    unit_basis_vectors = torch.eye(n=num_basis_vectors).reshape(
        num_basis_vectors, num_layers, hidden_size)
</code></pre>
<p>I then loop over the unit basis vectors (as above):</p>
<pre><code class="lang-auto">        jacobian_components = []
        for unit_vector in unit_basis_vectors:
            jacobian_component = torch.autograd.grad(
                outputs=rnn_output_hidden_states,  # shape (batch size, num layers=1, rnn hidden state size) 
                inputs=rnn_input_hidden_states,  # shape (batch size, num layers=1, rnn hidden state size)
                grad_outputs=torch.stack([unit_vector] * batch_size),  # repeat batch_size times
                retain_graph=True)[0]
            jacobian_component = torch.mean(jacobian_component, dim=(1,))
            jacobian_components.append(jacobian_component)
        jacobian = torch.stack(jacobian_components, dim=1)
</code></pre>
<p>Is this correct? How would I go about testing the correctness?</p>
Generating images of different sizes

mathematics: <p>I have a problem while generating images of different sizes from GANs.<br>
I have  input images of different varying shapes extremely like (220, 3500) or (2100,230) or different. If i reshape or modified it will be waste.</p>
<p>I want to get images given latent random distribution which generates a varying sizes of different images randomly.</p>
<p>How can i do that ?<br>
Please post your ideas.</p>

ptrblck: <p>You could try to use <a href="https://arxiv.org/abs/1411.1784">Conditional GANs</a> and treat the “shape” as a class information. You would have to define certain shape classes, but this might be OK instead of reshaping all input images to the same resulution.<br>
Also, the class label could be used to flip the transposed convolutions (swap height and width of the kernels), so that you could use the same model to create the mentioned outputs.<br>
I think using the functional API might be easier in that case than modules.</p>

mathematics: <p>Thank you. I will work on this project, if problem occur i will ask . <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Run pytorch on jupyter notebook

ph0123: <p>Hi,</p>
<p>I try to run example from tutorial with “GLoo” backend and Point to Point communication.</p>
<pre><code class="lang-auto">"""run.py:"""
#!/usr/bin/env python
import os
import torch
import torch.distributed as dist
from torch.multiprocessing import Process

def run(rank, size):
    tensor = torch.zeros(1)
    if rank == 0:
        tensor += 1
        # Send the tensor to process 1
        dist.send(tensor=tensor, dst=1)
    else:
        # Receive tensor from process 0
        dist.recv(tensor=tensor, src=0)
    print('Rank ', rank, ' has data ', tensor[0])

def init_process(rank, size, fn, backend='gloo'):
    """ Initialize the distributed environment. """
    os.environ['MASTER_ADDR'] = '127.0.0.1'
    os.environ['MASTER_PORT'] = '29500'
    dist.init_process_group(backend, rank=rank, world_size=size)
    fn(rank, size)


if __name__ == "__main__":
    size = 2
    processes = []
    for rank in range(size):
        p = Process(target=init_process, args=(rank, size, run))
        p.start()
        processes.append(p)

    for p in processes:
        p.join()
    print("done")
</code></pre>
<p>When I run it, only “done” is printed on jupyter notebook.<br>
How to run it with python?<br>
Thanks,</p>

mrshenli: <p>I tried this with colab, but cannot reproduce this problem. Sometimes there are weird behavior when using multiprocessing in notebook. If you directly launch this program using command line, are the outputs as expected?</p>

ph0123: <p>Yes. It work with python.<br>
But I wanna ask about run it on jupyter.<br>
How do it work on Jupyter? This is my question.<br>
Thanks,</p>

mrshenli: <p>As I cannot reproduce the error on my Jupyter notebook, I can only guess why the message from subprocess is not shown. Given that the main process prints “done”, I would assume the sub-processes are launched correctly. But since the subprocess didn’t print the message, it could be either 1) sub-process crashed 2) sub-process is not printing to stdout. For 1), you can check the <code>exitcode</code> of the subprocess, adding more logs will also help. For 2) you will need check local configures to see if it is redirected, or you explicitly redirect that print to file.</p>

ph0123: <aside class="quote no-group" data-username="mrshenli" data-post="4" data-topic="78951">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrshenli/48/12220_2.png" class="avatar"> mrshenli:</div>
<blockquote>
<p>o file.</p>
</blockquote>
</aside>
<p>Hi,<br>
Thank you so much!</p>

ph0123: <p>Hi all,</p>
<p>I find the solution for that.<br>
I run jupyter on macbook, and It worked.<br>
On Window, the program only printed “done”.</p>
<p>Thanks,</p>

mrshenli: <p>PyTorch distributed package does not support Windows yet. So most likely the subprocess crashed as <code>init_process_group</code>  is not available on Windows.</p>
2: RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

drezap: <p>Hi -</p>
<p>I’m trying to implement a custom loss function. Not differentiable, requires I implement gradients. The interpreter doesn’t like the current foo model:</p>
<pre><code class="lang-auto">class my_loss_fn(Function):

    def __init__(self, scalar=None):
        super(my_loss_fn, self).__init__()
        self.scalar = scalar
    
    @staticmethod
    def forward(self, i):
        out = (i - i).pow(2).sum()
        self.mark_non_differentiable(out)
        return out
    
    @staticmethod
    def backward(self, grad_output):
        grad_input = grad_output.clone()
        grad_input = 2.0*(grad_output)
        return grad_input, None

model = nn.Linear(2, 2)
x = torch.randn(1, 2)
target = torch.randn(1, 2)
output = model(x)
loss = my_loss_fn.apply(output)
loss.backward()
print(model.weight.grad)
</code></pre>
<p>I get error: <code>RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</code></p>
<p>I’ve tried throwing <code>requires_grad = True</code> in some places, per some suggestions in other posts, with no luck.</p>
<p>What am I missing?</p>
<p>I don’t care if the model does anything meaningful, for now.</p>

albanD: <p>Hi,</p>
<p>You can find more details on how to implement custom functions <a href="https://pytorch.org/docs/stable/notes/extending.html">here</a>.<br>
In particular, you don’t need the <code>__init__</code> method.</p>
<p>And also, because you’re writing the custom Function, you only need to mark an output as non-differentiable if you don’t need gradient for it. Here you do, so you should remove the <code>self.mark_non_differentiable(out)</code> line.</p>
Loading FastText embeddings results to HTTPError

Talita: <p>I am doing a text classification task in PyTorch (1.5), but</p>
<p><code>TEXT.build_vocab(train_data,                   max_size = MAX_VOCAB_SIZE,                   vectors = "fasttext.en.300d",                   unk_init = torch.Tensor.normal_)</code></p>
<p>Results to the following error:</p>
<p>HTTPError: HTTP Error 301: Moved Permanently</p>
<p>Any idea how to solve this problem?</p>

ptrblck: <p>Hi Talita,</p>
<p>I assume you are using <code>torchtext</code>?<br>
If so, could you please create an issue on <a href="https://github.com/pytorch/text/issues">GitHub</a> with this error message?</p>

Talita: <p>Yes!!</p>
<p>Thanks for replying :).</p>
Tabular dataset for multiple files

abhigenie92: <p>Is there a way to use <a href="https://torchtext.readthedocs.io/en/latest/data.html#tabulardataset" rel="nofollow noopener">https://torchtext.readthedocs.io/en/latest/data.html#tabulardataset</a> for loading multiple csv files? Can use a wildcard pattern in path?</p>
<p>Say the train data is located in multiple files.</p>
<pre><code class="lang-auto">
dataset = TabularDataset(path = "..", 
    format = "csv", skip_header = True, fields = csv_datafields)

</code></pre>

tom: <p>There isn’t, but you can grab the <a href="https://github.com/pytorch/text/blob/fd764ad02968e6310d28ee3c496e1d1d1157aa3e/torchtext/data/dataset.py#L251" rel="nofollow noopener">code</a> (which isn’t long and almost exclusively about loading the file) and wrap the <code>with</code> block linked in a for loop. If you then start <code>examples</code> with an empty list and use <code>examples += ...</code> after reading the file, you’re good to go.</p>
<p>Best regards</p>
<p>Thomas</p>

abhigenie92: <p><a class="mention" href="/u/tom">@tom</a> thanks for the reply! I see, I wasn’t looking to modify the package. I was thinking of concating Tabular datasets instead somehow.</p>

tom: <p>Not that there is anything wrong with modelling your own dataset after torchtets, but perhaps the following looks more straightforward:<br>
You cannot take the PyTorch dataset concatenation if you want to stick with the torchtext.data.Dataset. What you could do is take two (or more) tabular datasets, say <code>ds1</code> and <code>ds2</code>, and create a new <code>torchtext.data.Dataset</code> passing <code>ds1.examples + ds2.examples</code> and <code>ds1.fields</code> to the constructor (provided fields are compatible). That should give you a combined dataset supporting the torchtext extras like splits.</p>
<p>Best regards</p>
<p>Thomas</p>
Stack is leading to segmentation fault

jacobhm98: <p>Hello! I am trying to create a 4d tensor out of a  vector of 3d tensors, and am calling stack repeatedly. However, this is causing a segmentation fault, and Im not sure why. The size of the tensors shouldnt be the issue as stack works with torch::randn() of same dimensions.</p>
<pre><code class="lang-auto">         imageAndClass firstElem = images[0]; 
         imageAndClass secondElem = images[1]; 
         torch::Tensor training_data = firstElem.image; 
         cout &lt;&lt; "test" &lt;&lt; endl; 
         torch::Tensor test = torch::stack({training_data, secondElem.image}, 0); 
         cout &lt;&lt; "test" &lt;&lt; endl;

</code></pre>
<p>Where images is a std::vector of structs containing a torch::Tensor, and an integer classification.</p>

ptrblck: <p>Could you explain a bit, how you are calling <code>torch::stack</code> repeatedly? Are you calling it on the same tensor (<code>training_data</code>) or on different tensors?<br>
Is the segfault raised in the first call or later?</p>

jacobhm98: <p>After a little more debugging, I’ve realized the error is occurring earlier, when I am creating my tensor.</p>
<p>` torch::Tensor imgToTensor(std::string img_path) {</p>
<pre><code>      cv::Mat origImage;
      cv::Mat sizedImage(200, 500, CV_32FC3);
      origImage = cv::imread(img_path, 1);
      cv::resize(origImage, sizedImage, sizedImage.size(), 0, 0, cv::INTER_LINEAR);
      cout &lt;&lt; "test" &lt;&lt; endl;
      torch::Tensor input = torch::from_blob(sizedImage.data, 
      {1, sizedImage.rows, sizedImage.cols, 3});
      cout &lt;&lt; input &lt;&lt; endl;
      input = torch::transpose(input, 1, 3);
      input = torch::transpose(input, 2, 3);
      return input;
</code></pre>
<p>}`</p>
<p>A segmentation fault is thrown after test, when i try to print/access input in any type of way.</p>

LeviViana: <aside class="quote no-group" data-username="jacobhm98" data-post="3" data-topic="79682">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/9de0a6/48.png" class="avatar"> jacobhm98:</div>
<blockquote>
<p>torch::Tensor input = torch::from_blob(sizedImage.data, {1, sizedImage.rows, sizedImage.cols, 3});</p>
</blockquote>
</aside>
<p>Try this:</p>
<pre data-code-wrap="cpp"><code class="lang-cpp">/*cv::CV_8UC3*/
assert(
    sizedImage.type() == 16
);

torch::Tensor input = torch::from_blob(
    sizedImage.data,
    {1, sizedImage.rows, sizedImage.cols, 3},
    torch::kByte
);
</code></pre>

jacobhm98: <p>Thank you for the suggestion. I actually resolved it, and the bug is as you expected due to the type of the image not being correct. I needed to add this statement before I did the resize operation:</p>
<pre><code class="lang-auto">origImage.convertTo(origImage, CV_32FC3, 1/255.0);
</code></pre>
AttributeError: 'Tensor' object has no attribute 'items'

Hika-Kondo: <p>I want to log the loss of the train using the tensorboard in pytorch. I got an error there.</p>
<pre><code class="lang-auto">AttributeError: 'Tensor' object has no attribute 'items'
</code></pre>
<p>I want to solve this error and check the log using tensorboard. Here I show my code.</p>
<pre><code class="lang-auto">l_mse = mseloss(img,decoder_out)
writer.add_scalars("MSE",l_mse,n_iter)
</code></pre>
<p>then I have error blow.</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "main.py", line 39, in &lt;module&gt;
    main()
  File "main.py", line 22, in main
    solover.train(dataloader)
  File "path to my file", line 239, in train
    writer.add_scalars("MSE",l_mse,n_iter)
  File "/~~/anaconda3/lib/python3.7/site-packages/torch/utils/tensorboard/writer.py", line 378, in add_scalars
    for tag, scalar_value in tag_scalar_dict.items():
AttributeError: 'Tensor' object has no attribute 'items'
</code></pre>
<p>I tried</p>
<pre><code class="lang-auto">writer.add_scalars("MSE",l_mse,n_iter).eval()
writer.add_scalars("MSE",l_mse.item(),n_iter)
writer.add_scalars("MSE",l_mse.detach().cpu().numpy(),n_iter)
</code></pre>
<p>but still not work well.<br>
tensorboard:1.15.0<br>
tenosrboard-plugin-wit 1.6.0.post3<br>
pytorch:1.5.0</p>

ptrblck: <p>I’m not deeply familiar with the usage of Tensorboard, but it seems that a <code>dict</code> is expected, which has the <code>items()</code> operation.<br>
Could you wrap your data into a <code>dict</code> and retry the call?</p>

tom: <p>There is <a href="https://pytorch.org/docs/stable/tensorboard.html#torch.utils.tensorboard.writer.SummaryWriter.add_scalar" rel="nofollow noopener"><code>add_scalar</code></a> (singular, so no s at the end) that would seem to work roughly like you want (except for the <code>.eval()</code> in there).<br>
You are calling <a href="https://pytorch.org/docs/stable/tensorboard.html#torch.utils.tensorboard.writer.SummaryWriter.add_scalars" rel="nofollow noopener"><code>add_scalars</code></a> (plural) which takes name/value pairs in form of a dict if you want to add several.</p>
<p>Best regards</p>
<p>Thomas</p>
Creating model from pretrained model and keep its weights

arian: <p>Hi, I’m new to PyTorch so excuse me beforehand for my rookie mistakes.</p>
<pre><code class="lang-auto">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.conv1 = nn.Conv2d(256, 1, 1)
        self.flatten = nn.Flatten()
        self.fc1 = nn.Linear(14*14, 1)

    def forward(self, x):
        output1 = F.sigmoid(self.conv1(x))
        output2 = F.sigmoid(self.fc1(output1))
        return output1, output2


class Final(nn.Module):
    def __init__(self):
        super(Final, self).__init__()
        pretrained_model = torch.hub.load('pytorch/vision:v0.6.0', 'densenet201', pretrained=True)
        model = MyModel()
        list_of_layers = list(pretrained_model.features)[:8]
        list_of_layers.extend(list(model.children()))
        self.final_model = nn.Sequential(*list_of_layers)

    def forward(self, x):
        outputs = list()
        for ii, model in enumerate(self.final_model):
            x = model(x)
            if ii == 8 or ii == 10:
                outputs.append(x)
        return outputs
</code></pre>
<p>final = Final()<br>
optimizer = optim.Adam(final.parameters())</p>
<p>This is  the way I managed to create my desired model at last. Pay attention that I couldn’t simply create a sequential model because I want to have two outputs and it’s not possible in sequential models as far as I know.</p>
<ol>
<li>Any comment on how to make it more simpler if possible?</li>
<li>(main question) If I train this model do I start with random weights for every layer or my dense section will have Its own pretrained weights as the starting point of training?</li>
</ol>
Same Loss value for 15 to 20 iterations

chetan06: <p>I am working on Dog Breed Classification problem. I am using pretrained model for training.</p>
<pre><code class="lang-auto"># Creating a model
model = models.resnet50(pretrained=True)
for param in model.parameters():
    param.requires_grad = False
num_features = model.fc.in_features
fc_layers = nn.Sequential(
                nn.Linear(num_features, 4096),
                nn.ReLU(inplace=True),
                nn.Dropout(p=0.1),
                nn.Linear(4096, num_classes),
                nn.ReLU(inplace=True),
                nn.Dropout(p=0.1),
            )
model.fc = fc_layers

</code></pre>
<p>This is how I have modified the pretrained model. I have used Adam as optimizer. And using a batchsize of 32.</p>
<pre><code class="lang-auto">optimizer = optim.Adam(model.parameters(), lr=0.001)
def criterion(yhat, y):
    label = yhat.gather(1, y.view(-1, 1)).squeeze()
    softmax_output = torch.exp(label)/torch.sum(torch.exp(yhat), axis = 1)
    loss = -torch.log(softmax_output)
    return loss.sum()
</code></pre>
<p>As the dataset is too large so in order to keep track of the loss I am printing loss after iteration.</p>
<p>In first epoch only I am getting same loss value after 15 to 20 iterations. The loss value is 153.109 so I don’t think this may be the convergence point. So I am not getting how to get out of this situation</p>

ptrblck: <p>I would recommend to remove the last <code>ReLU</code> and dropout layers and directly return the logits.<br>
Also, what’s the reason you are implementing the loss manually instead of using <code>nn.CrossEntropyLoss</code>?</p>

chetan06: <p>I got my answer. Thanks for your help <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
Memory profile on the fly data augmentation

JM_Sarr: <p>I would like to compare offline and online data augmentation (DA). In terms of time and memory usage during training.</p>
<ul>
<li>Offline DA happens when the augmentation is done before training.</li>
<li>Online DA happens in the data class and is performed for each mini-batch.</li>
</ul>
<p>So I am looking in how to memory profile the dataloader in pytorch with and without DA.<br>
Then I would like to profile the memory usage of the model during training.</p>
<p>I think DA happens on the CPU and the model is trained on the GPU.</p>
<p>How would you do that ?</p>
Determine peak memory requirement

pmeier: <p>Suppose I perform <code>my_awesome_cuda_training()</code> and I want to determine the maximum amount of memory that was used. Is that as simple as using <code>torch.cuda.memory_stats()</code>?</p>
<pre><code class="lang-python">import torch


def my_awesome_cuda_training():
    torch.empty(2, *[1024] * 3, dtype=torch.uint8, device="cuda")


my_awesome_cuda_training()

stats = torch.cuda.memory_stats()
peak_bytes_requirement = stats["allocated_bytes.all.peak"]
print(f"Peak memory requirement: {peak_bytes_requirement / 1024 ** 3:.2f} GB")
</code></pre>
<pre><code class="lang-auto">Peak memory requirement: 2.00 GB
</code></pre>
<p>It seems to work as intended, but I would be grateful for confirmation.</p>

ecdrid: <p>Hi,</p>
<p>You can try the below!</p>
<pre><code class="lang-auto"># pip install pynvml
from pynvml import * # too lazy, not a good thing
def show_gpu(msg):
    nvmlInit()
    handle = nvmlDeviceGetHandleByIndex(0)
    info = nvmlDeviceGetMemoryInfo(handle)
    total = info.total
    free = info.free
    used = info.used
    pct = used/total;
    print('\n' + msg, f'Used {100*pct:2.1f}% ({used} out of {total} {free}')
</code></pre>

pmeier: <p>That would get me the currently used memory, but that is not what I need: I need the peak memory requirement up to a specific point.</p>
<p>In the snippet above no CUDA memory is used when I <code>print</code> the peak requirement.</p>

ecdrid: <p>You can try this then, <a href="https://github.com/anderskm/gputil" rel="nofollow noopener">https://github.com/anderskm/gputil</a></p>

pmeier: <p>As far as I can see this also only gives me the current load and not the maximum load since I’ve started my script.</p>

ecdrid: <p>Well you can keep on collecting the stats over a span of time?</p>
<p>Because it will change with time, so nothing can give you stats straightforward, right?</p>

pmeier: <p>The whole point of determining the peak memory usage is that I don’t want do intermediate manually checking. For me it is irrelevant how much memory is used at a specific time. All I want is to determine after my code has run how much memory was used at a maximum, i. e. how much memory is required to run my code.</p>

ptrblck: <p>Yes, the <code>.peak</code> stats will give you the maximum. You can use <code>torch.cuda.reset_peak_memory_stats()</code> to reset this peak if you need to monitor another peak usage.</p>
Output layer activation and loss function

Ratan: <p>Hey guys, I am new to ML. I tried to use InceptionV3 for <strong>3 class output</strong> using transfer learning. I freezed all the parameters(including aux_logits= False).<br>
My output layer:<br>
(fc): Sequential(<br>
(0): Linear(in_features=2048, out_features=1024, bias=True)<br>
(1): ReLU()<br>
(2): Dropout(p=0.3, inplace=False)<br>
(3): Linear(in_features=1024, out_features=512, bias=True)<br>
(4): ReLU()<br>
(5): Dropout(p=0.3, inplace=False)<br>
(6): Linear(in_features=512, out_features=3, bias=True)<br>
)</p>
<p>I have seen in many cases there people dont use any activation for output layer. What is the reason?<br>
I have used Softman() at output layer and CrossEntropyLoss(). I am see only decreses in train loss and val loss decreases very slowly</p>

ptrblck: <p>If you are working on a multi-class classification use case and use <code>nn.CrossEntropyLoss</code>, your model should output raw logits, as internally <code>nn.CrossEntropyLoss</code> will apply <code>F.log_softmax</code> and <code>nn.NLLLoss</code>.</p>

Ratan: <p>Hey thanks for your reply!<br>
I tried to train my inception v3 but the val loss does not seem to decrease! I tried different learning rate but no luck there. I have a feeling that my model is too complex for my dataset. Any suggestions?<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/6/96b5500d28db8416920f48b1b923e1cc74ad7299.png" alt="1" data-base62-sha1="lve52h72GTzeZHZ5sHjOERmZo1z" width="458" height="97"></p>

ptrblck: <p>Your model seems to be overfitting pretty quickly, so I would suggest to add some regularization such as weight decay or increase the drop probability in your dropout layers.</p>

Ratan: <p>Hey!<br>
Thanks for the suggestions. I have kept the model for training with dropout(0.8) and also weight decay 0.01 and also included a scheduler for lr adjustment.</p>
<p>Is it possible to asses how changing the output layer from one mentioned above to given below effect:</p>
<p>Linear(in_features=2048, out_features=1024, bias=True)<br>
…<br>
…<br>
Linear(in_features=1024, out_features=3, bias=True)</p>
<p>All i can infer is that there are less parameters and might takes less time to train.</p>

ptrblck: <p>The output layer returns the class logits in a classification setup.<br>
Besides the training time, your model architecture and thus the use case would be changed from 1024 classes to 3 classes.</p>

Ratan: <p>The issue was resolved. The model converged at 40th epoch. with 97% val accuracy</p>
Custom dataset+loader for files from multiple folders with K-fold CV

sathvik_udupa: <p>Considering the CIFAR10 data with 5 training batches, 1 test batch in different folders and 20% of each train batch for validation, and an option to perform 5 fold CV within those train batches, is it possible to make a single dataset class to handle this? How do I keep track of indices for validation set if I want to split randomly? Should I be using a default dataset class and then split as needed?</p>

ptrblck: <p>You could use <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.KFold.html">sklearn.model_selection.KFold</a> to create the folds and wrap your <code>Dataset</code> into <code>Subset</code>s to create the folds.<br>
Alternatively you could also use <code>SubsetSampler</code>s and pass them to your <code>DataLoader</code>.</p>
<p>Also, <a href="https://github.com/skorch-dev/skorch">skorch</a> is a scikit-compatible wrapper for PyTorch, which might make the whole workflow easier.</p>
Can I split my batch in the forward method?

hollowcodes: <p>Hey, I am creating a DeepFake in Pytorch. My model has following architecture:<br>
x := input-image<br>
x -&gt; encoder -&gt; if: label_of_x == 0: -&gt; decoder1, elif label_of_x == 1: -&gt; decoder2</p>
<p>Its a conditional flow in which all the images pass through the same encoder but go through different decoders dependent on their class. Until now I used batchsize = 1 because I dont know if/how I can split the batch into two smaller batches (each containing all the input-images of the same class) and then merge them together for the backpropagation. Is this possible? And if so: How?<br>
I really appreciate any help!</p>

ecdrid: <p>IMO, yes you can do it by passing in another flag in your dataset (csv file) and selecting images based on that make, make sure that in a batch you have similar no of images of all classes; And then in the forward you can select what you want to-do accordingly easily;</p>

hollowcodes: <p>So do you think this is possible (pseudo-code):</p>
<pre><code class="lang-auto">...
def _split_classes(batch, targets):
    class_a, class_b = [], []
    for i, sample in enumerate(batch):

        if targets[i] == 1:
            class_a.append(sample)
        else:
            class_b.append(sample)

    return torch.Tensor(class_a), torch.Tensor(class_b)


def forward(x, targets):
    x = encoder(x)
    x_a, x_b = self._split_classes(x, targets)

    x_a = decoderA(x_a)
    x_b = decoderB(x_b)

    x = concat((x_a, x_b))
    return x
</code></pre>

ecdrid: <p>Yep similar to the pseudo code pretty much would be my init attempt; Note that during test_time you won’t have the targets with you;</p>

hollowcodes: <p>Okay thank you, ill try that!</p>

hollowcodes: <p>Wait, but how is backpropagation possible then?</p>

ecdrid: <p>Hey, Apologize for a delayed response. Why do you think backprop won’t happen or have issues with it?</p>
<p>We can always break up a bigger batch into smallers and still consider all those smallers ones as a single batch altogether. Similarly, when we have split out batch, now different I/ps are passing to different model code blocks, post forward pass, the grads will be combined and all of that will be send back together before the splitting point.</p>
<p>Also you night get little poor results but that depends on your BS as well,</p>
<p>Also you might want to look into collate_fn!</p>
Forward pass through different branches

welp: <p>Suppose I have this module. If the first node of the output of fc_type is higher than the second node, I want to forward pass through fc_1, else I want to forward pass through fc_2. Currently, my implementation requires me to put a for loop over each input in the batch. Is this sequential or does it get automatically parallelized somehow? If not, is there a better way of parallelizing it? Thanks!</p>
<pre><code>class mm(nn.Module):
    def __init__(self):
        super(mm, self).__init__()
        self.fc_type = nn.Linear(4,2)
        self.fc_1 = nn.Linear(4,1)
        self.fc_2 = nn.Linear(4,1)

    def forward(self, input):
        out = self.fc_type(input)
        y = torch.zeros(out.shape[0],1).to(input)
        for i in range(out.shape[0]):
             if out[i][0] &gt; out[i][1]:
                y[i] = self.fc_1(input[i:i+1])
             else:
                y[i] = self.fc_2(input[i:i+1])
        return y</code></pre>

ptrblck: <p>Instead of iterating each row in <code>out</code>, you could split <code>out</code> before into both inputs, call <code>fc_1</code> and <code>fc_2</code>, and create the output tensor using the results.<br>
Depending on the batch size, the performance difference might be insignificant.</p>
Making fully connected layer bi-directional

Shanto_Islam: <p>As far as my knowledge, when we make <code>nn.LSTM(.. hidden_dim, bidirectional = True)</code> we tend to make the next fully connected layer <code>self.fc1 = nn.Linear(hidden_dim*2..)</code>, with this I believe we make this fully connected layer bi-directional as well. If so, then could we make <code>self.fc2</code> bi-directional as well?</p>
<p>Please rectify me if I have wrong understanding.</p>

ptrblck: <p>You are doubling the number of input features, as the preceding LSTM gives you <code>num_directions * hidden</code> features.<br>
The linear layer itself is not “birirectional” in the sense as an LSTM. The incoming activation won’t be processed in a sequential/directional manner.<br>
You could of course increase the number of hidden units in the second linear layer as well.</p>
Reading of Torch Codes

111137: <p>Hi All,</p>
<p>I started to read source code of PyTorch in order to understand how construct deep learning framework.</p>
<p>At first, I checked what files are in torch directory.<br>
Current version is here;<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">
      <a href="https://github.com/IAMAl/Anatomy_of_PyTorch" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/52284359?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/IAMAl/Anatomy_of_PyTorch" target="_blank" rel="nofollow noopener">IAMAl/Anatomy_of_PyTorch</a></h3>

<p>Contribute to IAMAl/Anatomy_of_PyTorch development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>Best,<br>
S.Takano</p>
Deserialize tensor on iOS

seeker: <p>Our team are trying to deploy our model to iOS platform. We encountered some mysterious problems and we want to deserialize the tensors in iOS to locate the problem more precisely. As for now I cannot find any resource regarding achieving so I want to ask:</p>
<ol>
<li>Is it possible?</li>
<li>If so, how? And on what platforms those deserialized tensors can be read?</li>
</ol>

seeker: <p>Correction: Not deserialize but serialize.<br>
We want to check the value of tensors to see if we constructed it correctly, so we want to serialize tensors to files and deserialize to python to check whether the value is correct.</p>
<p>(BTW How can I edit my post?)</p>
Implementing multiple Keras Losses in PyTorch

sambaths: <p>Hi,</p>
<p>I am trying to implement the following Keras codes in PyTorch.</p>
<p>This is a loss function I am trying to port to pytorch, I couldn’t seem to figure out how to implement this in Pytorch</p>
<pre><code class="lang-auto">def gradient_penalty_loss(y_true, y_pred, averaged_samples,
                          gradient_penalty_weight):
    
    gradients = K.gradients(y_pred, averaged_samples)[0]
    gradients_sqr = K.square(gradients)
    gradients_sqr_sum = K.sum(gradients_sqr,
                              axis=np.arange(1, len(gradients_sqr.shape)))
    gradient_l2_norm = K.sqrt(gradients_sqr_sum)
    gradient_penalty = gradient_penalty_weight * K.square(1 - gradient_l2_norm)
    return K.mean(gradient_penalty)

  partial_gp_loss = partial(gradient_penalty_loss,
                          averaged_samples=averaged_samples,
                          gradient_penalty_weight=GRADIENT_PENALTY_WEIGHT)
</code></pre>
<p>Any help regarding implementing the loss function and porting the above codes to pytorch will be a great help.</p>
<p><strong>UPDATE</strong>:  I found the pytorch implementation in this <a href="https://github.com/caogang/wgan-gp/blob/ae47a185ed2e938c39cf3eb2f06b32dc1b6a2064/gan_cifar10.py#L116-L137" rel="nofollow noopener">link</a>. I am posting it here so that its going to help someone out in the future.</p>
Training using DDP with world_size 4 on a multi-gpu machine runs with only two GPUs being used

Daisy_Deng: <p>I want to run a test to see how the synchronization works. I assume that at the end of each batch, DDP would wait for the processes on the world_size GPUs to reach the synchronization point like backward pass to synchronize gradients. If only 2 GPUS processes started, I assume that at the end of first batch, the synchronization on the existing 2 GPUS would time out as the other two never started. What I observed is that the training continued with only 2 GPU processes. How to explain this? Is my understanding not correct?</p>

mrshenli: <p>Sorry about the delay.</p>
<blockquote>
<p>I assume that at the end of each batch, DDP would wait for the processes on the world_size GPUs to reach the synchronization point like backward pass to synchronize gradients. If only 2 GPUS processes started, I assume that at the end of first batch, the synchronization on the existing 2 GPUS would time out as the other two never started</p>
</blockquote>
<p>Yes, this is correct.</p>
<blockquote>
<p>What I observed is that the training continued with only 2 GPU processes. How to explain this? Is my understanding not correct?</p>
</blockquote>
<p>It should block on the DDP construct or the backward call. Could you please share a code snippet that reproduces the above behavior?</p>
Double dispatch for tensors

almeetb: <p>Is double dispatch automatically invoked for tensors to the appropriate kernels or does it have to be registered?</p>

ptrblck: <p>Could you give an example of your use case?<br>
Are you writing a custom extension or are you referring to the PyTorch backend in general?</p>

almeetb: <p>I am writing my own extension and I was wondering just for tensor creation <code>torch.randn(2, 2, device)</code> is dispatch stub always needed?</p>
<p>I noticed that not all devices in the device type / backend enums are handled in dispatch stub.</p>

ptrblck: <p>Ah OK.<br>
Sorry, I’m not familiar enough with the <code>DispatchStub</code> implementation especially for new types. <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
Backpropagation with requires_grad=False

sigma_x: <p>This is just to confirm my understanding on how autograd works, as I found the solution neither <a href="https://pytorch.org/docs/stable/autograd.html" rel="nofollow noopener">here</a> nor <a href="https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html" rel="nofollow noopener">here</a>.</p>
<p>In the following setup <code>m1, m2 and m3</code> are Pytorch Sequential models, <code>l1</code> is a loss function, <code>lab</code> are labels. Parts of m2 have <code>requires_grad=False</code>:</p>
<pre><code class="lang-auto">input = torch.randn(10)
o1 = m1(input)
o2 = m2(o1)
o3 = m3(o2)
l=l1(o3,lab)
l.backward()
</code></pre>
<p>My question is: since part (or even all) of <code>m2</code> do not compute gradients, will the gradients be automatically copied from m3 to m1 automatically by <code>backward</code>, or I have to do it outside of it?</p>

ptrblck: <p>Autograd won’t copy the gradients, but will properly backpropagate through all models up to the first parameter, which requires gradients.<br>
E.g. you could also freeze all models and set <code>requires_grad=True</code> for the input and will still get valid gradients for the input tensor.<br>
The frozen parameters won’t get their <code>.grad</code> attribute populated.</p>

sigma_x: <p>Thanks. By ‘freeze’, do you mean <code>requires_grad_(False)</code>?</p>

ptrblck: <p>Yes, that’s what I meant. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Assign a tensor to be the part of another tensor

jiashenC: <p>Let’s say I have two tensors</p>
<pre><code class="lang-auto">A = [[0, 0, 0, 0],
     [0, 0, 0, 0],
     [0, 0, 0, 0]]

B = [[1, 1, 1, 1],
     [2, 2, 2, 2],
     [3, 3, 3, 3]]
</code></pre>
<p>In addition to that, I have index tensor</p>
<pre><code class="lang-auto">Indices = [True, False, True]
</code></pre>
<p>So I want to put B[0] at A[0], and B[2] at A[2], but not B[1] because it is False.</p>
<pre><code class="lang-auto">C = [[1, 1, 1, 1],
     [0, 0, 0, 0],
     [3, 3, 3, 3]]
</code></pre>
<p>What would be the good API to achieve this function? I have tried scatter, index_fill, but I am not able to achieve that kind of usage. I understand this can be easily achieved by using iterative method, but I wonder can I achieve this from a vectorized way.</p>

jiashenC: <p>NVM</p>
<p>A[Indices] = B[Indices]</p>
<p>will do the job.</p>
Forgetting some information about some specific input elements in LSTM hidden state

Isaac_Kargar: <p>Hi,<br>
I have an LSTM network and I want to forget some information from prev time steps. I mean I don’t want to reset the whole hidden state, but just want to do not consider the history of some elements of input vector. Does anyone have any idea?</p>

googlebot: <p>Hidden state entangles input vector elements, so this is impossible I think. You’d have to somehow split your vector before rnn layer, and conditionally combine it back to produce rnn input (residual networks, multi-layer rnns, something like that…)</p>