akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Unable to infer on cpu a model trained on fp16

Raghav: <p>Hi I have trained the transformer network from fairseq using fp16 on turing gpu. The result I get a model about half the size of original fp32 model I trained. Now I want to infer the fp16 model trained on cpu and I get the following error.<br>
<code>RuntimeError: _th_index_select is not implemented for type torch.HalfTensor</code></p>
<p>Is there something I missed ? Please help</p>

ShangxuanWu: <p>I have the same problem. I export a FP16 model, and it works fine on GPU, but once switch to CPU I got this:</p>
<pre><code class="lang-auto">_thnn_conv2d_forward is not implemented for type CPUHalfType (_thnn_conv2d_forward at aten/src/ATen/TypeDefault.cpp:1940)
</code></pre>
Filter normalisation

sebastienwood: <p>Hi <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I’m trying to multiply filters by the average value of another filter of the same dimension (that is a copy of this filter, with few modifications). When I get params they look like 4 dimensional or 3 dimensional tensors, with if I got it correctly the filter are at the last two dims.</p>
<p>Then I do :<br>
<code>param_tensor *= torch.mean(param_tensor, (2,3)).expand_as(param_tensor)</code></p>
<p>I expect that the tensor would thus have an element wise mult with the average being expanded in the 2 lasts dimensions.</p>
<p>However it fails with the following error message : <code>The expanded size of the tensor (3) must match the existing size (16) at non-singleton dimension 2. Target sizes: [16, 3, 3, 3]. Tensor sizes: [16, 3]</code>.</p>
<p>I looked into weight norm but it seems slightly different from what I want to achieve. If anyone knows how to do so it would help, thanks ! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>Try to pass <code>keepdim=True</code> to the <code>mean</code> call:</p>
<pre><code class="lang-python">param_tensor *= torch.mean(param_tensor, (2,3), keepdim=True).expand_as(param_tensor)
</code></pre>
RuntimeError: Invalid index in scatter

snoow: <p>My error’s details are as follows:<br>
<code>Traceback (most recent call last): File "train.py", line 184, in &lt;module&gt; train_hist = train(train_loader, num_classes, device, net, optimizer, criterion) File "train.py", line 123, in train loss = criterion(outputs, masks) File "/home/yp/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__ result = self.forward(*input, **kwargs) File "/media/yp/My Passport/dl_code2/satellite_segmentaion-master/utils/loss.py", line 76, in forward masks = torch.zeros(N, C, H, W).to(targets.device).scatter_(1, targets.view(N, 1, H, W), 1).permute(1, 0, 2, 3) RuntimeError: Invalid index in scatter at /opt/conda/conda-bld/pytorch_1550802451070/work/aten/src/TH/generic/THTensorEvenMoreMath.cpp:549</code><br>
The code of loss:`<br>
def forward(self, inputs, targets):</p>
<pre><code>    N, C, H, W = inputs.size()

    softs = nn.functional.softmax(inputs, dim=1).permute(1, 0, 2, 3)
    masks = torch.zeros(N, C, H, W).to(targets.device).scatter_(1, targets.view(N, 1, H, W), 1).permute(1, 0, 2, 3)
    print("mIoULoss2d-------outputs'size:{},masks'size:{}".format(softs.size(),masks.size()))

    inters = softs * masks
    unions = (softs + masks) - (softs * masks)

    miou = 1. - (inters.view(C, N, -1).sum(2) / unions.view(C, N, -1).sum(2)).mean()

    return max(miou, self.nll_loss(nn.functional.log_softmax(inputs, dim=1), targets)) `  
</code></pre>
<p>How can I solve it? Thanks</p>

ptrblck: <p>Could you post the values of <code>targets</code> as well as <code>inputs.size()</code> so that we can take a look?</p>
Help with GAN with rectagular data

rodrigodupleich: <p>Hi all, i’m working on time series data applications of GANs, the dataset is usually non-square (e.g. Time 100 days x 10 features, 1 channel).</p>
<p>Lots of codes (e.g. in github) are designed for square images. If you modify the code and once you change input/output sizes, kernel sizes, strides and padding; tbh i cannot the sizes right different layers of the of the CNN (in particular for generator net).</p>
<p>any help here? tricks? thanks</p>

Taxaria: <p>Hey <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
I’m doing actually a similar thing for my bachelorthesis. My GAN has to work with CNNs and an additional context vector, which I haven’t implemented yet.<br>
I tried to build a GAN with STGCN. That are CNNs for spatio temporal data. They use non square kernelsizes for the convolution.</p>

rodrigodupleich: <p>thanks. this a good reference Taxaria:</p>
<aside class="onebox pdf">
  <header class="source">
      <a href="https://arxiv.org/pdf/1603.07285.pdf" target="_blank" rel="nofollow noopener">arxiv.org</a>
  </header>
  <article class="onebox-body">
    <a href="https://arxiv.org/pdf/1603.07285.pdf" target="_blank" rel="nofollow noopener"><span class="pdf-onebox-logo"></span></a>
<h3><a href="https://arxiv.org/pdf/1603.07285.pdf" target="_blank" rel="nofollow noopener">1603.07285.pdf</a></h3>

<p class="filesize">892.34 KB</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Resizing images in with DataLoader

MasayoMusic: <p>I am going through the ant bees transfer learning tutorial, and I am trying to get a deep understanding of preparing data in Pytorch. I removed all of the transformations except ToTensor, but it seems<br>
you need to make sure images need to be resized?</p>
<p>So I am trying this:</p>
<blockquote>
<p>train_data = ImageFolder(root = os.path.join(root_dir, ‘train’),    transform=transforms.Compose([Resize(256),ToTensor()]))<br>
train_loader = iter(dataloader)<br>
x,y = next(train_loader)</p>
</blockquote>
<p>However I am still getting a runtime Error:</p>
<pre><code>`Sizes of tensors must match except in dimension 0. Got 371 and 375 in dimension 2 at ..\aten    \src\TH/generic/THTensor.cpp:711`
</code></pre>
<p><code>Preformatted text</code>Is there another transformation I need to add?</p>
<p>PS: Is there no preview button on these boards? I am trying out different formating options but I can’t seem to find a way to preview the options. Thank you. .</p>

Balamurali_M: <p>If size is a sequence like (h, w), output size will be matched to this.</p>
<p>If size is an int, smaller edge of the image will be matched to this number. i.e, if height &gt; width, then image will be rescaled to (size * height / width, size)</p>
<p>I think this is the problem. Instead of Resize(256) try, Resize(256,256). This will alter the aspect ratio. But, first check and let us know the outcome.</p>

MasayoMusic: <p>Yes that seems to be case. Thank you so much.</p>
Custom 1D kernel: Set Columns as zeros

Manish_Keshri: <p>Hi,<br>
Given below is my Conv1d code:<br>
nn.Conv1d(2, 4, kernel_size = 6, stride = 1, padding = 1)<br>
In these 4 kernels, I want them to have different columns to be zeros. For example, let’s say first kernel has second column as all zeros. So, it would be something like this:<br>
[w11, 0, w13, w14, w15, w16]<br>
[w21, 0, w23, w24, w25, w26]<br>
How Can I do this?</p>

ptrblck: <p>You can manipulate the parameters in a <code>torch.no_grad()</code> block:</p>
<pre><code class="lang-python">conv = nn.Conv1d(2, 4, kernel_size = 6, stride = 1, padding = 1)
with torch.no_grad():
    conv.weight[0, :, 1] = 0.
</code></pre>

Manish_Keshri: <p>Actually, my code looks like this:<br>
self.convnet1 = nn.Sequential(<br>
nn.Conv1d(2, 4, kernel_size=6, stride=1, padding=1),<br>
nn.ReLU(inplace=True),)<br>
Can you tell me how to implement it here?</p>

Manish_Keshri: <p>Also, I want to learn the non-zero weights.</p>

Manish_Keshri: <p>I implemented something like this:<br>
self.weight = torch.randn((2, 4, 6), requires_grad = True)</p>
<p>self.convnet1 = nn.Sequential(<br>
nn.Conv1d(2, 4, kernel_size=6, stride=1, padding=1),<br>
nn.ReLU(inplace=True),)</p>
<p>self.convnet1.weight = nn.Parameter(self.weight)</p>
<p>But the weights are not updating…</p>

Manish_Keshri: <p>I tried to implement like this:<br>
self.weight = torch.randn((2, 4, 6), requires_grad = True)<br>
with torch.no_grad():<br>
self.weight[:,0,0] = 0<br>
self.weight[:,1,1] = 0<br>
self.weight[:,2,0] = 0<br>
self.weight[:,3,1] = 0<br>
self.convnet1 = nn.Sequential(<br>
nn.Conv1d(2, 4, kernel_size=6, stride=1, padding=1),<br>
nn.ReLU(inplace=True),)<br>
self.convnet1.weight = nn.Parameter(self.weight)</p>
<p>But I get error in the following line:<br>
for p in model.parameters():<br>
p.grad.data.div_(batch_size)  # divide by the actual batch size<br>
total_norm += p.grad.data.norm() ** 2</p>
<p>Error: p.grad has no variable named data.</p>

Balamurali_M: <p>Since you want to update it for conv layer alone. The code should not be self.convnet1.weight, it should be self.convnet1[0].weight. Make this change and check.<br>
self.weight = torch.randn((2, 4, 6), requires_grad = True)<br>
self.convnet1[0].weight = nn.Parameter(self.weight)</p>
Being notified once lr changed by lr_scheduler?

ljj7975: <p>I would like to log the change in lr throughout the training when it happens.</p>
<p>What is the right way of checking if lr has changed?</p>
<p>I am currently using stepLR but I guess it should be vaild for any lr scheduler</p>
<p>Also if there is a way to check what the current step is from lr_scheduler instance. that can help since most lr change depends on # of steps.<br>
This part can be done by checking # of epoch but for debugging purpose, it must come from lr_scheduler instatnce</p>

LeviViana: <p>You could write your own lr_scheduler by basically copying and pasting the <a href="https://pytorch.org/docs/stable/_modules/torch/optim/lr_scheduler.html#StepLR" rel="nofollow noopener">source code of StepLR</a>. You could integrate a variable <code>verbose</code> that when set True, it warns you when the LR is being changed. It is actually already implemented in the <code>ReduceLROnPlateau</code>.</p>

ljj7975: <p>hrm… I am surprised that it’s not available.<br>
okay. thanks</p>
Asking for more than 128G of memory

Geeks_Sid: <p>Hi all,</p>
<p>So I trained a 3D-UNet with 16 base filters and 5 layers deep. Now I am trying to infer it on a 240x240x155 on a CPU. I have allocated 128GB of ram, it still pops out with an error.</p>
<p>RuntimeError: $ Torch: not enough memory: you tried to allocate 0GB. Buy new RAM! at /opt/conda/conda-bld/pytorch</p>
<p>I do not have more money to buy new ram, The model should require at the most 32GB of ram for that image.</p>
<p>Can I know where I may be going wrong?</p>
<p>Thanks,<br>
Siddhesh</p>

driazati: <p>Can you post the code for the model? Also did this occur when executing a TorchScript function/module or a normal <code>nn.Module</code>?</p>

Geeks_Sid: <p>I cannot post the model since it is an ongoing work. But I can confirm that I trained this model on a 16GB GPU.</p>

ptrblck: <p>As far as I understand your issue, the training script takes 16GB at most running on the GPU and more than 128GB on the CPU?<br>
If that’s correct, do you see an increasing memory usage during training or does your script run out of memory during the first iteration?<br>
Did you change something in your data loading pipeline, e.g. are you loading the complete dataset into RAM?</p>

Geeks_Sid: <p>Oh I was not even talking about training, it is the cost of inference on a single example.</p>
RuntimeError: Expected hidden[0] size (1, 64, 256), got (64, 256)

Shandilya21: <p>I unable to solve this problem, i even tried with print all the h_0 , and decoder has batch_first = True.<br>
I have two encoder and after concatenating the representation of these two , i will get the required output.<br>
my final encoder shape after concatenation [64,43,256] : [B x Seq_len x hidden_dim ]</p>
<p>here is the code snippet:</p>
<p>def forward(self, input_src, input_trg, courteous_template, ctx_mask=None, trg_mask=None):<br>
“”“Propogate input through the network.”""<br>
src_emb = self.src_embedding(input_src)<br>
trg_emb = self.trg_embedding(input_trg)<br>
temp_emb = self.temp_embedding(courteous_template)</p>
<pre><code>    self.h0_encoder, self.c0_encoder = self.get_state(input_src)
    self.h1_encoder, self.c1_encoder = self.get_courteous(courteous_template)

    src_h, (src_h_t, src_c_t) = self.encoder(
        src_emb, (self.h0_encoder, self.c0_encoder)
    )
    

    temp_h, (tmp_h_t, tmp_c_t) = self.temp_encoder(
        temp_emb, (self.h1_encoder, self.c1_encoder)
    )

    out = torch.cat((src_h, temp_h),1)


    out = out.reshape(out.size(1),
    	out.size(0),out.size(2))

    # print(out.size())

    h_t = out[-1]
    h = h_t.view(h_t.size(0),h_t.size(1))



    trg_h, (_,_) = self.decoder(
    	trg_emb, h_t.view(
    		h_t.size(0),
    		h_t.size(1))
    	)
</code></pre>
<p>I got this error while training<br>
RuntimeError: Expected hidden[0] size (1, 64, 256), got (64, 256),<br>
i just can’t find out what is the problem?</p>
Why is nn.Module not a decorator?

vainaijr: <p><span class="mention">@nn.Module</span><br>
class Net():</p>
<p>why do we not have this?</p>

ptrblck: <p>I’m not sure what the benefits would be, as you would still need to override the usual methods (<code>__init__</code> and <code>forward</code>). Could you elaborate a bit on this topic?</p>

vainaijr: <p>I got idea by listening that decorators are replacing inheritance in PyCon 2019 talk.</p>
<p>some reasons are listed here -<br>
<aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/4842978/decorator-pattern-versus-sub-classing" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/389288/gainster" target="_blank" rel="nofollow noopener">
    <img alt="Gainster" src="https://www.gravatar.com/avatar/cd4470a71e115beb008aad94d23a79db?s=128&amp;d=identicon&amp;r=PG" class="thumbnail onebox-avatar" width="60" height="60">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/4842978/decorator-pattern-versus-sub-classing" target="_blank" rel="nofollow noopener">Decorator pattern versus sub classing</a>
</h4>

<div class="tags">
  <strong>design-patterns, decorator</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/389288/gainster" target="_blank" rel="nofollow noopener">
    Gainster
  </a>
  on <a href="https://stackoverflow.com/questions/4842978/decorator-pattern-versus-sub-classing" target="_blank" rel="nofollow noopener">12:52PM - 30 Jan 11 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
Estimate memory taken by network for one epoch

rooks: <p>Hi</p>
<p>I would like to have a method to compute/estimate the amount of memory that would be used by the network during one epoch. I am not considering the size of the inputs here, only the memory taken by the network and its internal parameters.</p>
<p>I know the network will computes the gradients of the parameters during the backward pass, and I think it is straightforward to compute the memory taken by the gradients.</p>
<p>But I guess there are several things computed and stored during the backward and forward passes of the training. But to be honest I have no idea what is stored and takes memory.</p>
<p>Could you advise me a strategy for this?</p>
<p>Thanks.</p>

Pranavan_Theivendira: <p>Hi,</p>
<pre><code class="lang-auto">from torchsummary import summary

summary(net, (3, 32, 32))
</code></pre>
<p>Here net(torch.nn) is your particular network and replace (3,32,32) with your image size. This gives you estimated parameter size as well as the total size. There may be other metrics. This may help you.</p>
Pycharm fails at transfer learning tutorial

FaultyBagnose: <p>Sorry if this is off topic!<br>
Tryng to run the transfer learning tutorial<br>
<a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html</a><br>
on Windows 10.<br>
It works like a charm in the Anaconda command shell, but fails in Pycharm.<br>
It crashes at the statement</p>
<h1>Get a batch of training data</h1>
<p>inputs, classes = next(iter(dataloaders[‘train’]))</p>
<p>The error message suggests a problem with forking and mentions freeze_support, but this seems unlikely.</p>
<p>Has anyone been able to run this tutorial in Pycharm?</p>
<p>All the best<br>
FaultyBagnose</p>

FaultyBagnose: <p>More specifically, it’s iter(dataloaders[‘train’]) that fails.<br>
The Dataloader object looks reasonable in the debugger.<br>
I’ve tried adding<br>
multiprocessing.freeze_support()<br>
prior to the statement, to no effect.</p>
<p>Since the code works in the Anaconda CMD Prompt, it’s basically a Pycharm problem, right?</p>

FaultyBagnose: <p>Solved.<br>
Apparently, the problem did have to do with forking, and the solution was to put the code (not just the iter part) in a<br>
if <strong>name</strong> == ‘<strong>main</strong>’:<br>
block.</p>
Learning speed according to the number of parameters

9c36bfac2ae128568766: <p>I removed FC LAYER and replaced it with global average pooling to reduce the number of parameters in VGG16. Actually, the parameter was 10 times different. But the actual speed of learning doesn’t seem to have improved much. What is the reason?</p>

ptrblck: <p>Did you profile your code to see where the bottleneck is?<br>
It might be that e.g. your data loading is the bottleneck so that reducing the operations in your model won’t change anything.</p>
<p>Also, did you time the linear vs. pooling layer properly and made sure the pooling layer is indeed faster?</p>
MNIST autograd profiling

dhakshiin: <p>Hi,</p>
<p>In the code <a href="https://github.com/pytorch/examples/blob/master/mnist/main.py" rel="nofollow noopener">https://github.com/pytorch/examples/blob/master/mnist/main.py</a> I wanted to estimate the profiling information for each layer in my model.<br>
Is the placement of the profiler function correct.</p>
<p>def train(args, model, device, train_loader, optimizer, epoch):<br>
model.train()<br>
for batch_idx, (data, target) in enumerate(train_loader):<br>
data, target = data.to(device), target.to(device)<br>
optimizer.zero_grad()</p>
<pre><code>	***with torch.autograd.profiler.profile() as prof:***
</code></pre>
<p>***			output = model(data)***<br>
***			loss = F.nll_loss(output, target)***<br>
***			loss.backward()***<br>
***		print(prof)***</p>
<pre><code>	optimizer.step()
	if batch_idx % args.log_interval == 0:
		print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
		epoch, batch_idx * len(data), len(train_loader.dataset),
		100. * batch_idx / len(train_loader), loss.item()))
</code></pre>
<p>Do I have to make <em><strong>"requires_grad=True"</strong></em> anywhere in the code before running the profiler</p>
Sort a subpart of an Aten tensor?

wshen: <p>Hi there,</p>
<p>I’m wondering if we can sort a subpart of an ATen tensor, specified with a start index and an end index? More generally, can we create inplace an ATen sub-tensor from full tensor? Similar to C++ array, where we just pass in a different pointer.</p>
<p>Thanks.</p>

LeviViana: <p>You can simply use <code>std::sort</code> (for CPU tensors) or <code>thrust::sort</code> (for CUDA tensors) on the tensor’s data pointer, given that your tensor is contiguous.</p>
<p>The snippet below is built for libtorch, but the behavior is exactly the same in ATen:</p>
<pre><code class="lang-auto">#include &lt;torch/torch.h&gt;
#include &lt;iostream&gt;

int main(){
	torch::Tensor x = torch::tensor({1, 3, 4, 2, 0, 5, 9});
	int *x_ptr = x.data&lt;int&gt;();
	std::sort(x_ptr + 1, x_ptr + 4, std::less&lt;int&gt;()); // sorting {3, 4, 2}
	std::cout &lt;&lt; x &lt;&lt; std::endl; // prints 1 2 3 4 0 5 9
	return 0;
}
</code></pre>

wshen: <p>Thanks for the help. Appreciate it.</p>
Saving the out of the layer to view them as an image

Naina_Dhingra: <p>Hi, I am working with residual networks. I want to view features of intermediated layer as an image. How do I do that if</p>
<pre><code>               def residual_unit:
                return  res1, outAfterMask, mask
</code></pre>
<p>when I call that in my model:</p>
<pre><code>                 self.res1 = resudal_unit(128, 128)
</code></pre>
<p>How do I save mask and view as an image. What I think if I can save them as an numpy array in .hf5 file and then view it using matplotlib. I am not sure how to save using a numpy array. Is it when I call the method in my model ? Any suggestions will be helpful</p>

Ierezell: <p>Hi,<br>
You can do it with matplotlib :</p>
<pre><code class="lang-auto">plt.imshow(MyTensor.permute(1, 2, 0).cpu().detach().numpy())
</code></pre>
<p>Or with PIL :</p>
<pre><code class="lang-auto">from torchvision import transforms
transforms.ToPILImage()(MyTensor).show()
</code></pre>
<p>In any case, I think your tensor needs to be 3xWidthxHeight</p>

Naina_Dhingra: <p><span class="mention">@lerezell</span> Thank you for your answer. My input is a video. So, I want to see one of its frame. But my question is when I call that method, do I just call it as</p>
<pre><code>                self.res1, self. outAfterMask, mask = residal_unit(128, 128)
</code></pre>
<p>Later,</p>
<pre><code>                              output=model(input)
</code></pre>
<p>How do I get that MyTensor?</p>

Ierezell: <p>MyTensor should be any tensor (3xHxW) that you want to see.</p>
<p>From a video you can get it with openCv or other tools</p>
<p>For you architecture, I don’t know what you’re trying to do.</p>
Fast way to generate sample neural networks from a per-weight distribution

LMA: <p>I have the distribution of each weight (and bias) of a neural network. What is a fast way to generate sample neural networks? To be specific, the distribution is assumed to be Gaussian and the mean and variance are stored in two dictionaries.</p>
<p>I have a small ResNet with about 400,000 parameters (weights and biases). The fastest way I got so far takes about 1 minute to generate a sample neural network on a GPU. Any ideas how to speed it up?</p>
<p>My code is as follows. It iterates through individual weights and biases to generate a sample of a scalar Gaussian random variable at a time.</p>
<p>I also tried generating a vector (the same size as that of <code>param</code>) at a time by using torch.distributions.MultivariateNormal and diag() matrices, but I got an out-of-memory error.</p>
<pre><code class="lang-auto">def generate_net(dic1, dic2, sample_net):  
    # dic1: dictionary storing mean
    # dic2: dictionary storing variance
    # sample_net: model of a network
    for name, param in sample_net.named_parameters():
        param1 = dic1[name]
        param2 = dic2[name]
        variance_final = param2 - torch.mul(param1, param1)
        size_tmp = param2.size()
        param1 = param1.view(-1, 1)  # make it a ?x1 vector
        variance_final = variance_final.view(-1, 1)
        param = param.view(-1,1)
        indx = 0
        for wt1, var in zip(param1, variance_final):
            rv = torch.distributions.Normal(wt1, torch.sqrt(var) * damp_factor)
            param[indx, 0] = rv.sample()
         
        param = param.view(size_tmp)  # get back to original size
        param1 = param1.view(size_tmp)
        param2 = param2.view(size_tmp) 

</code></pre>
<p>Thanks.</p>
KeyError: 'module name can\'t contain "."'

HaziqRazali: <p>I am trying to run a 3D DenseNet <a href="https://github.com/kenshohara/3D-ResNets-PyTorch" rel="nofollow noopener">https://github.com/kenshohara/3D-ResNets-PyTorch</a> but receive the error <code>KeyError: 'module name can\'t contain "."'</code></p>
<p>I found out that it is because of the 0.4.0 version of pytorch. Which version of pytorch do I need in order to not have that error? I have tried 0.3.1 but I get the error <code>AttributeError: module 'torch.nn' has no attribute 'ModuleDict'</code></p>

Balamurali_M: <p>Can you give more details ? Where exactly in the code you got the error ? When did you get the error during training or testing.? Is the error due to model loading ?</p>
DQN eps_decay represents what?

Milky: <p>I dont fully understand the eps_decay parameter in the DQN example. It is explained the eps_start represents the chance to select a random action at the beginning of the training and eps_ends is the chance to select a random action at the end of the training. With these two variables, one would assume the decay rate from the eps_start to the eps_end chance would depend on the number of episodes the training takes. So, from 0.9 chance to 0.1 is a difference of 0.8 and you can divide this by the number of episodes that will happen, and you get the decay rate per episode. So, my question is how does eps_decay come into this? I can understand if you want it to fully decay not at the end of the training but let’s say at half of the episodes, you could double the decay rate. However, the given example parameter is for example 200 in the cartpool environment. What does this 200 represent?</p>

Oli: <p>Hi there <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>TLDR: it’s just a fancy way to decay the epsilon</p>
<p>As you say, if you want the parameter to decay halfway you can change this via this parameter. Have a look at this <a href="https://www.desmos.com/calculator/cdepi0lme1" rel="noopener nofollow ugc">link</a> where you can change the value to see how the decay changes. If eps_decay = 100, then the eps will be 0.363 at time step 100.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/4/403539c4ba2dd87c2f01f9fbfa08f555263c2dda.png" data-download-href="https://discuss.pytorch.org/uploads/default/403539c4ba2dd87c2f01f9fbfa08f555263c2dda" title="42"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/4/403539c4ba2dd87c2f01f9fbfa08f555263c2dda_2_690x158.png" alt="42" data-base62-sha1="9a0ANCPu3cG03S01Px2LoFOeN06" width="690" height="158" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/4/403539c4ba2dd87c2f01f9fbfa08f555263c2dda_2_690x158.png, https://discuss.pytorch.org/uploads/default/optimized/2X/4/403539c4ba2dd87c2f01f9fbfa08f555263c2dda_2_1035x237.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/4/403539c4ba2dd87c2f01f9fbfa08f555263c2dda_2_1380x316.png 2x" data-dominant-color="F8F9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">42</span><span class="informations">2232×514 49.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
GAN with CNNs and an additional context as input

Taxaria: <p>Hey <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I’m trying to build a GAN with CNNs and an additional context as input.<br>
The context should be input for the Generator and the Discriminator. The Generator should generate data with that specific context and the Discriminator knows what input comes in.<br>
Actually I’m using normal CNNs for my GAN and the MNIST dataset. I want to expend my model to other datasets later.<br>
If I use linear layers, I can just concat the input and my context (for example the label of the image as one-hot), but I dont’t know how it’s possible to give the CNNs an additional context.</p>
<p>Thank you for your help!</p>

ptrblck: <p>Your use case sounds similar to <a href="https://arxiv.org/abs/1411.1784" rel="nofollow noopener">Conditional GANs</a>. You can find some implementations on GitHub, e.g. <a href="https://github.com/malzantot/Pytorch-conditional-GANs" rel="nofollow noopener">here</a>.</p>

Taxaria: <p>Thanky you for your answer.<br>
I already implemented a Conditional GAN for normal MNIST. For the Generator I concat the noise with the label as one-hot vectors. That works very well.<br>
My current project is a GAN with a kind of Graph-CNNs, which should generate MNIST-Superpixel-Graphs.<br>
My problem is, I can’t imagine how I connect my input for the Generator and the Discriminator with the context.<br>
Actually I try to implement a working Generator for MNIST-Superpixel-Graphs. This is still a problem, too.</p>
Passing List of Variable-size Tensors to Module

Eric_Taw: <p>Current, I can pass a list of Tensors, all of which vary in size, to a Module and run forward on them. However, because they’re variable in size, I cannot use <code>torch.bmm()</code> or similar to quickly multiply each Tensor by another Tensor. Instead, I’m using a for-loop, finding the right weights to use based on the shape of the Tensor, and multiplying in a sequential fashion. Since these matrices are small, I’m not effectively using my GPU. Is there any way to optimize this operation?</p>
<p>Edit: padding is not an option for me.</p>

ptrblck: <p>If padding is not an option, would it be possible to somehow sort the samples and pass equally sized batches to the model?</p>

Eric_Taw: <p>Yeah that might be my best bet. I was just hoping for something slightly more graceful.</p>
Two PyTorch runs on a single GPU take twice as much time as each separately

Deeply: <p>I am using a 12 GB GPU (Titan Xp) and I was surprised to notice that running two experiments (each takes around 4GB of memory; at different terminals) of the same PyTorch program need twice as much time as each separately.</p>
<p>I’m not sure if I miss something here, or I need to adjust some parameters to improve speed performance. Or, so be the situation in this case!</p>
<p>NB. I am using 8 workers in the dataloader.</p>

albanD: <p>Hi,</p>
<p>You can check with <code>nvidia-smi</code> but most network are not limited by memory but how much the GPU can compute. It is the field called <code>GPU-Util</code> in nvidia-smi.<br>
If one job already uses 100% (or close), then the GPU is fully used and so the jobs will have to run one after the other, leading to the slower runtime.</p>

Deeply: <p>Gush, you are absolutely right, I am confusing memory with utilization.</p>
Using square roots in activations functions

Pablo: <p>Hello everyone,</p>
<p>I was trying to reproduce a model where the activation functions is defined as:<br>
x / sqrt(1 + x**2)</p>
<p>were x are vectors</p>
<p>I was wondering how can I define this operation and still benefit from autograd magic.</p>
<p>Thank you very much in advance.</p>

alepack: <p>Hi,</p>
<p>I think that if you write a simple function like</p>
<p>import torch</p>
<p>def sqrt_act_fun(x):<br>
num = x<br>
den = torch.sqrt(1.0 + torch.mul(x,x))<br>
return torch.div(num, dev)</p>
<p>Provided that “x” is a torch tensor and that you use the output to compute (directly or indirectly) the loss value L, then the gradients will be automatically handled by the autograd package when you call L.backward().</p>
<p>Alternatively you can define a custom nn.Module, overriding its forward() method with the same code as in the function above, and then use it as a new layer in a DNN (for example).</p>
<p>Hope this helps <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Pablo: <p>Thank you very much.<br>
I tried the first solution you propose and worked perfectly!!</p>
Linux binaries of version 0.4.0 for cuda90 missing

PRUDHVI_KRISHNA_NIKK: <p>I require the cuda90 cp36 build of pytorch version 0.4.0 and I am unable to find it in pytorch previous versions. Could anyone help me finding it ?</p>

albanD: <p><code>pip install torch==0.4.0 -f https://download.pytorch.org/whl/cu90/stable</code> works for me. If this gives you an error that means that no binary exist for your version of python which is possible as we did not have that many builds back then.</p>

PRUDHVI_KRISHNA_NIKK: <p>Thanks a lot it is working for me.</p>
Difference between torch.sum(x, args) and x.sum when x is tensor

Deeply: <p>It seems that <code>torch.sum(x, keepdim=True, dim=...)</code> provides more control over <code>x.sum()</code>, when <code>x</code> is <code>tensor</code>.</p>
<p>However, which form is more appropriate to be used in customized loss functions that are used in <code>backpropagation</code>?</p>
<p>And, how about speed?</p>

albanD: <p>Hi,</p>
<p>As of latest version, they actually have exactely the same arguments. They will do the exact same thing with the autograd.</p>
Deploying Pytorch Model to server for Android

mabdullahrafique: <p>Hello, Guys !<br>
Sorry this question may be not related to pytorch problem, but I am not able to find this answer any where.<br>
Actually I am working on deploying GAN model written in pytorch to Android Device.<br>
But the problem I am facing is, it takes 2 gb ram for one image. So please tell me, which server does support pytorch models deployment, and how can I connect it with mobile device.<br>
Thanks</p>

albanD: <p>Hi,</p>
<p>You can use pytorch on any cloud provider. You might even have machines already setup with pytorch if you use big ones.<br>
Then I guess you will have to send the image to the server which is going to answer with the prediction. Using http requests for that should be fine.</p>

Shbnm21: <p>Can u tell ,how did u deploy pytorch gan model on Android ?</p>

Shbnm21: <p>How to deploy  pytorch based gan model  on Android ?</p>
Why does model.named_children() not return nn.Parameter?

yunjey: <p>Hi,</p>
<p>I observed that <code>model.parameters()</code> returns <code>nn.Parameter</code>, but <code>model.children()</code> (and also <code>named_children()</code>) does not return <code>nn.Parameter</code>. Is this intended? I think <code>model.children()</code> should also return <code>nn.Parameter</code>.</p>
<p>I wrote a simple reproducible code below. I tested the code in v1.0.0.</p>
<p>Thanks,</p>
<pre><code class="lang-python">import torch.nn as nn


class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc = nn.Linear(512, 512)
        self.alpha = nn.Parameter(torch.tensor(1.0))

    def forward(self, x):
        return self.alpha * self.fc(x)

model = MyModel()

print(list(model.named_children())) # this retuns only self.fc
</code></pre>

albanD: <p>Hi,</p>
<p>The <a href="https://pytorch.org/docs/stable/nn.html?highlight=children#torch.nn.Module.children" rel="nofollow noopener">doc</a> is maybe not super clear but <code>.children()</code> returns all the <code>nn.Module</code>s that are fields of this one. <code>parameters()</code> will return  the <code>nn.Parameter</code>.</p>
Model.summary in pytorch?

MasayoMusic: <p>Hi. I am going over the transfer learning tutorial with ants and bees, and I am coming from Keras with no experience in Pytorch.</p>
<p>Is there a built-in equivalent of model.summary in pytorch?</p>
<p>I found this medium article but it requires installation of another framework:</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://cdn-static-1.medium.com/_/fp/icons/favicon-rebrand-medium.3Y6xpZ-0FSdWDnPM3hSBIA.ico" class="site-icon" width="16" height="16">
      <a href="https://towardsdatascience.com/model-summary-in-pytorch-b5a1e4b64d25" target="_blank" title="10:28AM - 27 September 2018" rel="nofollow noopener">Towards Data Science – 27 Sep 18</a>
  </header>
  <article class="onebox-body">
    

<h3><a href="https://towardsdatascience.com/model-summary-in-pytorch-b5a1e4b64d25" target="_blank" rel="nofollow noopener">Model summary in PyTorch</a></h3>

<p>Keras style model.summary() in PyTorch</p>

  <p><span class="label1">Reading time: 3 min read</span>
    </p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Just wondering if that is indeed what I should be doing.<br>
Thank you.</p>

yuanzhoulvpi: <p>you can use modelsummary which is a python library ，有个python包，是可以的，github：<a href="https://github.com/graykode/modelsummary" rel="nofollow noopener">https://github.com/graykode/modelsummary</a></p>
Loss and accuracy curve

MasterofPLM: <p>What does this kind of curve represent? Under-fitting? Why only the training accuracy is weird?<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/6/6b7baa51120ade00c05fa7db6e7a2b734215d238.jpeg" alt="train" data-base62-sha1="fkQ2xGiFDdmEGw2dKkCV6vuaUYw" width="640" height="480"></p>

LeviViana: <p>It could under-fitting. But, since the model is training correctly, I guess these gaps show that the distributions of your train &amp; validation sets aren’t the same. For instance, say that you can easily identify two kinds of inputs: <strong>easy</strong> vs <strong>hard</strong> to predict. Imagine that you got the following distribution:</p>
<div class="md-table">
<table>
<thead>
<tr>
<th></th>
<th>Easy</th>
<th>Hard</th>
</tr>
</thead>
<tbody>
<tr>
<td>Train set</td>
<td>20%</td>
<td>80%</td>
</tr>
<tr>
<td>Validation set</td>
<td>50%</td>
<td>50%</td>
</tr>
</tbody>
</table>
</div><p>You would likely find the loss and accuracy curves like you pasted here. So, I’d recommend you to check that the distributions of train and validation sets are the same.</p>

Balamurali_M: <p><a class="mention" href="/u/leviviana">@LeviViana</a> is right. But that can be one possible explanation. More comments can be given if you please share about the dataset used, about the train/validation split and loss function used for training. Because in some cases train loss reduction doesn’t mean improvement in performance.</p>
Ever changing onnx, - roadplan questions

xvdp: <p>Hi,<br>
Im working on dynamic networks and I leverage onnx parser in the heuristics -translating to a networkx representation over which I operate.<br>
I had to rewrite it between torch 0.4 and 1.0 and I just ran in with v 1.1 and I found a couple breaking changes. port.isTensor() to port.isCompleteTensor() is one, im still digging for others.</p>
<p>I understand that this is a WIP and that the primary goal of this is probably network IO. I understand that Deep Nets themselves are a huge WIP so it is hard to predict what will change. Networkx gives me access to graph algorithms; it is itself a bit clunky but serviceable for now.  Dynamic networks and much larger multimodal networks are on the horizon.</p>
<p>Is there a plan to to make a complete stable object oriented graph parser? Itd be great to have.</p>
Can cuda unified memory be leveraged in tensors?

xvdp: <p>when declaring tensor in cpu, numpy conversions are by default references of the tensor.<br>
This can be useful in some cases to leverage stuff written in numpy.</p>
<p>when declaring tensor in cuda that isnt so. Is there any way of making this happen?<br>
or<br>
do i have to switch to caffe2 and if so, then are there any examples of this?</p>
<p>using pytorch 1.1. ubuntu 16</p>
Pretrain detector using vgg16-bn

solsol: <p>Hi there,</p>
<p>I want to  train detector using vgg16-bn as a backbone in <a href="https://github.com/rowanz/neural-motifs/blob/master/models/train_detector.py" rel="nofollow noopener">https://github.com/rowanz/neural-motifs/blob/master/models/train_detector.py</a><br>
however, I faced this error:</p>
<p>File models/…/lib/fpn/box_utils.py", line 44, in bbox_preds<br>
xys = prior_centers[:, :2] + prior_centers[:, 2:] * deltas[:, :2]<br>
File “/home/user/anaconda3/lib/python3.6/site-packages/torch/tensor.py”, line 321, in <strong>mul</strong><br>
return self.mul(other)<br>
RuntimeError: invalid argument 3: sizes do not match at /pytorch/torch/lib/THC/generated/…/generic/THCTensorMathPointwise.cu:351</p>
<p>Any help please?</p>
Please help me with this function backward calculation

wty: <p>Hello, I have a relative complex function, which I want use backward()</p>
<pre><code class="lang-auto">a = torch.zeros([3,3])
a[0,0] =1.0
a[1,2] =1.0
a[2,1] =1.0
b1 = torch.tensor([[1.0,2.0,3.0],[2.0,3.0,4.0],[3.0, 4.0,5.0]])
b2 = torch.tensor([[3.0,2.0,1.0],[4.0,3.0,2.0],[5.0, 4.0,3.0]])
c1 = torch.tensor([[1.0],[2.0],[3.0]])
c2 = torch.tensor([[4.0],[5.0],[6.0]])

c1 = Variable(c1, requires_grad=True)
c2 = Variable(c2, requires_grad=True)

b = c1.mul(b1) + c2.mul(b2)
b = Variable(b, requires_grad=True)
c = torch.div(a, b) - 1
c = Variable(c, requires_grad=True)
ref = torch.tensor([[1.0,0.0,0.0],[1.0,0.0,0.0],[1.0, 0.0,0.0]])
c.backward(ref, retain_graph=True)
d = b.grad
</code></pre>
<p>I want calculate the gradient of b and c1, but I alwazs got d = None. Please help me with this problem</p>
<p>Thank you</p>

Sunshine352: <p>Because that ‘replace’ operation<br>
【a = torch.zeros([3,3])<br>
a[0,0] =1.0<br>
a[1,2] =1.0<br>
a[2,1] =1.0】<br>
is not differentiable</p>

wty: <p>Hi, thank you for your reply.</p>
<p>I have changed code remove</p>
<pre><code class="lang-auto">c = Variable(c, requires_grad=True)
</code></pre>
<p>how I got right b.grad, but for c1.grad is still None.</p>

wty: <p>Thank you, the problem solved.</p>

Sunshine352: <p>Only leaf Tensors will have their <a href="https://pytorch.org/docs/stable/tensors.html?highlight=requires_grad#torch.Tensor.grad" rel="nofollow noopener"> <code>grad</code> </a> populated during a call to <a href="https://pytorch.org/docs/stable/tensors.html?highlight=requires_grad#torch.Tensor.backward" rel="nofollow noopener"> <code>backward()</code> </a>. To get <a href="https://pytorch.org/docs/stable/tensors.html?highlight=requires_grad#torch.Tensor.grad" rel="nofollow noopener"> <code>grad</code> </a> populated for non-leaf Tensors, you can use <a href="https://pytorch.org/docs/stable/tensors.html?highlight=requires_grad#torch.Tensor.retain_grad" rel="nofollow noopener"> <code>retain_grad()</code> </a>.</p>
How to load dataset from harddisk for cnn

chinnu: <p>How to load dataset from harddisk for cnn?</p>

ptrblck: <p>The <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">Data Loading Tutorial</a> might be a good starter. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Fix the randomness of one model

alan_ayu: <p>I want to fix the randomness of the training for compare the performance of different super-parameters. Now I fix the random seed of random.seed(), numpy.random.seed(), torch.manual_seed() and torch.cuda.manual_seed(), but I found I still can’t get the same result in different training process, even I set the p of dropout as 0. Now the result seems to be similar but there is still some small difference, especially the AUC value.<br>
I guess the SGD may still start from different random state but I can’t check well. And I still want to use dropout with non-zero p in the training but I don’t know how to keep it in the same random state.  Please give me some advice if you have idea about this question. Thank you !</p>

ptrblck: <p>Additionally to setting the seeds you should also disable <code>cudnn.benchmark</code> and enable the cudnn deterministic behavior, if you are using a GPU. Have a look at the <a href="https://pytorch.org/docs/stable/notes/randomness.html" rel="nofollow noopener">Reproducibility docs</a> for more information.</p>
<p>Also, how large are the differences? As explained in the docs, some atomic operations might yield non-determinisitc behavior.</p>

alan_ayu: <p>Thank you, it works. Now the results are strictly the same.</p>
Add new layers to the pretrained model

amin_sabet: <p>I’m using VGG16 pre-trained model as my base model. However, I need to add a new layer (quantizing layer) before every convolution layer in VGG 16. How can I add this new layer before conv layers if I want to keep the value of VGG16 parameters (weights/Biases)?</p>

ecdrid: <p>you need to write your own VGG16 then (better) and add the intermediate layers plus loading the weights for those specific original vgg layers as well?</p>

amin_sabet: <p>Thank you for your help.<br>
the name of layers parameters are changed in my VGG16 model. How can I load parameter from pretrained VGG16 model with different parameter names ?</p>
<p>Here is pretrained parameter names</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>features.0.weight</th>
<th>torch.Size([64, 3, 3, 3])</th>
</tr>
</thead>
<tbody>
<tr>
<td>features.0.bias            torch.Size([64])</td>
<td></td>
</tr>
<tr>
<td>features.2.weight        torch.Size([64, 64, 3, 3])</td>
<td></td>
</tr>
<tr>
<td>features.2.bias            torch.Size([64])</td>
<td></td>
</tr>
<tr>
<td>features.5.weight        torch.Size([128, 64, 3, 3])</td>
<td></td>
</tr>
<tr>
<td>features.5.bias            torch.Size([128])</td>
<td></td>
</tr>
<tr>
<td>features.7.weight        torch.Size([128, 128, 3, 3])</td>
<td></td>
</tr>
<tr>
<td>features.7.bias            torch.Size([128])</td>
<td></td>
</tr>
<tr>
<td>features.10.weight      torch.Size([256, 128, 3, 3])</td>
<td></td>
</tr>
</tbody>
</table>
</div><p>And Here my VGG16 parameter names - Step size is the name of parameter in my quantization layer</p>
<p>|features.0.step_size | torch.Size([1])|</p>
<p>|features.1.weight         torch.Size([64, 3, 3, 3])|<br>
|features.1.bias             torch.Size([64])|<br>
|features.3.step_size    torch.Size([1])|<br>
|features.4.weight         torch.Size([64, 64, 3, 3])|<br>
|features.4.bias             torch.Size([64])|<br>
|features.7.step_size    torch.Size([1])|<br>
|features.8.weight         torch.Size([128, 64, 3, 3])|<br>
|features.8.bias             torch.Size([128])|<br>
|features.10.step_size  torch.Size([1])|</p>

ecdrid: <p>It’s a kinda tedious task, but you have to map the layer’s weights now, like what was the later in the actual VGG16 and what’s in your own implementation, just access the .weight/.bias attribute and load the weights from the pretrained version to yours…</p>
Multiple classification problems with one net

Daniel_Strachan: <p>This is not multi-class classification! What I want to do, is create a predictor which simultaneously predicts 100 classification problems at once, each of which can have the labels 0,1,2. So the output size, for each batch, needs to be:</p>
<p>N x c x p</p>
<p>Where N is the batch size, c=100 is the number of classification problems and p=3 is the dimension of each classification problem. MultilabelMarginLoss only supports the N x p problem, where there is only one problem to be solved. In my case, each of the c classification problems are very similiar, so it makes sense to use the same net for each problem simultaneously.</p>
<p>How do I implement this?</p>