akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
[Solved] Visualizing Images Augmented by ToTensor()

username: datacrisis
cooked: <p>Hello! Thanks for dropping by.</p>
<p>I have a quick question about images that are processed by ToTensor() - I’ve simply swapped the dimensions from a channel leading configuration to the standard HxWxC configuration and visualize them with matplotlib.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/9/96bad5278ebb252ef5d09d156f7aaba1dc32927f.png" data-download-href="https://discuss.pytorch.org/uploads/default/96bad5278ebb252ef5d09d156f7aaba1dc32927f" title="aug1"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/9/96bad5278ebb252ef5d09d156f7aaba1dc32927f_2_690x324.png" alt="aug1" data-base62-sha1="lvpUgYmAQ5AmRNXuYnLqXAODIzJ" width="690" height="324" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/9/96bad5278ebb252ef5d09d156f7aaba1dc32927f_2_690x324.png, https://discuss.pytorch.org/uploads/default/optimized/2X/9/96bad5278ebb252ef5d09d156f7aaba1dc32927f_2_1035x486.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/9/96bad5278ebb252ef5d09d156f7aaba1dc32927f.png 2x" data-dominant-color="AEAEAE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">aug1</span><span class="informations">1366×643 214 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
The original image is simply a full scale image of a single grid; e.g. an image of a single car. The only other transformation applied before is normalization. I have isolated each transformation individually, and it seems that ToTensor() is the cause.</p>
<p>The outputs seem to be these 3x3 images, which are not I have expected. Is this the way its supposed to be and I have simply missed out something about the nature of the transformation, or have I gotten something catastrophically wrong?</p>
<p>Thank you for reading <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=12" title=":grinning:" class="emoji" alt=":grinning:" loading="lazy" width="20" height="20"></p>

username: ptrblck
cooked: <p>Are you using <code>reshape</code> or <code>view</code> to swap the axes?<br>
If so, try to use <code>permute</code>, since <code>view/reshape</code> might yield these interleaved outputs.</p>

username: datacrisis
cooked: <p>Ah yes, problem solved, I was using view indeed.</p>
<p>Thank you very much <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Transition from Training to Testing from PyTorch Examples

username: CRWayman
cooked: <p>Hi all,</p>
<p>I’m looking at the Learning PyTorch with Examples page (see example code below).</p>
<p><a href="https://pytorch.org/tutorials/beginner/pytorch_with_examples.html#pytorch-custom-nn-modules" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/pytorch_with_examples.html#pytorch-custom-nn-modules</a></p>
<p>I’m a little confused about where to go from here in terms of testing my model now. It is unclear to me how I apply my new model/linear relationship to “forecasting” to hindcasting on my data.</p>
<p>I also am not sure how to extract the relevant weights for each input layer.</p>
<p>Any help would be appreciated. Thanks!</p>
<p>Callum</p>
<pre><code class="lang-auto">class TwoLayerNet(torch.nn.Module):
    def __init__(self, D_in, H, D_out):
        """
        In the constructor we instantiate two nn.Linear modules and assign them as
        member variables.
        """
        super(TwoLayerNet, self).__init__()
        self.linear1 = torch.nn.Linear(D_in, H)
        self.linear2 = torch.nn.Linear(H, D_out)

    def forward(self, x):
        """
        In the forward function we accept a Tensor of input data and we must return
        a Tensor of output data. We can use Modules defined in the constructor as
        well as arbitrary operators on Tensors.
        """
        h_relu = self.linear1(x).clamp(min=0)
        y_pred = self.linear2(h_relu)
        return y_pred
</code></pre>
Fluctuating validation accuracy

username: Buse
cooked: <p>Hello all,</p>
<p>I am having problems with my validation accuracy and loss.<br>
Although my train set keep getting higher accuracy through the epochs my validation accuracy is unstable. I am using pretrained DenseNet-121 model for my binary classification with train set containing 50000 images and 10000 images for validation.</p>
<p><a href="https://github.com/basveeling/pcam" rel="noopener nofollow ugc">Complete dataset</a></p>
<p>I have only changed output number of classifier of the model.</p>
<pre><code class="lang-auto">criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.003)
batch_size = 16
num_epochs = 16
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/d/d831c62e75f2142332349d50c7f9d844a7020c6f.png" alt="indir%20(1)" data-base62-sha1="uQxM09CoukENBm6bcSSfAdZvVsb" width="389" height="278"></p>
<p>I couldn’t figure out the problem. Am I overfitting here or is there something wrong with my epoch number or batch size? O should I look into more tuning parameters of optimizer etc.?</p>
<p>Thanks! <img src="https://discuss.pytorch.org/images/emoji/apple/relaxed.png?v=12" title=":relaxed:" class="emoji" alt=":relaxed:" loading="lazy" width="20" height="20"></p>
Getting trouble while converting .model to .onnx

username: nurettinabaci
cooked: <p>Hello, I have modified AlexNet model trained with PyTorch. The trained file has .model extension file. I want to convert it to ONNX model and export to c++.<br>
I tried the basic convertion code but  it gives error:</p>
<pre><code class="lang-auto">import torch.onnx

device = torch.device('cpu')
model = torch.load('SiamRPN.model', map_location=device)
dummy_input = torch.randn(1, 3, 224, 224)

torch.onnx.export(model, dummy_input,"my_onnx_model.onnx")
</code></pre>
<p>When I run the last line with Python 2, it gives error like below. What is the problem here?</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "torch_to_onnx.py", line 7, in &lt;module&gt;
    torch.onnx.export(model, dummy_input,"my_onnx_model.onnx")
  File "/usr/local/lib/python2.7/dist-packages/torch/onnx/__init__.py", line 32, in export
    return utils.export(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/torch/onnx/utils.py", line 170, in export
    example_outputs=example_outputs, strip_doc_string=strip_doc_string, dynamic_axes=dynamic_axes)
  File "/usr/local/lib/python2.7/dist-packages/torch/onnx/utils.py", line 423, in _export
    _retain_param_name, do_constant_folding)
  File "/usr/local/lib/python2.7/dist-packages/torch/onnx/utils.py", line 317, in _model_to_graph
    graph, torch_out = _trace_and_get_graph_from_model(model, args, training)
  File "/usr/local/lib/python2.7/dist-packages/torch/onnx/utils.py", line 267, in _trace_and_get_graph_from_model
    orig_state_dict_keys = _unique_state_dict(model).keys()
  File "/usr/local/lib/python2.7/dist-packages/torch/jit/__init__.py", line 244, in _unique_state_dict
    state_dict = module.state_dict(keep_vars=True)
AttributeError: 'OrderedDict' object has no attribute 'state_dict'

</code></pre>
Loading pretrained weights for updating some layers in the model

username: Hasan_Sait_ARSLAN1
cooked: <p>Hi,</p>
<p>I have trained a model, and at the moment I would like to use that model as a layer in my new model.</p>
<p>How it would be possible to initialize my layer with the pretrained model I did before?</p>
<p>Thanks,</p>

username: JuanFMontesinos
cooked: <pre><code class="lang-auto">class newmodel(nn.Module):
    __init__():
    super(...)
    self.model_as_layer = model_constructor(*args,**kwargs)
    self.model_as_layer.load_state_dict(pretrained_weights)
</code></pre>
<p>Just loads weighs while you instantiate your layer</p>
Can't load properly a saved model

username: Vincent_Brisse
cooked: <p>Reading this bear in mind that I’m a beginner with pytorch. I think saving the model went wine, I did it using :</p>
<pre><code class="lang-auto">	torch.save({
                'epoch': self.epochs,
                'model_state_dict': self.model.state_dict(),
                'optimizer_state_dict': self.optimizer.state_dict()
                }, 'sav
</code></pre>
<p>inside the for loop on epochs.</p>
<p>Then, to load the model, I do this after initializing the model and the optimizer.</p>
<pre><code class="lang-auto">    if os.path.exists('saved') :
        print('Loading saved model')
        checkpoint = torch.load('saved')
        model.load_state_dict(checkpoint['model_state_dict'])
        optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
        epoch = checkpoint['epoch']
</code></pre>
<p>Here’s the error :</p>
<pre><code class="lang-auto">Missing key(s) in state_dict: "frame_level_rnns.0.h0", "frame_level_rnns.0.input_expand.bias", "frame_level_rnns.0.input_expand.weight_g", "frame_level_rnns.0.input_expand.weight_v", "frame_level_rnns.0.rnn.weight_ih_l0", "frame_level_rnns.0.rnn.weight_hh_l0", "frame_level_rnns.0.rnn.bias_ih_l0", "frame_level_rnns.0.rnn.bias_hh_l0", "frame_level_rnns.0.rnn.weight_ih_l1", "frame_level_rnns.0.rnn.weight_hh_l1", "frame_level_rnns.0.rnn.bias_ih_l1", "frame_level_rnns.0.rnn.bias_hh_l1", "frame_level_rnns.0.upsampling.bias", "frame_level_rnns.0.upsampling.conv_t.weight_g", "frame_level_rnns.0.upsampling.conv_t.weight_v"
</code></pre>
<p>and then shortly after</p>
<pre><code class="lang-auto">	Unexpected key(s) in state_dict: "model.frame_level_rnns.0.h0", "model.frame_level_rnns.0.input_expand.bias", "model.frame_level_rnns.0.input_expand.weight_g", "model.frame_level_rnns.0.input_expand.weight_v", "model.frame_level_rnns.0.rnn.weight_ih_l0", "model.frame_level_rnns.0.rnn.weight_hh_l0", "model.frame_level_rnns.0.rnn.bias_ih_l0", "model.frame_level_rnns.0.rnn.bias_hh_l0", "model.frame_level_rnns.0.rnn.weight_ih_l1", "model.frame_level_rnns.0.rnn.weight_hh_l1", "model.frame_level_rnns.0.rnn.bias_ih_l1", "model.frame_level_rnns.0.rnn.bias_hh_l1", "model.frame_level_rnns.0.upsampling.bias"
</code></pre>
<p>So instead of loading ‘a’ it loads ‘model.a’ I guess, but I don’t get what I am doing wrong. Can you help me out please ?</p>

username: Vincent_Brisse
cooked: <p>Also, I’d like to add that the optimizer loads just fine (when putting the line about the optimizer just before the line for the model) and also the epochs.</p>

username: Vincent_Brisse
cooked: <p>So I hacked it, basically removing the “model.” part just before key. Still I’m not tagging it as solved because I wanna know why I had this error and also find a more elegant way to deal with it !</p>

username: ptrblck
cooked: <p>I guess the <code>model</code> attribute is added, as it seems you are using another class to store your <code>self.model</code>, <code>self.optimizer</code> and probably other objects.<br>
Are you working with some high-level API or did you create this wrapper manually?</p>
Question on extracting intermediate features from pretrained models

username: hij
cooked: <p>I am trying to get intermediate features from a pretrained resnet. As per this post <a href="https://discuss.pytorch.org/t/how-to-extract-features-of-an-image-from-a-trained-model/119/3">https://discuss.pytorch.org/t/how-to-extract-features-of-an-image-from-a-trained-model/119/3</a> we can just  do something like this</p>
<pre><code class="lang-auto">new_classifier = nn.Sequential(*list(model.classifier.children())[:-1])
</code></pre>
<p>Looking at the code for pretrained resnet50, it is broken up into several blocks. The forward function then take these blocks and apply batchnorm, relu, maxpool, reshape, etc to them.</p>
<pre><code class="lang-auto">    def forward(self, x):
        x = self.conv1(x)
        x = self.bn1(x)
        x = self.relu(x)
        x = self.maxpool(x)

        x = self.layer1(x)
        x = self.layer2(x)
        x = self.layer3(x)
        x = self.layer4(x)

        x = self.avgpool(x)
        x = x.reshape(x.size(0), -1)
        x = self.fc(x)
</code></pre>
<p>So if the pretrained model is not just one large nn.Sequential module, does the above way of extracting features work? Will the new classifier apply the relu,maxpool,reshape,etc in this manner?</p>

username: ptrblck
cooked: <p>It generally depends on the original implementation, i.e. if the base model was implemented using <code>nn.Sequential</code> (or in a very similar manner), this approach might work.<br>
However, if some functional calls were used in the <code>forward</code> method, you will lose them.</p>
<p>In the example of <code>resnet50</code>, this approach will work, since even the non-linearity is defined as a module.</p>
How to access ctx objects directly?

username: vargonis
cooked: <p>Hi there. I’d like to keep track of some statistical properties of activations in a network, in an output-dependent fashion. I do not see how I could use a forward hook, because the output of intermediate layers is not given as an argument to such hooks. Of course, I could also register output-saving forward hooks for each layer, but my understanding is that I would then be replicating context-saving functionality that is already implemented for backward passes. How can I access that functionality for my custom needs?</p>
Different learning rate for different class

username: se7en
cooked: <p>How to set adaptive learning rate for different class in multi-class classification while training? Like, when some classes’ loss get the threshold, then set their learning rate * 0.1, wile other classes keep their learning rate to not to change.</p>

username: JuanFMontesinos
cooked: <p>What do you exactly want to do? LR is inherent to model parameters but it seems you want to modify it depending on your output. A mini-batch may contain different classes so could you in-depth explain it?</p>
<p>You can set a threshold for a class and modify learning rate the mini-batch will be backpropagated with before backpropagating but it still does not solves the fact the batch will contain different classes and they will be backpropagated with the same LR.</p>
<p>You may consider to penalize loss depending on this th, this way gradients will be penalized too.</p>
Modify output forward hooks

username: JuanFMontesinos
cooked: <p>Hi,<br>
According to the V1.1.0 docs it’s possible to modify a module output using forward hooks.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/1/12c0264ae97d807d6d913bb9a7d1f0e2ddc1a414.png" data-download-href="https://discuss.pytorch.org/uploads/default/12c0264ae97d807d6d913bb9a7d1f0e2ddc1a414" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/1/12c0264ae97d807d6d913bb9a7d1f0e2ddc1a414.png" alt="image" data-base62-sha1="2FSgVAsAXRH2jhQEw2YFo6YwoxS" width="690" height="435" data-dominant-color="F5F5F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">793×501 26.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>However sourcecode is not consistent:</p>
<pre><code class="lang-auto">
    def __call__(self, *input, **kwargs):
        for hook in self._forward_pre_hooks.values():
            hook(self, input)
        if torch._C._get_tracing_state():
            result = self._slow_forward(*input, **kwargs)
        else:
            result = self.forward(*input, **kwargs)
        for hook in self._forward_hooks.values():
            hook_result = hook(self, input, result)
            if hook_result is not None:
                raise RuntimeError(
                    "forward hooks should never return any values, but '{}'"
                    "didn't return None".format(hook))
</code></pre>
<p>As you can see if <code>hook_result</code> is not <code>None</code>, it throws an error.<br>
What did I miss?</p>
Converted ONNX model doesn't pass check model

username: anton147
cooked: <p>Hi everyone!<br>
I have trained pytorch SSD512 model on custom dataset using pre-trained VGG16 as featurizer. Now I want to convert it to ONNX format to be able to make predictions using <a href="http://ML.NET" rel="nofollow noopener">ML.NET</a>. Model is converted without errors, nevertheless it doesn’t pass onnx.checker.check_model.</p>
<p>I use macOS 10.13.6, pytorch 1.1.0, onnx 1.5.0<br>
My model was trained on GPU server.</p>
<p>Here is my code to convert:</p>
<pre><code class="lang-auto">from ssd.config import cfg
from ssd.modeling.detector import build_detection_model
import torch
from torch import nn

if __name__ == '__main__':
    cfg.merge_from_file('configs/vgg_ssd512.yaml')
    model = build_detection_model(cfg)
    state_dict = torch.load('model_final.pth', map_location=lambda storage, loc: storage)['model']
    model.load_state_dict(state_dict)
    dummy_input = torch.randn(1, 3, 512, 512, device='cpu')
    torch.onnx.export(model, dummy_input, 'ssdpytorch.onnx', opset_version=10, input_names=['image'], output_names=['scores', 'boxes'], verbose=True)
</code></pre>
<p>And the following error occurs:</p>
<pre><code class="lang-auto">ValidationError                           Traceback (most recent call last)
&lt;ipython-input-30-b6beadd98bdc&gt; in &lt;module&gt;()
----&gt; 1 onnx.checker.check_model(model)

//anaconda/lib/python3.6/site-packages/onnx/checker.py in check_model(model)
     84         C.check_model_path(model)
     85     else:
---&gt; 86         C.check_model(model.SerializeToString())
     87 
     88 

ValidationError: Node () has input size 1 not in range [min=3, max=5].

==&gt; Context: Bad node spec: input: "306" output: "308" op_type: "Slice" attribute { name: "axes" ints: 2 type: INTS } attribute { name: "ends" ints: 2 type: INTS } attribute { name: "starts" ints: 0 type: INTS }
</code></pre>
<p>Here is onnx model: <a href="https://gofile.io/?c=wGr587" rel="nofollow noopener">https://gofile.io/?c=wGr587</a></p>
About cpu training speed

username: Yang_Yang2
cooked: <p>I have tried building pytorch from source. I print torch.<strong>config</strong>.show() and the ‘MKLDNN’ is on. For my model, training one epoch needs 5min.</p>
<p>However, I also tried conda install pytorch 1.1.   I also print torch.<strong>config</strong>.show() and  the ‘MKLDNN’ is also on. But for the same model, training one epoch needs 8min.</p>
<p>How can I speed up training on CPU when I conda install pytorch?</p>
How to make nn.Embedding support one-hot encoded vector input?

username: jangbi
cooked: <p>From the documentation, i learn that <code>nn.Embedding</code> takes “LongTensor of arbitrary shape containing the indices to extract” as input.<br>
But let’s say i have a data field named movie_genre for each sample  <code>movie</code> , it is selected from the following genres:</p>
<pre><code class="lang-auto">Action
Adventure
Animation
Comedy
...
</code></pre>
<p>And for each  <code>movie</code> , it might contain multiple genres:</p>
<pre><code class="lang-auto">mid    genres
1      Action | Adventure
2      Animation
3      Comedy | Adventure | Action
</code></pre>
<p>If i use one hot vector to encode the  <code>genre</code> , Action can be encoded as (1, 0, 0, 0), Adventure can be encoded as(0, 1, 0, 0), and so on.</p>
<p>So movie with mid1 can be encoded as (1, 1, 0, 0), mid2’s genre can be encoded as (0, 0, 1, 0), and so on.</p>
<p>And how should i do to make <code>nn.Embedding</code> support such kind of input?</p>
Check if variable is part of graph

username: bananacode
cooked: <p>I’m trying to debug my code. I predict two variables A, B and compute losses L1(A) and L2(B) on them. However, the way the network currently behaves is as if L2 is affecting A, even though it shouldn’t according to my code. I want to make absolutely sure that during backpropagation, A is not affected by L2. I thought one easy way of doing this is to check if A is part of the graph which produces L2.</p>
<p>How would one proceed to check this?</p>

username: bananacode
cooked: <p>I’m still having this problem. Can anyone help?</p>
CUDA out of memory after loading model

username: Vincent_Brisse
cooked: <p>So I know my GPU is close to be out of memory with this training, and that’s why I only use a batch size of two and it seems to work alright.<br>
The problem arises when I first load the existing model using torch.load, and then resume training. When resuming training, it instantly says :</p>
<pre><code class="lang-auto">RuntimeError: CUDA out of memory. Tried to allocate 64.00 MiB (GPU 0; 1.96 GiB total capacity; 1.36 GiB already allocated; 46.75 MiB free; 38.25 MiB cached)
</code></pre>
<p>I don’t know how to get rid of this error. When using nvidia-smi right after the error, I obtain this :</p>
<pre><code class="lang-auto">+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0       991      G   /usr/lib/xorg/Xorg                           120MiB |
|    0      1350      G   cinnamon                                      29MiB |
|    0      2878      G   ...incent/anaconda3/envs/newtor/bin/python     2MiB |
+-----------------------------------------------------------------------------+

</code></pre>
<p>So it’s definitely something about loading the model that makes it break. Can you help me out ? Maybe hack it and reset Cuda memory usage after loading the model ? Thanks for helping me out.</p>

username: Vincent_Brisse
cooked: <pre><code class="lang-auto">del checkpoint
</code></pre>
<p>was the solution, though once again i’m not sure why. Cuda magic I guess</p>
Training a network with multiple heads

username: raghav18
cooked: <p>I’m trying to train a network with multiple heads branching from a common body. I wanted to understand what these two following pieces of code are really doing and how they are different.</p>
<pre><code class="lang-auto">o1, o2 = model(input) #outputs

l1 = loss_fn(o1, labels)
l1.backward(retain_graph=True)
l2 = loss_fn(o2, labels)
l2.backward(retain_graph=True)

optimizer.step()
</code></pre>
<p>and this :</p>
<pre><code class="lang-auto">o1, o2 = model(input) #outputs

l1 = loss_fn(o1, labels)
l2 = loss_fn(o2, labels)

torch.autograd.backward([l1, l2])

optimizer.step()
</code></pre>
Torch.nn.functional.softmax

username: Megh_Bhalerao
cooked: <p>Hi,<br>
I read the documentation of torch.nn.functional.softmax, but I am not very clear upon the usage of the <code>dim</code> argument.<br>
My output is of the following dimensions :<br>
<code>(batchsize, num_classes, length, breadth, height)</code> - so it is a 5D output<br>
I am doing a multi class semantic segmentation problem,  so I want the pixel-wise softmax of corresponding pixels in <code>(length, breadth, height)</code> along all classes. So do I use <code>dim = 1</code> while using <code>torch.nn.functional.softmax(output, dim = 1)</code> , this way?<br>
Please advise.<br>
Thanks a lot!</p>

username: ptrblck
cooked: <p><code>torch.nn.functional.softmax(output, dim = 1)</code> will yield an output, where the probabilities for each pixel sum to 1 in the class dimension (dim1).<br>
This probability tensor can be used as a sanity check or for visualization purposes.</p>
<p>However, note that e.g. <code>nn.CrossEntropyLoss</code> expects raw logits as the model’s output, since internally <code>F.log_softmax(output, 1)</code> and <code>nn.NLLLoss</code> will be used, so that you shouldn’t use the softmax method manually.</p>

username: Megh_Bhalerao
cooked: <p>Alright, thanks a lot!</p>
Adding MLP Into the intermediate layer of Conv2D

username: datlombardo
cooked: <p>Hi PyTorchers,<br>
I am currently attempting to implement a network similar to Network in Network (2014; M.Lin Q.Chen). Which takes the output of each particular sliding window of a 2D Convolutional Layer and passes it through multiple linear layers prior to the pooling down sampling. Which looks like the following:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/f/f4ef0d812b446c1fd0584e2fdfc75916c2b4131c.png" alt="NIN" data-base62-sha1="yWMFB7Xcmhr2BAVlGZ1bBh5T77m" width="403" height="146"> .<br>
I am curious if there is a way to manually extract all of the features from each window as the Conv2D creates feature maps, so I can add my linear MLP layers prior to the down sampling.</p>
Difference Between broadcast_coalesced and dist_broadcast_coalesced

username: kipsora
cooked: <p>I want to have experiments on my own strategy of distributed data parallel and I have read through the codes in <code>torch.distributed</code>. But I am confused by the following two C++ function. It seems that these two methods are doing the same job, namely broadcasting the tensors from a tensor from one gpu to the others.</p>
<p>My PyTorch version is <code>v1.0.0</code>.</p>
<ol>
<li>
<code>broadcast_coalesced</code> in <code>torch/csrc/cuda/comm.cpp</code>
</li>
<li>
<code>distBroadcastCoalesced</code> in <code>torch/csrc/csrc/distributed/c10d/ddp.cpp</code>
</li>
</ol>
<p>I am wondering the difference of usages between these two methods. Thanks!</p>

username: ngimel
cooked: <p><a href="https://pytorch.org/docs/stable/cuda.html?highlight=broadcast_coalesced#torch.cuda.comm.broadcast_coalesced" rel="nofollow noopener">broadcast_coalesced</a> is used in a single process situation, when one process controls multiple gpus. distBroadcastCoalesced is used when there are multiple processes, and each process makes this call.</p>

username: pietern
cooked: <p>FWIW, the function in <code>ddp.cpp</code> should be considered a private API, only to be used by <code>DistributedDataParallel</code>. For the upcoming release, it was removed and replaced with a function also called <code>broadcast_coalesced</code> in the <code>c10d</code> namespace. See <a href="https://github.com/pytorch/pytorch/blob/caa0d0c/torch/csrc/distributed/c10d/comm.h" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/caa0d0c/torch/csrc/distributed/c10d/comm.h</a>. See <a href="https://github.com/pytorch/pytorch/pull/20234" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/20234</a> for the PR that added it.</p>
Found Reason for Cuda out of Memory, but not a solution

username: N3m0
cooked: <p>Hello everyone,<br>
im currently working with a LSTM to predict rest lifetime of mechanical parts. I got a problem when i try to use the trained network to predict multiple timesteps ahead. I noticed that everytime i use the <strong>predicted output</strong> as <strong>input</strong>, i get this error message after a few timesteps(mostly between 50-200 timesteps):</p>
<p>cuda out of memory. tried to allocate 2.50 mib (gpu 0; 8.00 gib total capacity; 6.28 gib already allocated; 1.55 mib free; 1.55 mib cached</p>
<p>This is the code:<br>
inp, label = Validset[1]<br>
print(inp.shape)<br>
y = []<br>
for i in range(100):<br>
print(i)<br>
torch.no_grad()<br>
inp = torch.tensor(inp)<br>
inp = inp.view(1,-1,1)<br>
inp = inp.to(device)<br>
inp = inp.float()<br>
out, hn_cn = model.forward(inp)<br>
del inp<br>
out = out.reshape(-1)<br>
y.append(out[-1])<br>
inp = out.clone().detach()<br>
del out<br>
torch.cuda.empty_cache()</p>
<p>I used a gtx1070 and a gtx1080 sli for this task but it seems like both setups fail to predict enough timesteps.</p>
<p>I noticed this problem also while training a LSTM  There i tried to use a output Tensor (1x1) as an Input aswell and i get the same error message. Is there a way to free memory after i predicted one timestep? Or has anyone a different solution?</p>
<p>Thank you in advance</p>
Prospects for running pytorch / cuda on a laptop with an external gpu?

username: KFrank
cooked: <p>Hello Forum!</p>
<p>Are things likely to go smoothly if I try to run pytorch on an external<br>
gpu on a laptop?  (Assume recent mainstream hardware.)  Or are<br>
things likely to be finicky or not work?</p>
<p>Would I expect to be able to download a pre-built version of<br>
pytorch?  I imagine that matching my gpu, os, and cuda version<br>
should be enough, and that pytorch will neither know nor care<br>
that the gpu is external.  Is this right?</p>
<p>As I understand it, external gpus are typically connected through<br>
a thunderbolt port.  Would I expect to get (nearly) full gpu<br>
performance, or would the thunderbolt port introduce some kind<br>
of bottleneck?</p>
<p>And, of course, if anyone has actually done this, what hardware<br>
and os did you have success with, and did you encounter any<br>
noteworthy problems?</p>
<p>Thanks for any advice!</p>
<p>K. Frank</p>
Loading GAN generator

username: sami
cooked: <p>Hey everyone,</p>
<p>I’ve looked into this quite some time now, and I haven’t found any explanation, yet. I’ve trained a few GANs with different hyperparameters and saved the checkpoints every epoch. Now, some of the runs I’ve done have checkpoints I can load and generate data without a problem, and some just generate rubbish, as if the generator has not been trained at all or collapsed at some point.<br>
I have changed the network structure at one point, doubling the number of filters in the first layer of the generator, but also reverted to that network structure when I loaded the checkpoints. I also don’t any error messages when loading the state dict, which should mean that everything is loaded correctly.</p>
<p>I have trained the network using Pytorch 0.4.1 and try to evaluate on Pytorch 0.4.0, but deleted the num_batches_tracked in the batch norm layers accordingly. It also works on some checkpoints.</p>
<p>I’m utterly confused on why that would happen, has anyone experienced something of that kind?</p>
<p>I load my generator checkpoint like this:</p>
<pre><code class="lang-auto">    generator = dcgm.GeneratorNe(nc, ngf=128, ngpu=ngpu)
    dict = torch.load(opt.loadG, map_location='cuda:0' if torch.cuda.is_available() else 'cpu')
    if torch.__version__ == '0.4.0':
        del dict['net.1.num_batches_tracked']
        del dict['net.4.num_batches_tracked']
        del dict['net.7.num_batches_tracked']
        del dict['net.10.num_batches_tracked']
        del dict['net.13.num_batches_tracked']
    generator.load_state_dict(dict)
    generator.to(gpu)
    generator.eval()
</code></pre>

username: shania3322
cooked: <p>Hi sami,</p>
<p>I have the same issue. I loaded weights file and generated image from weights. It seemed that using random noise not really learnt weights. Have you figured out why is that? I haven’t successfully generated one image for now.</p>
<p>I’m using torch 0.4.1.</p>
<p>Weights file is saved via torch.save(generator.state_dict(), path) during training phase.<br>
During testing phase, I did:</p>
<p>model = generator()<br>
checkpoint = torch.load(‘path/001_G.pth’, map_location = str(device))<br>
model.load_state_dict(checkpoint, strict=False)<br>
model.to(device)<br>
model.float()<br>
model.eval()</p>
<p>def label_sampel():<br>
label = torch.LongTensor(batch_size, 1).random_()%n_class<br>
one_hot= torch.zeros(batch_size, n_class).scatter_(1, label, 1)<br>
print(device)<br>
return label.squeeze(1).to(device), one_hot.to(device)</p>
<p>z = torch.randn(batch_size, z_dim).to(device)<br>
z_class, z_class_one_hot = label_sampel()</p>
<p>fake_images = model(z, z_class_one_hot)<br>
save_image(denorm(fake_images.data), os.path.join(path, ‘1_generated.png’))</p>
Inconsistent behaviour in tensor division

username: StatisticDean
cooked: <p>In pytorch, if both tensors are long, an integer division will be performed :</p>
<pre><code class="lang-auto">tensor([242240, 226320, 186240, 171840, 165680]) / torch.tensor(694)
Out : tensor([349, 326, 268, 247, 238])
</code></pre>
<p>I need a regular division so I tried to convert my second tensor to float, it yields the same result :</p>
<pre><code class="lang-auto">tensor([242240, 226320, 186240, 171840, 165680]) / torch.tensor(694).float()
Out : tensor([349, 326, 268, 247, 238])
</code></pre>
<p>However, if the first tensor is of size 1, this works :</p>
<pre><code class="lang-auto">torch.tensor(10) / torch.tensor(10).float()
Out : tensor(1.0)
</code></pre>
<p>This seems really weird. Any thoughts?</p>

username: JuanFMontesinos
cooked: <p>It’s not the same doing</p>
<pre><code class="lang-auto">import torch

a=torch.tensor(10) / torch.tensor(10).float()
b=torch.tensor([10]) / torch.tensor(10).float()
print(a, 'Dimensions: %s'%a.ndimension(),' Type %s'%a.type())
print(b,'Dimensions: %s'%b.ndimension(),' Type %s'%b.type())
</code></pre>
<pre><code class="lang-auto">/usr/bin/python3.6 /home/jfm/.PyCharm2019.1/config/scratches/scratch.py
tensor(1.) Dimensions: 0  Type torch.FloatTensor
tensor([1]) Dimensions: 1  Type torch.LongTensor
</code></pre>
<p>That’s why the result is different</p>

username: StatisticDean
cooked: <p>I agree with you that the number of dimensions is different. That doesn’t explain why the type is different though.</p>

username: JuanFMontesinos
cooked: <p>The constructor generates different data types depending on if the input is numerical or an array. I edited my previous reply.</p>
<p>The short answer is that these kind of operations depends on the dimensionality, it does not threat 0-dimensional tensors than 1-dimensional tensors of 1 element.</p>
Tensor: Is there a way to repeat each matrix element into diagonal submatrix?

username: zfzhang
cooked: <p>Hi,<br>
Suppose I have a matrix tensor</p>
<pre><code class="lang-auto">a  b
c  d
</code></pre>
<p>and want to repeat it into</p>
<pre><code class="lang-auto">a 0 b 0
0 a 0 b
c 0 d 0
0 c 0 d
</code></pre>
<p>In numpy code, it is <code>y = np.kron(x,np.eye(2))</code>, I want to know pytorch equivalent.<br>
My actual usage is higher than 2D, which has additional dimensions at the tail, though.<br>
Thank you very much!</p>

username: tom
cooked: <p>I don’t think it’s ready-mades, but starting with zeros and copying x into <code>y[::2, ::2]</code> and <code>y[1::2, 1::2]</code> should do the trick.<br>
If your x isn’t tiny and the “stencil matrix” is reasonably small, a <code>for</code> foop over the stencil isn’t a performance headache.</p>
<p>Best regards</p>
<p>Thomas</p>

username: zfzhang
cooked: <p>Hi Tom,<br>
I’m operating on module parameters, will this copying &amp; looping approach provide autograd support?<br>
And how to use broadcasting semantics in the indexed assignment?</p>
<p>Thanks!</p>

username: tom
cooked: <ol>
<li>Yes</li>
<li>How would broadcasting look like?</li>
</ol>
<p>P.S.: Please don’t ask the same question twice.</p>

username: zfzhang
cooked: <p>Sorry, I mis-interpreted the y[::2, ::2] syntax.</p>
Option to force device specification for leaf tensors?

username: atiyo
cooked: <p>I had a quick question about best practices for device agnostic coding. Some context: I prototype my code on my laptop (CPU only), before training in the cloud.</p>
<p>Right now, I follow the following pattern.</p>
<blockquote>
<p>device = torch.device(“cuda:0” if torch.cuda.is_available() else “cpu”<br>
model.to(device)</p>
</blockquote>
<p>for everything. This works quite nicely, and I’m happy that this functionality has been added to PyTorch.</p>
<p>But there are situations like the following where everything will work well on my local CPU environment, then fail in a GPU environment.</p>
<blockquote>
<p>a = torch.tensor([1]).to(device)<br>
b = torch.tensor([1])<br>
c = a+b</p>
</blockquote>
<p>which is entirely correct, and the behaviour I would expect.</p>
<p>What I would like to be able to impose restrictions on myself which ensure that errors like that above would fail both in my local CPU environment and in the GPU environment where I do my training.</p>
<p>Are there any existing solutions to this? If not, I think what I would like is an option I could enable which would <em>force</em> me to specify the device on which all leaf tensors live. Does such an option exist? Am I the only person who would be interested in this?</p>

username: tom
cooked: <aside class="quote no-group" data-username="atiyo" data-post="1" data-topic="50589">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/atiyo/48/3775_2.png" class="avatar"> atiyo:</div>
<blockquote>
<p>Are there any existing solutions to this? If not, I think what I would like is an option I could enable which would <em>force</em> me to specify the device on which all leaf tensors live. Does such an option exist? Am I the only person who would be interested in this?</p>
</blockquote>
</aside>
<p>The canonical thing is to test in an environment that resembles the target better.</p>
<p>There are <a href="https://github.com/pytorch/pytorch/issues/7535" rel="noopener nofollow ugc">19 other people</a> who want this. However, there are reservations to the feature request because it would replace an obvious way to get things wrong with a more subtle one.</p>
<p>Best regards</p>
<p>Thomas</p>

username: atiyo
cooked: <p>Thanks for the reply. I agree that setting the default to the GPU could be problematic, so I was careful in my original post not to suggest this.</p>
<p>I posted what’s below already in your linked GitHub issue, but posting it here too for the sake of continuity of this thread:</p>
<p>What about things like  <code>torch.set_default_device()</code>  and  <code>torch.get_default_device()</code>, where the default device type could initially be the CPU?</p>
<p>There is already similar functionality for dtypes with  <code>torch.set_default_tensor_type()</code>  and  <code>torch.get_default_dtype()</code> .</p>
The problem of out of memory

username: Farry_zhou
cooked: <p>Please for help.</p>
<p>I trained the model, which was constructed in Pytorch.<br>
But there is a error. It shows in the figure.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/f/f031f4e862c67c04ca07ced6fd35235eae33b832.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/f031f4e862c67c04ca07ced6fd35235eae33b832" title="problem"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/f/f031f4e862c67c04ca07ced6fd35235eae33b832_2_690x303.jpeg" alt="problem" data-base62-sha1="ygRCIThHlE7EhYEpx6zyLGQH1nQ" width="690" height="303" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/f/f031f4e862c67c04ca07ced6fd35235eae33b832_2_690x303.jpeg, https://discuss.pytorch.org/uploads/default/original/2X/f/f031f4e862c67c04ca07ced6fd35235eae33b832.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/f/f031f4e862c67c04ca07ced6fd35235eae33b832.jpeg 2x" data-dominant-color="1A1A1A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">problem</span><span class="informations">1027×451 63.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The environment is:<br>
Ubuntu 16.04<br>
Python 3.6<br>
Pytorch 1.1.0</p>
<p>My GPU is GeForce GTX 1080.</p>

username: caonv
cooked: <p>The first thing you should do is set the mini-batch size to a smaller number, e.g a half of your current size.<br>
This error occurs when your network need more memory than the available amount in your system.<br>
Hope this help!</p>

username: Farry_zhou
cooked: <p>Thank you for your suggestion<br>
I will test it</p>
Slicing/selecting tensors from a dataset?

username: ajhanwar
cooked: <p>Hi all, I’m working with the MNIST dataset available through <code>torchvision</code> and am trying to visualize the data (average digit, overlaid digits, etc.). While I am able extract/transform/load the dataset I can’t seem to find a way to extract the data itself to perform operations such as slicing and subsetting.</p>
<p>Any clarification would be greatly appreciated!</p>

username: ptrblck
cooked: <p>You could access the underlying <code>.data</code> and <code>.target</code> attributes:</p>
<pre><code class="lang-python">import torchvision.transforms.functional as TF

dataset = datasets.MNIST(root='./data',
                         transform=transforms.ToTensor())
data = dataset.data
print(data.shape) # torch.Size([60000, 28, 28])
img = TF.to_pil_image(data[0])
</code></pre>
Values with the same index in a sparse tensor, how to only remain the max value when convert the sparse tensor to an dense one

username: peter_zhu
cooked: <p>Values with the same index in a sparse tensor are accumulated when  use torch.cuda.sparse.FloatTensor(i, v, torch.Size([2,3])).to_dense().</p>
<p>How to only remain the max value when  convert the sparse tensor to an dense one.<br>
Any tricks to implement this?</p>
<p>Specifically,<br>
indices = torch.cuda.LongTensor([[0, 1, 0], [2, 0, 2]])<br>
values = torch.cuda.FloatTensor([3, 4, 5])<br>
t = torch.cuda.sparse.FloatTensor(i, v, torch.Size([2,3])).to_dense()<br>
then<br>
t’s values are [[ 0  0  5],[  4  0  0]] rather than [[ 0  0  8],[  4  0  0]].</p>

username: tom
cooked: <p>This is <strong>NOT</strong> what sparse tensors want to do! The sparse tensor represents <code>[[ 0 0 8],[ 4 0 0]]</code>.</p>
<p>The operation you want is requested in <a href="https://github.com/pytorch/pytorch/issues/22378" rel="nofollow noopener">#22378</a>, which also links to a great <a href="https://pytorch-scatter.readthedocs.io/en/latest/functions/max.html" rel="nofollow noopener">third party implementation of <code>scatter_max</code></a>.</p>
<p>Best regards</p>
<p>Thomas</p>
How to train my custom parameter?

username: wonchulSon
cooked: <p>I have a idea to research.<br>
So I need to train my own parameter(<code>alpha</code>).<br>
For example, <code>alpha</code> x conv_filter formula.<br>
To train <code>alpha</code> with conv_fitler together, how should I code this?<br>
Thank you.</p>

username: akernel
cooked: <p>You could write your own <code>nn.Module</code> that includes a trainable <code>alpha</code> parameter:</p>
<pre><code class="lang-auto">class MyLayer(nn.Module):
    def __init__(self, *args, **kwargs):
        self.alpha = nn.Parameter(torch.tensor(0.))
        # Other necessary setup
        super().__init__(*args, **kwargs)

    def forward(self, x):
        # Necessary forward computations
        return output
</code></pre>
<p>Using this <code>MyLayer</code> module as part of a network architecture will allow you to learn <code>alpha</code> along with all of the other usual parameters.  Hope this helps!</p>

username: wonchulSon
cooked: <p>Thank you!<br>
I will try this^^</p>
Parameter sharing along a dimension

username: zfzhang
cooked: <p>Hi,<br>
Suppose I have a weight tensor of shape (4,1), but I want to expand it along the second dimension so that it becomes (4,4) but the underlying storage does not change, and used as (16,1).<br>
In code:</p>
<pre><code class="lang-auto">nn.parameter.Parameter(torch.Tensor(4,1).expand(4,4).view(16))
</code></pre>
<p>Note the above code will throw exception in the view() call, but is there (will there be) support for such usage?<br>
Thanks!</p>

username: tom
cooked: <p>You need to keep the Parameter of shape <code>4 x 1</code> and only do the expand in <code>.forward</code>.<br>
You need reshape instead of view, as it’ll need to instantiate the larger matrix.</p>
<p>Best regards</p>
<p>Thomas</p>

username: zfzhang
cooked: <p>Right, thank you very much!</p>
Tensor normalization

username: sh0416
cooked: <p>I have some question about normalization technique.</p>
<p>I encounter very weird phonomena doing normalization.</p>
<pre><code class="lang-auto">x = torch.rand(10, 10)
print((x - x.mean(dim=1, keepdim=True)).mean(dim=1, keepdim=True))
</code></pre>
<p>It doesn’t give zero tensor! The result is very small but it is not zero.</p>
<p>What is going on???</p>

username: tom
cooked: <p>This is numerical precision - floating point arithmetic isn’t exact, so you will not get the exact mean. If you switch to double, you get from 5e-8ish to 5e-17ish in your example. This can be more substantial if you have larger tensors.</p>
<p>Best regards</p>
<p>Thomas</p>
Calculate gradients when network is set to .eval()

username: Tahlor
cooked: <p>I have 2 networks (call them <em>preprocess</em> and <em>final).</em> I’m trying to feed an output from <em>preprocess</em> into <em>final,</em> where the <em>preprocess</em> network is being trained based on a loss from <em>final</em>, and the <em>final</em> network is being evaluated but not trained (e.g. eval(), where dropout is evaluated correctly and weights are frozen etc.).</p>
<p>Something like this:</p>
<pre><code class="lang-auto"># Freeze final
for p in final.parameters():
	p.requires_grad = False

final.eval()
preprocess.train()

output = final(preprocess(input))
loss(output)

preprocess.backward()
optimizer.step()
</code></pre>
<p>I’m getting the error “RuntimeError: cudnn RNN backward can only be called in training mode”.  What’s the idiomatic way of using train/eval networks in concert/simultaneously?</p>
<p>Notes:</p>
<ul>
<li>Both networks are actually being trained in an alternating fashion</li>
<li>final actually has two parts; the input to preprocess is actually an intermediate output of final, which is later fed back into a different part of final</li>
</ul>

username: tom
cooked: <p>This is a limitation of the CuDNN implementation (combined with the layering of autograd over ATen, which does not allow the CuDNN implementation in ATen to see whether gradients are required). An easy way is to fix this is to set the entire network to <code>eval</code> but then put the RNNs into <code>train</code> (this re-enables dropout you might have in them, though). Another way is to disable the CuDNN backend (<code>with torch.backends.cudnn.flags(enabled=False): ...</code>) or use <a href="https://pytorch.org/blog/optimizing-cuda-rnn-with-torchscript/" rel="nofollow noopener">JITed RNNs</a>.</p>
<p>Best regards</p>
<p>Thomas</p>
Requires_grad in tensor vs in nn.Parameter

username: Raphikowski
cooked: <p>Dear all,</p>
<p>I have a question concerning two different possibilities of initializing a nn.Parameter. In particular, would you expect a different behaviour of the following two possible initializations:</p>
<p>a = nn.Parameter(torch.zeros(5, requires_grad=True))</p>
<p>or</p>
<p>a = nn.Parameter(torch.zeros(5), requires_grad=True)</p>
<p>Thank you in advance!</p>

username: Raphikowski
cooked: <p>would you expect a different behaviour during training for these two cases?</p>

username: tom
cooked: <p><code>nn.Parameter</code> will enable gradients by default, so the right way to achieve the same thing is just <code>a = nn.Parameter(torch.zeros(5))</code>.<br>
Your first variant works, but it is wrong - it pretends that by changing to <code>requires_grad=False</code> there would be a difference.<br>
The second variant just spells out the default parameter.</p>
<p>Best regards</p>
<p>Thomas</p>
Memory consumption of computation graph

username: Sourena_Yadegari
cooked: <p>Is there a way to determine how much memory the computational graph of<br>
a tensor occupies in memory? This is important to know sometimes for memory-bottlenecked networks as one can move the network to a less memory hungry model (if such model exists) but with comparable performance (for example GRU instead of LSTM).</p>

username: tom
cooked: <p>For GPU memory, you can use <a href="https://pytorch.org/docs/stable/cuda.html#torch.cuda.max_memory_allocated" rel="nofollow noopener"><code>torch.cuda.max_memory_allocated</code></a> and <code>torch.cuda.memory_allocated</code>.</p>
<p>Best regards</p>
<p>Thomas</p>
How to get higher order gradients w.r.t inputs?

username: zhaopku
cooked: <p>I am trying to compute hessian matrix of input. It is straightforward to get the 1st derivative of input. But is there a way to compute 2nd order gradients of loss w.r.t to input in pytorch?</p>

username: tom
cooked: <p>You can compute Hessian-Vector-Products, the trick is to pass <code>build_graph=True</code> to <code>torch.autograd.grad</code> or <code>x.backward</code>. Then you can use gradients in computation and <code>y.backward</code> or <code>torch.autograd.grad</code> again will do the right thing.</p>
<p>Best regards</p>
<p>Thomas</p>