akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Quantization-aware training conv1D, LSTM support

Paul_Creaser: <p>I am just taking a look at using Quantization-aware training. At the moment I am making my way through the tutorials.</p>
<p>My current models make use of layers such as conv1D,and in some cases LSTM and GRU etc…</p>
<p>According to the documention this is not currently supported for those particular layer types. As a work around, one possibility would be to replace the conv1D with conv2D. I would convert the data to 2D (actually remains 1D, but 2D in dimensions).</p>
<p>With regards to LSTM, GRU, I am not yet sure.</p>

Paul_Creaser: <p>It seems this may partially answer my question.</p>
<aside class="quote" data-post="1" data-topic="68016">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/snakers41/48/5641_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/quantization-support-for-1d-convolutions/68016">Quantization support for 1D convolutions?</a> <a class="badge-category__wrapper " href="/c/quantization/17"><span data-category-id="17" data-drop-close="true" class="badge-category " title="This category is for questions, discussion and issues related to PyTorch’s quantization feature."><span class="badge-category__name">quantization</span></span></a>
  </div>
  <blockquote>
    Hi! 
Are you planning to add support for 1D convolutions for <a href="https://pytorch.org/docs/stable/quantization.html" rel="noopener nofollow ugc">quantization</a>?
  </blockquote>
</aside>


dskhudia: <p><a class="mention" href="/u/paul_creaser">@Paul_Creaser</a>,<br>
Conv1d is now available in nightly builds. There is <a href="https://pytorch.org/docs/stable/quantization.html#torch-nn-quantized-dynamic" rel="nofollow noopener">LSTM available with dynamic quantization</a> and GRU is currently not available. However, an <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/quantized/dynamic/modules/rnn.py#L67" rel="nofollow noopener">RNN base class is available with dynamic quantization</a>.</p>
What is the best practices of logging in distributed training?

XYJin: <p>Did some googling and found very few discussions on this matter. Is it best to perform all-reduce on, say, loss values, and keep track of it within the process with rank 0, like what the official tutorial recommends for checkpoints?</p>

ibro45: <p>That seems to be the case, check out how they do it in the <a href="https://github.com/facebookresearch/maskrcnn-benchmark/blob/57eec25b75144d9fb1a6857f32553e1574177daf/maskrcnn_benchmark/engine/trainer.py#L88" rel="nofollow noopener">implementation of Mask-RCNN</a>. They use <code>reduce</code> instead of <code>all_reduce</code> because, for logging, you only need the reduced and averaged values on the rank 0 process.</p>
Masked Batch Normalization

Novak: <p>Consider a usage of BatchNorm1d, with batches and channel data, where the convolutional axis is time:</p>
<p>If the batch size is one, the shape of the input tensor is [1, C, T] and normalization proceeds as appropriate.</p>
<p>If the batch size is (say) eight, the shape of the input tensor is [8, C, T] and normlization proceeds under the assumption that all of the inputs are the same size and that ever value of the input tensor is valid and contributes to the statistics of the batch.</p>
<p>But this is not always a good assumption with temporal data-- we might have 8 sound files with different length, so that the input tensor might better be described as [8, C, T_max], where T_max is the T of the longest sample, and the other batch entries being padded/masked somehow.</p>
<p>Here is the rub:  I cannot think of an appropriate padding/masking scheme.  If one masks with 0, the normalization will assume that is actual data, which it is not.  np.nan and np.inf are sticky and will convert the entire output to np.nan or np.inf, which is obviously not desirable.</p>
<p>Is there some way to convey to a normalizer that some data is masked and should not be considered for normalization, in a way which keeps batch parallelization and speedups on a GPU.</p>
<p>(Note:  This is not a recurrent neural network application, it is a convolutional application.  I am really looking for a way to stay in the convolutional format.)</p>

Jan_Vainer: <p>I would also be interested in a solution to this problem. But perhaps for now, it is necessary to implement custom layer</p>

amiasato: <p>I implemented a custom layer with a simple trick which computes the variance based on the Var(X) = E(X^2) - E(X)^2 formula. So far, I haven’t noticed any significant slowdowns: <a href="https://gist.github.com/amiasato/902fc14afa37a7537386f7b0c5537741" rel="nofollow noopener">https://gist.github.com/amiasato/902fc14afa37a7537386f7b0c5537741</a></p>
How the multiple GPUs interconnect when using the DataParallel to train model on multi-GPUs

zzsunshine: <p>Hi all,</p>
<p>I try to train a model by using multi-GPUs on single machine. And I want to figure out how the multiple GPUs connect with each other, (I mean the shape of the connection, full connect? in parallel? or in series?</p>
<p>Thanks!</p>

ptrblck: <p>If you want to check how the devices are connected inside your machine, you could run <code>nvidia-smi topo -m</code> in your terminal.</p>
<p>I might have misunderstood the question, but <code>nn.DataParallel</code> replicated the model on each device as explained <a href="https://pytorch.org/tutorials/beginner/former_torchies/parallelism_tutorial.html">here</a>.</p>

zzsunshine: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thanks for your reply!  I used the DataParallel module, and after I checked the code, I found that the GPU0 will gather the results from other GPUs like GPU1,2 and then it processes some computations by itself and then scatter to others. So I want to check the network between them (the way of communication). And how the GPU communicate with the CPU in this DataParallel approach. I tried to dive into the C++ code, however, I am not clear where could I search for this.</p>
<p>Thanks</p>

ptrblck: <p>You can find the implementation in <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/parallel/data_parallel.py">data_parallel.py</a>.</p>

zzsunshine: <p>It helps. Thanks for your help</p>
Not using register_buffer correctly

edyuan: <p>Hi Guys,</p>
<p>I keep getting this error with a custom module I’m trying to create:</p>
<pre><code>"expected device cuda:0 and dtype Float but got device cpu and dtype Float"
</code></pre>
<p>From what I can understand, I need to use register_buffer on the offending tensor so that it gets set to CUDA when I use model.CUDA().  I think I’m doing that below, but I keep getting the same error.</p>
<pre><code>class NoiseInjection(nn.Module):
    def __init__(self, shape):
        super(NoiseInjection, self).__init__()
        self.injected_noise = self.build_noise_injection(shape)

    def build_noise_injection(self, shape):
        noise =  torch.randn([shape[0], 1, shape[2] , shape[3]], requires_grad=False) #NCHW
        self.register_buffer('noise', noise)
        weights = nn.Parameter(torch.zeros([1, shape[1], 1 , 1])) # different weight for each channel
        noise_weighted = weights * self.noise
        return noise_weighted

     def forward(self,x):
         out = x + self.injected_noise
         return out
</code></pre>
<p>Anyone know what’s going on?</p>

charan_Vjy: <p>The input ‘x’ may have been pushed to cuda. Make sure you also push self.injected_noise to cuda. Something along the lines of<br>
<code>self.injected_noise = self.build_noise_injection(shape).cuda()</code></p>

edyuan: <p>I did this and the code now runs:</p>
<pre><code> self.injected_noise = nn.Parameter(self.build_noise_injection(shape))
</code></pre>
<p>I set it has a parameter since it is composed of weights * self.noise, where weights is trainable.</p>
<p>So to clarify - do I need to push every single variable in the graph to cuda, either defining it as a parameter or as a buffer?</p>
<p>Much thanks</p>

charan_Vjy: <p>Yeah, to avoid conflict, yeah. You cant operate on variables that are on different devices.</p>
How to average two optimizers

divinho: <p>This is related to doing distributed training.</p>
<p>If I have two Adam optimizers (used on the same model arch) how can I create a new optimizer with a state that is the average of the two?</p>
<p>The problem I have is that it seems the way the moving average are stored is in <code>optim.state_dict()['state']</code>, which is a dictionary where the key is a hash of a parameter (and the values what I’m interested in). But the hash depends on the value of the parameter, so given <code>opt_a</code> and <code>opt_b</code> how would I loop over the two in such a way that guarantees I’m getting the right pairings?</p>
How to extract the largest value of a tensor while keep it's origin position

han.liu: <p>I have a tensor:</p>
<pre><code class="lang-auto">a = torch.Tensor([[0.0, 0.9, 0.8, 0.0, 0.2, 0.0],
                  [0.2, 0.0, 0.0, 0.8, 0.7, 0.0],
                  [0.6, 0.0, 0.0, 0.0, 0.0, 0.7]])
</code></pre>
<p>and I want to extract the max value of dim=1, and keep it’s original position, like this:</p>
<pre><code class="lang-auto">res = torch.Tensor([[0.0, 0.9, 0.0, 0.0, 0.0, 0.0],
                  [0.0, 0.0, 0.0, 0.8, 0.0, 0.0],
                  [0.0, 0.0, 0.0, 0.0, 0.0, 0.7]])
</code></pre>
<p>How to implement this?</p>
<p>Thank you!</p>

charan_Vjy: <p>This should do the trick</p>
<pre><code class="lang-auto">import torch
a = torch.Tensor([[0.0, 0.9, 0.8, 0.0, 0.2, 0.0],
                  [0.2, 0.0, 0.0, 0.8, 0.7, 0.0],
                  [0.6, 0.0, 0.0, 0.0, 0.0, 0.7]])
b = torch.argmax(a,dim=1).unsqueeze(1)
one_hot = torch.zeros(a.shape).scatter(1,b,1)
req_op = a * one_hot
</code></pre>
Symetric convolution weights

obadia_yohan: <p>If you have prior knowledge that your convolution weights should be symetrical on a given axis, how would you enforce that constraint ?</p>

obadia_yohan: <p>I came up with a solution that avoids hard constraints by increasing the loss if the weights are not symetrical.</p>
<pre><code>def sym_loss(w, alpha=0.1, dims=(0,)):
    nb_w = w.shape[0]*w.shape[1]
    return alpha * torch.pow(w - w.flip(dims=dims), 2).sum().sqrt()/nb_w
</code></pre>
<p>Here, <code>w</code> corresponds to the weights of the convolutional layer of interest. The default <code>dims</code> enforces an horizontal symetry. The coefficient <code>alpha</code> indicates the importance of this loss in the overall loss.</p>
<p>If you have any feedback on this approach, please comment.</p>
<p>Personnaly I see one draw back which is that you still have to search and update weights on both sides of the symetrical axis while you could constrain your search space more efficiently and precisely is a string constraint defined in advance. Here you won’t have a guarentee of a symetry.</p>

JuanFMontesinos: <p>Hi,<br>
I think you can achieve this by using a custom nn.Module.<br>
You can write a nn.Module which has only half of the parameters for the convolution.<br>
In the forward pass you just need to build the whole kernel but using that reduced amount of parameters and call the convolution functional.</p>
<p>This way you need no loss contrains.</p>

obadia_yohan: <p>Thank you for your feed back. I’ll try that and post here the final module I came up with. In the mean time, you get the solution reward <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
PicklingError when using torch.save

plus_real: <p>when runing the code below<br>
print(‘Saving best accuracy…’)<br>
state = {<br>
‘model’: model.modules,<br>
‘best_acc’: test_best_acc,<br>
‘epoch’: epoch<br>
}<br>
if not os.path.isdir(‘checkpoint’):<br>
os.mkdir(‘checkpoint’)<br>
torch.save(state, ‘checkpoint/ckpt.t7’)</p>
<h2>it happens</h2>
<p>PicklingError                             Traceback (most recent call last)<br>
 in ()<br>
64 for epoch in range(1, 300):<br>
65     train_epoch(epoch, baseCNN, train_loader, rms_baseCNN, cross_entropy)<br>
—&gt; 66     vali_epoch(epoch, baseCNN, vali_loader, cross_entropy)</p>
<p> in vali_epoch(epoch, model, data_loader, criterion)<br>
59         if not os.path.isdir(‘checkpoint’):<br>
60             os.mkdir(‘checkpoint’)<br>
—&gt; 61         torch.save(state, ‘checkpoint/ckpt.t7’)<br>
62<br>
63</p>
<p>/usr/local/anaconda2/lib/python2.7/site-packages/torch/serialization.pyc in save(obj, f, pickle_module, pickle_protocol)<br>
118         f = open(f, “wb”)<br>
119     try:<br>
–&gt; 120         return _save(obj, f, pickle_module, pickle_protocol)<br>
121     finally:<br>
122         if new_fd:</p>
<p>/usr/local/anaconda2/lib/python2.7/site-packages/torch/serialization.pyc in _save(obj, f, pickle_module, pickle_protocol)<br>
184     pickler = pickle_module.Pickler(f, protocol=pickle_protocol)<br>
185     pickler.persistent_id = persistent_id<br>
–&gt; 186     pickler.dump(obj)<br>
187<br>
188     serialized_storage_keys = sorted(serialized_storages.keys())</p>
<p>PicklingError: Can’t pickle &lt;type ‘instancemethod’&gt;: attribute lookup <strong>builtin</strong>.instancemethod failed</p>
<p>So how can I solve this problem? Thanking very much</p>

albanD: <p>Hi,</p>
<p>Please see the documentation for serialization: <a href="http://pytorch.org/docs/notes/serialization.html">http://pytorch.org/docs/notes/serialization.html</a><br>
In particular, I think you should be saving <code>model.state_dict()</code> and not <code>model.modules</code>.</p>

plus_real: <p>Thanking you, I solve the problem by following your advice</p>

mkamp: <p>I have seen this recommendation many times, to save the state dict only, but what about the model instantiation then? How do you know which arguments you used to instantiate the model? Do you write custom code, just to capture those and store these separately?</p>

albanD: <p>In general I would say yes.<br>
You can torch.save a tuple that contains both the arguments to re-create the model and the state dict.<br>
This is a limitation of python not to be able to serialize complex objects in a convenient and reliable way <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>

mkamp: <p>Thanks for the quick answer.</p>
<p>Ok, I will try that. I suspect it also means that I need to call the constructor originally with a dict and keep the dict around, so that I can use it for save as well. Not a huge burden, but feels not all that elegant.</p>
RuntimeError when use DataParallel

yf_zhang: <pre><code class="lang-auto">class Add(nn.Module):
    def __init__(self, scope=None):
        nn.Module.__init__(self)

    def forward(self, bottoms):
        # bottoms is a list of tensor.
        assert len(bottoms) &gt; 1, "The length of bottoms must be larger than 1."
        result = bottoms[0]
        for i in range(1, len(bottoms)):
            result = result + bottoms[i]
        return [result]
</code></pre>
<p>My model use the Add layer, it’s the subclass of nn.Module. When I use DataParallel to train my model, the runerror happened, “Expected tensor for argument <span class="hashtag">#1</span> ‘input’ to have the same device as tensor for argument <span class="hashtag">#2</span> ‘weight’;”. It’s OK when I use a single GPU;<br>
What should I do make it train normally;</p>

charan_Vjy: <p>Make sure you mount the data on the same device as that of the model. When you load the model use,  <code>torch.nn.DataParallel(model, device_ids=[0, 1,2,..])</code>. The first device(in the above example, device 0 will be where the model is mounted before it is replicated).<br>
Make sure you mount input on this device itself. The first gpu will be used as a platform to scatter it onto other devices.</p>
Cannot update ModelDict with another ModelDict

Tobimaru: <p>Hi. I’m new to this forum, so I hope I’m making myself clear enough <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:">. I’m working on a project using pytorch 1.4.0 and I’m getting an error while trying to merge two ModelDict instances. According to the documentation of <a href="https://pytorch.org/docs/stable/nn.html?highlight=module#torch.nn.Module" rel="nofollow noopener">torch.nn.ModelDict</a>, a ModelDict object can be appended to another one using the update() function. I tried this with the following code:</p>
<pre><code class="lang-auto">model1 = torch.nn.ModuleDict(OrderedDict([
          ('conv1', torch.nn.Conv2d(1,20,5)),
          ('relu1', torch.nn.ReLU())]))

model2 = torch.nn.ModuleDict(OrderedDict([
          ('conv1', torch.nn.Conv2d(1,20,5)),
          ('relu1', torch.nn.ReLU())]))

model1.update(model2)
</code></pre>
<p>but this gives the following error:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
&lt;ipython-input-157-967ebf8b86d0&gt; in &lt;module&gt;
     10 print(type(model2))
     11 
---&gt; 12 model1.update(model2)

c:\python36\envs\modern_pointcn\lib\site-packages\torch\nn\modules\container.py in update(self, modules)
    348                     raise ValueError("ModuleDict update sequence element "
    349                                      "#" + str(j) + " has length " + str(len(m)) +
--&gt; 350                                      "; 2 is required")
    351                 self[m[0]] = m[1]
    352 

ValueError: ModuleDict update sequence element #0 has length 5; 2 is required
</code></pre>
<p>by looking into the source code, it seems a torch.nn.ModelDict is not recognized as a Mapping. Is this a bug? If not, how can I append two ModelDict instances?</p>

ptrblck: <p>This might be a bug, as the condition is apparently checking the length of the <code>key</code> instead of the <code>tuple</code>.<br>
Could you please create an issue on <a href="https://github.com/pytorch/pytorch/issues">GitHub</a>?</p>

Tobimaru: <p>Thank you for the feedback. Yes, I’ll open an issue on Github.</p>
CDF for Beta Distribution missing, any implementation tips?

rasbt: <p>It seems that the cdf for most distributions works, e.g.,</p>
<pre><code class="lang-python">import torch
from torch.distributions.laplace import Laplace

m = Laplace(torch.tensor([0.0]), torch.tensor([1.0]))
m.cdf(-0.0886)
</code></pre>
<p>However, for Beta distributions, it does not seem to be implemented yet:</p>
<pre><code class="lang-python">from torch.distributions.beta import Beta

m = Beta(torch.tensor([0.5]), torch.tensor([0.5]))
m.cdf(0.0029)

---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
&lt;ipython-input-10-b5e9a04733d4&gt; in &lt;module&gt;
      2 
      3 m = Beta(torch.tensor([0.5]), torch.tensor([0.5]))
----&gt; 4 m.cdf(0.0029)

~/miniconda3/lib/python3.7/site-packages/torch/distributions/distribution.py in cdf(self, value)
    152             value (Tensor):
    153         """
--&gt; 154         raise NotImplementedError
    155 
    156     def icdf(self, value):

NotImplementedError: 

</code></pre>
<p>I was searching for GitHub PRs/Issues but couldn’t find any. Does anyone have any tips regarding an efficient implementation in PyTorch for this? I was wondering, maybe someone implemented this already in a numerically stable way or so. Any pointers would be appreciated!</p>

jamie: <p>Something like this with numerical integration using <code>torch.trapz</code>?</p>
<pre><code class="lang-python">import torch
from torch.distributions import Beta

conc1, conc0 = 2.0, 3.0
bt = Beta(conc1, conc0)
def beta_cdf(b, npts=100):
    x = torch.linspace(0, b, npts)
    return torch.trapz(bt.log_prob(x).exp(), x)
beta_cdf(0.65)
</code></pre>
Optimizer: scheduler

CaspervE: <p>I am using MultiStepLR to change the learning rate which is great but there is a problem:</p>
<p>First let me sketch the context: I am using Colab to train, validate and test large networks (currently ResNet44 architectures). This obstructs me from doing the whole training process at once: I have to save the network from time to time. Since I am using MultiStepLR, it has to remember somehow in which epoch it was. My question is therefore, how does MultiStepLR knows which epoch it is and can I change that knowledge?</p>
<p>Thanks.</p>
Automatic loading/unloading models from hdd to gpu

yegane: <p>I am implementing a project that uses e.g. 100 models, and it trains random samples of the models in each iteration. The problem is my GPU is  not sufficient, so I was wondering if it is possible to have a pointer to model in HDD, so when it is time for training, it retrieve them and after the training they save on the disk. Therefore, we would have only limited number of active models in GPU, and the rest would be in hibernation, and wait for the call.<br>
I would be happy if pytorch would have this option, and it wouldn’t be necessary to implement this part myself.<br>
In short: I want to save and load a list of models, without going into making a dictionary and saving the files separately.</p>
Clarification on re-initializing optimizer in every epoch

shubhvachher: <p>You shouldn’t be doing that, right? fmi as well as <a href="https://github.com/psyec1/Lipreading-PyTorch/issues/7" rel="nofollow noopener">my issue here</a></p>

ptrblck: <p>It’s usually not wanted or necessary.<br>
However, if your optimizer does not use any internal state, reinitializing won’t change anything.</p>

shubhvachher: <p>In what situation would an optimizer have an internal state? I see <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html#what-is-a-state-dict" rel="nofollow noopener">here</a> an example of the state_dicts, but it seems, in that example the “state” key of the optimizer’s state_dict is empty.</p>

ptrblck: <p><code>torch.optim.Adam</code> for example has internal states, as it computes the running averages of gradient and its square.</p>
<p>If you don’t need to reset the optimizer (there might be use cases I’m not aware of), I would recommend to initialize it once and just use it inside the training loop.</p>

shubhvachher: <p>Oh. I was using Adam! Thanks for clarifying.</p>
<p>A not so thorough reading of, and lack of resolution in, <a href="https://discuss.pytorch.org/t/adaptive-learning-rate/320/36">old learning rate updation queries like this</a> might be behind <a href="https://github.com/psyec1/Lipreading-PyTorch/issues/7" rel="nofollow noopener">such confusion</a>.</p>
<p>Maybe the optimizers where the internal state matters should be tagged at the <a href="https://pytorch.org/docs/stable/optim.html" rel="nofollow noopener">torch optim</a> page? Also, possibly at <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-loading-a-general-checkpoint-for-inference-and-or-resuming-training" rel="nofollow noopener">saving and loading models for inference</a>. Thanks again for the prompt response.</p>

Peter_Featherstone: <blockquote>
<p>there might be use cases I’m not aware of</p>
</blockquote>
<p>What if the EMA averaging of the gradients and its squares are killing the training? I guess that’s where you might want to swap to something like SGD</p>
Object detection and counting but with labels that are images themselves

PrathamSolanki: <p>Hello everyone,</p>
<p>I have a set of main images and an image per class/label.</p>
<p>The main images contain a number of objects that belong to one of the fixed number of classes. I want a class wise count of the number of objects in the main image. Now, the thing is the main images are not annotated with pixel wise location for the objects. Instead I have smaller images for each of the classes. One each for a class.</p>
<p>For example, think of the main image as a fruit basket. The labels are oranges, apples and grapes. Alongside the main images, I have smaller, particular images of an orange, apple and grape. My task to return the number of oranges, apples and grapes given an image of fruit basket.</p>
<p>How would you go about to do it?</p>
Unable to add noise to DCGAN

therohanpandey: <p>Hi! I’m really new to GAN’s and was trying DCGAN for generating samples of COVID-19 Chest-Xrays. I was exploring the possibility of using GAN’s to increase the dataset and to see if it helps improve a classifier. I was trying to add white noise to the Discriminator and I am unable to figure out how to do so. Can someone help? I understand that I need to add the noise to the input but I am unable to do so. <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/6/d6c0a2aca7452018149510d2ac6428135b39e8cf.png" data-download-href="https://discuss.pytorch.org/uploads/default/d6c0a2aca7452018149510d2ac6428135b39e8cf" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/6/d6c0a2aca7452018149510d2ac6428135b39e8cf.png" alt="image" data-base62-sha1="uDMTGKsdpFbUHbiaCfu7zukG715" width="690" height="53" data-dominant-color="F2F2F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">725×56 1.85 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>You could define the mean and std and add the noise directly to your input image:</p>
<pre><code class="lang-python">mean = 0.
std = 10.
noise = (torch.randn(1, 3, 224, 224) + mean) * std
input = input + noise
</code></pre>
<p>Let me know, if that would work for you.</p>

therohanpandey: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> This seems to be working! However after reading up, I did find that it is a good practice to decay/ reduce the noise for every iteration. Any inputs as to how that can be incorporated?</p>

ptrblck: <p>Good to hear it’s working!<br>
You could divide the standard deviation of the noise with the current epoch or use any other decaying function.</p>

therohanpandey: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks a lot for your inputs! If there’s any other suggestions you might have for training GANs, I’d be grateful! Anyways thanks for the help!</p>

ptrblck: <p>My best suggestion is a lot of patience. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

therohanpandey: <p>That’s the best advice! Thanks! <img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=9" title=":grin:" class="emoji" alt=":grin:"></p>
AttributeError: 'Net' object has no attribute 'module'

yw_tt: <p>state_dict = net.module.state_dict()</p>
<p>when I execute above code, I got the error:  AttributeError: ‘Net’ object has no attribute ‘module’, how can I deal with it? Thanks a lot.</p>

MariosOreo: <p>If you want to get the state of the whole network, you should use <code>net.state_dict()</code>.</p>
<p>If you just want to get the state of a specific sub-module, you should use the sub-module name like <code>net.sub_module_name.state_dict()</code>.</p>

yw_tt: <p>Thank you very much. I will try this right away.</p>

youngyh: <p>In official tutorial, they suggest “torch.save(model.module.state_dict(), PATH) " when using Dataparallel<br>
But when I try this, I met same problem [‘Net’ object has no attribute ‘module’].<br>
Why? So should I  always save model using  " torch.save(net.state_dict()”?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/8/87fbb4adff0b711ffea5396b024a58d799c13a2a.png" alt="image" data-base62-sha1="joXJBRGomdI7F3jL4fJ4Q5nIyG6" width="431" height="152"></p>

yw_tt: <p>Always use net=torch.nn.DataParallel(net) even if you only have one GPU. The problem will not occur.</p>

youngyh: <p>Thank you! I will try this when saving and loading models.<br>
But still don’t understand why the official tutorial add “module” when saving model.</p>

mathematics: <p>It feels pretty head banging ,when model is trained for 5 minutes and get this type of error</p>
Create two-channel ResNet34 model to train from scratch

Praveen_G_B: <p>Hi,<br>
I want to train a ResNet34 architecture from scratch using two-separate gray scale images, Can someone help me with this.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/5/25e2401988bdcf7c24746afd566ebc71bc28f7b6.png" alt="15524c_patch_01_down_sampled_spatial_image" data-base62-sha1="5p8p9faJOnpUTdxlm5kWCPJm9ym" width="224" height="224"></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/9/4948d46289417406669134d45cd4e675cf165337.png" alt="10191c_patch_01_original_mosaic_image" data-base62-sha1="asiTjeXjKB3fietUUHosBvUFqGb" width="224" height="224"></p>
<p>These are the two grayscale images I want to feed into ResNet34 network.</p>
<p>Thanks in Advance</p>
Pip install pytorch or conda install pytorch

Imene.b: <p>hy,</p>
<p>1- I would like to know the difference between pip install pytorch and conda install pytorch<br>
and<br>
2- if i choose to install pytorch vias cuda, is it necessary to have GPU card</p>

cyril9227: <p>If you only need the CPU version you can specify it :</p>
<pre><code class="lang-auto">conda install pytorch torchvision cpuonly -c pytorch

</code></pre>
<p>or</p>
<pre><code class="lang-auto">pip install torch==1.4.0+cpu torchvision==0.5.0+cpu -f https://download.pytorch.org/whl/torch_stable.html

</code></pre>

Imene.b: <p>thank you<br>
and if i install pytorch with cuda is it necessary to have GPU card?</p>

JuanFMontesinos: <p>In pytorch you can choose whether to use gpu or cpu.<br>
Pytorch with cuda is nothing but  binary files with cuda within. Cpu version will work I would say.</p>
<p>Conda is (anaconda) a set of pre-compiled python packages. I personally don’t like it.</p>

Imene.b: <p>now  understand thank you</p>
Jaccard coefficient right definition

Alex5: <p>Hi,<br>
I use the Jaccard coefficient to validate my binary segmentation model. I’ve found a definition:</p>
<blockquote>
<p>def get_jaccard(y_true, y_pred):<br>
epsilon = 1e-15<br>
intersection = (y_pred * y_true).sum(dim=-2).sum(dim=-1).sum(dim = -1)<br>
union = y_true.sum(dim=-2).sum(dim=-1).sum(dim=-1) + y_pred.sum(dim=-2).sum(dim=-1).sum(dim = -1)<br>
return (intersection / (union - intersection + epsilon)).mean()</p>
</blockquote>
<p>Input and output have the same shape: [N, 1, 256, 256] where N is the batch size following by the image size. So it basically performs intersection and union on each image in the batch then takes the mIoU over the batch.</p>
<p>I’ve made a lightened implementation:</p>
<blockquote>
<p>def jaccard_coeff(input, target):<br>
eps = 1e-15<br>
input = input.view(-1)<br>
target = target.view(-1)<br>
intersection = (input * target).sum()<br>
union = (input.sum() + target.sum()) - intersection<br>
return (intersection / (union + eps))</p>
</blockquote>
<p>I directly flatten the tensors and perform operation without computing the mean. It returns different results and I wanted to know which one is the true one ?<br>
The results are not equal so I would like to be sure I’m using the metric in the right way for validation</p>
<p>Thanks</p>

vfdev-5: <p>There are several repositories where it was already implemented (probably in the correct way):</p>
<ul>
<li><a href="https://github.com/pytorch/vision/blob/master/references/segmentation/utils.py#L95" rel="nofollow noopener">https://github.com/pytorch/vision/blob/master/references/segmentation/utils.py#L95</a></li>
<li><a href="https://github.com/pytorch/ignite/blob/master/ignite/metrics/confusion_matrix.py#L129" rel="nofollow noopener">https://github.com/pytorch/ignite/blob/master/ignite/metrics/confusion_matrix.py#L129</a></li>
</ul>
<p>In your case, as far as I understand the difference is the following:</p>
<ul>
<li>first implementation computes the mean over N images of Jaccard Index per image</li>
<li>second implementation computes Jaccard Index as all N images were concat.</li>
</ul>
<p>Maybe, the second is more standard.</p>
<p>HTH</p>

Alex5: <p>Thanks for your answer <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:"><br>
Indeed there are similarities with the second option.</p>
Convolution algorithm doesn't work with sparse tensor

cyril9227: <p>Hi guys,</p>
<p>Just wondering if I made a mistake or if it’s not currently implemented, when I try to use a conv2d layer on a sparse tensor I get the following error :</p>
<pre><code class="lang-auto">RuntimeError: sparse tensors do not have is_contiguous

</code></pre>
<p>I used this code :</p>
<pre><code class="lang-auto">import torch
class Net(torch.nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = torch.nn.Conv2d(1, 6, 3)

    def forward(self, x):
        x = self.conv1(x)
        return x
    
inputs = torch.sparse.FloatTensor(1, 1, 500, 500)
model = Net()
model(inputs)
</code></pre>
LSTM zeroing hidden state

tcapelle: <p>Hello,<br>
I have a question about zeroing the hidden state before each epoch.<br>
Some code do <code>None</code> re initialization and othser do <code>self.h = [torch.zeros(2, bs, n_hidden) for _ in range(n_layers)]</code><br>
Do both are equivalent?<br>
I am mostly interested because the second one, needs a fixed batch size.</p>
RMSLE loss function

soadrhcp: <p>Hello world,</p>
<p>I’m planning to use the Root Means Squared Log Error as a loss function for an image to image regression problem (these are not properly images but fields with variable norms). I did not find this function as part of the torch.nn.modules.loss implemented functions. What do you think would be the best way to implement Root Means Squared Log Error with PyTorch ?</p>

Pranaymodukuru: <p>One way of doing this is to create a <code>nn.Module</code> class,</p>
<pre><code class="lang-auto">class RMSLELoss(nn.Module):
    def __init__(self):
        super().__init__()
        self.mse = nn.MSELoss()
        
    def forward(self, pred, actual):
        return torch.sqrt(self.mse(torch.log(pred + 1), torch.log(actual + 1)))
</code></pre>
<p>The equation I took as reference here is from a <a href="https://www.kaggle.com/c/ashrae-energy-prediction/discussion/113064" rel="nofollow noopener">Kaggle discussion</a>.</p>
<p>and using it just like any other loss function provided by pytorch.</p>
<pre><code class="lang-auto">pred = torch.tensor([600.], requires_grad=True)
actual = torch.tensor([1000.])

criterion = RMSLELoss()

rmsle = criterion(pred, actual)
print(rmsle)
# tensor(0.5102, grad_fn=&lt;SqrtBackward&gt;)
</code></pre>
<p>I hope this helps.</p>

soadrhcp: <p>Thanks mate, indeed this is a direct way to do so.</p>
Reusing nn.Dropout/Relu/Maxpool in forward

AladdinPerzon: <p>From my understanding it’s ok to reuse nn.Relu and nn.Maxpool in the forward because they don’t have trainable parameters. I’m not entirely sure of why this works, perhaps someone can clarify this for me. Let’s say I have</p>
<pre><code class="lang-auto">class NN(nn.Module):
    def __init__(self, input_size, num_classes):
        super(NN, self).__init__()
        self.relu = nn.ReLU()
        self.fc1 = nn.Linear(input_size, 50)
        self.fc2 = nn.Linear(50, 25)
        self.fc3 = nn.Linear(25, num_classes)
    
    def forward(self, x):
        x = self.relu(self.fc1(x))
        x = self.relu(self.fc2(x))
        x = self.fc3(x)
        return x
</code></pre>
<p>How does Pytorch create the graph? I’m thinking if I reuse same nn.ReLU then there would be a problem for Pytorch to remember which gradients to set to 0/1. From my understanding it’s not a good idea to reuse dropout, why is this the case?</p>

AladdinPerzon: <p>Someone that could help me out with this?</p>

ptrblck: <p>You can reuse parameter-less modules, as only the computation will be tracked.<br>
A theoretical example would be reusing <code>nn.Add(a, b)</code> instead of <code>a + b</code>.</p>

AladdinPerzon: <p>That makes sense, but if we have dropout with the same droprate can we reuse that as well?</p>

ptrblck: <p>Yes, that should also work.</p>

AladdinPerzon: <p>That’s good to know, do you know where can I read more about how Pytorch does this backprop or why this works in this way?</p>
<p>Thanks a lot, I appreciate your help!</p>

ptrblck: <p>A general rule might be that all operations without parameters (as in <code>nn.Parameter</code>) are fine to reuse, since Autograd will only track the operation.</p>
<p>Of course you can also reuse modules with parameters, but this could be seen as “weight sharing”.</p>
RuntimeError: invalid argument 2: non-empty vector or matrix expected at /opt/conda/conda-bld/pytorch_1550780889552/work/aten/src/THCUNN/generic/ClassNLLCriterion.cu:32

111215: <p>Experts, What’s the reason for that?<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/e/fe2029f4de73be1641b742de3bc827d6bab926a4.png" data-download-href="https://discuss.pytorch.org/uploads/default/fe2029f4de73be1641b742de3bc827d6bab926a4" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/e/fe2029f4de73be1641b742de3bc827d6bab926a4_2_690x144.png" alt="image" data-base62-sha1="Ag6bFT2hyT8pobQ1c2pLRZ7z2BK" width="690" height="144" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/e/fe2029f4de73be1641b742de3bc827d6bab926a4_2_690x144.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/e/fe2029f4de73be1641b742de3bc827d6bab926a4_2_1035x216.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/e/fe2029f4de73be1641b742de3bc827d6bab926a4_2_1380x288.png 2x" data-dominant-color="EFEBF3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2542×534 142 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Could you check the input shapes of all tensors that you are passing to your criterion?<br>
Apparently some of them are empty. However, the error seems to point to the <code>weight</code> argument.<br>
Are you using weighted cross entropy or are you passing the <code>weight</code> manually to the loss calculation?</p>

Kiran_Ikram: <p>Hi Patrick, Im using transformers to finetune an input sequence of dialogues. I get the same error, but how would I check the input shapes of the tensors ?<br>
Thanks!</p>

ptrblck: <p>You could print the shapes via:</p>
<pre><code class="lang-python">output = model(input)
print(output.shape)
print(target.shape)
loss = criterion(output, target)
</code></pre>
<p>If you are seeing the same error message, the <code>target</code> tensor might be empty.<br>
Feel free to post your training code, so that we can have a look, if you get stuck.</p>
Pytorch with low GPU util

ccffcc: <p>Hi,</p>
<p>I am training an RNN model on GPU, but I find the GPU-util is only as low as 20%. I browsed some of the earlier posts, some of which point out that the Dataloader maybe a bottleneck. However, I preload my whole dataset to GPU, so I guess it’s not a problem, is it?</p>
<p>I am doing hyperparameter tuning. I generate 50 models with different optimizers. During each epoch, I train all the 50 models one after another. I use a dictionary to store the (address of) models and optimizers. Is there a bottleneck in these tasks? I notice my CPU usage is always 100% so there might be a bottleneck on the CPU side but I really didn’t find anything.</p>
<p>Could anyone help me on this problem? Or could anyone suggest any methods to debug?</p>
<p>Many Thanks!</p>

ccffcc: <p>BTW, I am generating a random number in the forward step of the network, which I guess may be using cpu power. Is there a way to generate a random number on GPU?</p>
GPU memory keep increasing

111281: <p>Hello,</p>
<p>I try to train an SSDA model.<br>
The following is part of code for training.</p>
<pre><code class="lang-auto">        data = torch.cat((im_data_s, im_data_t), 0)
        target = torch.cat((gt_labels_s, gt_labels_t), 0)
        output = G(data)
        out1 = F1(output)
        loss = criterion(out1, target)
        loss.backward(retain_graph=True)
        optimizer_g.step()
        optimizer_f.step()
        zero_grad_all()
  
     
        output = G(im_data_tu)  &lt;================ lead to increase gpu memory
        loss_t = adentropy(F1, output, 0.1)
        loss_t.backward()
        optimizer_f.step()
        optimizer_g.step()
       
        G.zero_grad()
        F1.zero_grad()
        zero_grad_all()
</code></pre>
<p>Result shows issue of memory raising:</p>
<p>Step:100<br>
[Memory usage:2938.139136 MB]</p>
<p>Step:200<br>
[Memory usage:3676.467712 MB]</p>
<p>Step:300<br>
[Memory usage:4413.829632 MB]</p>
<p>Not sure what happen, is there any way to fix that problem?</p>
<p>Thanks!</p>

ptrblck: <p>Could you remove <code>retain_graph=True</code> or why is it needed in your code?</p>

111281: <p>Indeed, I don’t need to retain the graph in this case but still confuse why memory keep growing.</p>
<p>For each training step, this <code> loss.backward(retain_graph=True)</code> retain the graph and next</p>
<pre><code class="lang-auto"> output = G(im_data_tu) 
 loss_t = adentropy(F1, output, 0.1)
 loss_t.backward()
</code></pre>
<p>create another graph and these should be the maximum usage of memory. Did increased memory come from the retained graph ? I did comment out <code>output = G(im_data_tu) </code>  and memory usage seem normal. Could you explain that ? Thanks !</p>
Append feature maps to intermediate layers of network

motor_junkie: <p>I have the following network architecture shown below. It takes in input of size 16x8x2048, and gives an output of size 256x128x3. I have a feature map which I wish to append along the channel dimension after every upsampling layer in the below architecture. I would be scaling the feature maps to the appropriate resolution before appending. How can I append these feature maps in the intermediate layers of the network?</p>
<pre><code class="lang-auto">model = []
model += [Conv2dBlock(2048, 256, 3, 1, 1, norm='bn', activation=activ, pad_type=pad_type)]
model +=  [nn.Upsample(scale_factor=2, mode='bilinear')] 
model +=  [Conv2dBlock(256, 128, 3, 1, 1, norm='bn', activation=activ, pad_type=pad_type)]
model +=  [nn.Upsample(scale_factor=2, mode='bilinear')]
model +=  [Conv2dBlock(128, 64, 3, 1, 1, norm='bn', activation=activ, pad_type=pad_type)]                        
model +=  [nn.Upsample(scale_factor=2, mode='bilinear')] 
model +=  [Conv2dBlock(64, 32, 3, 1, 1, norm='bn', activation=activ, pad_type=pad_type)]
model +=  [nn.Upsample(scale_factor=2, mode='bilinear')] 
model +=  [Conv2dBlock(32, 32, 3, 1, 1, norm='bn', activation=activ, pad_type=pad_type)]                        
model += [Conv2dBlock(32, 3, 3, 1, 1, norm='none', activation=activ, pad_type=pad_type)]                        
model = nn.Sequential(*model)
</code></pre>
How does model.eval() affect densenet dropout

Dennis_Hugle: <p>Hey Community,</p>
<p>lets say i want to change the dropout in densenet in the last layers, so i choose layer 10 and 11 and update the respective dropout like shown. As you can see, the dropout remains 0.2 even after model.eval() is called.</p>
<p>Why is that the case?<br>
Is there a more appealing way to change the dropout in the respective layers and turn in on/off when in training/evaluation mode?<br>
Or do I need to run the change_dropout function each time i change between train and evaluation mode?</p>
<p>The output is pretty long, but it shows that layer.drop_rate remains 0.2 in layers 10 and 11 after model.eval() was called.</p>
<p>Output:</p>
<pre><code class="lang-auto">###### BEFORE CHANGES #######
train: 0.0
train: 0.0
... cut to the last to lines ...
train: 0.0
train: 0.0

eval: 0.0
eval: 0.0
... cut to the last to lines ...
eval: 0.0
eval: 0.0

###### AFTERCHANGES #######
train: 0.0
train: 0.0
... cut to the last to lines ...
train: 0.2
train: 0.2

eval: 0.0
eval: 0.0
... cut to the last to lines ...
eval: 0.2
eval: 0.2
</code></pre>
<p>Code: (should run on your machine)</p>
<pre><code class="lang-auto">from torchvision import models
def print_dropouts(model, phase):
    for feature in model.features:
        if type(feature) == models.densenet._DenseBlock:
            for layer in feature.modules():
                if type(layer) == models.densenet._DenseLayer:
                    print(f"{phase}: ", layer.drop_rate)

def change_dropout(model, layers_to_update, drop_rate):
    for idx, feature in enumerate(model.features):
        if (idx in layers_to_update) and (type(feature) == models.densenet._DenseBlock):
            for layer in feature.modules():
                if type(layer) == models.densenet._DenseLayer:
                    layer.drop_rate = drop_rate
# 
if __name__ == "__main__":
    model = models.desnenet121(pretrained=True)
    
    print("###### BEFORE CHANGES #######")
    model.train()
    print_dropouts(model, "train")
    model.eval()
    print_dropouts(model, "eval")

    # change happens here
    change_dropout(model, [10, 11], 0.2)

    print("###### AFTER CHANGES #######")
    model.train()
    print_dropouts(model, "train")
    model.eval()
    print_dropouts(model, "eval")
</code></pre>
<p>Cheers</p>

ptrblck: <p><code>self.drop_rate</code> is an attribute of the <code>_DenseLayer</code> module as seen <a href="https://github.com/pytorch/vision/blob/684f48db4e6f619389da3a6957b3edebf794ae79/torchvision/models/densenet.py#L35">here</a>.<br>
The value won’t change by calling <code>model.train()</code> or <code>model.eval()</code>.<br>
Instead the functional dropout call in <a href="https://github.com/pytorch/vision/blob/684f48db4e6f619389da3a6957b3edebf794ae79/torchvision/models/densenet.py#L88-L89">this line of code</a> will use the <code>self.training</code> attribute to decide if dropout should be used or not.</p>
How do I specify the path to libtorch folder in a camke script?

Jesper.Lindberg: <p>Hello!</p>
<p>How do I set the path for my torch files? When I use <code>find_package(Torch REQUIRED)</code> it rather finds the packages installed by pip in AppData than the folder of libtorch that I have downloaded? I tried setting TORCH_HOME to the folder I downloaded but it did not help?</p>
<p><span class="bbcode-s">So far I have been using libtorch with CPU only. However, it’s time to changed that so I have downloaded the libtorch-cuda package. Following the <a href="https://pytorch.org/cppdocs/installing.html#minimal-example" rel="nofollow noopener">Minimal Example</a>-guide works fine and I threw in:</span></p>
<p><span class="bbcode-s"> if (torch::cuda::is_available()) {<br>
std::cout &lt;&lt; “CUDA available! Training on GPU.” &lt;&lt; std::endl;<br>
} else {<br>
std::cout &lt;&lt; “Training on CPU.” &lt;&lt; std::endl;</span></p>
<p><span class="bbcode-s">which confirms that cuda is working.</span></p>
<p><span class="bbcode-s">Now I want to implement this in my project, the current way I build torch with my C++ project is that I:</span><br>
<span class="bbcode-s"><code>set TORCH_LIBRARIES=C:\Users\a3772\Documents\libtorch\include</code></span></p>
<p><span class="bbcode-s">and then in my project cmake-file I use <code>find_package(Torch REQUIRED)</code>. However when I later check if cuda is available in my project (after rebuilding), its not?..</span></p>
<p><span class="bbcode-s">It seems like that <code>find_package(Torch REQUIRED)</code> finds the one I installed with pip half a year ago?.. </span></p>

Jesper.Lindberg: <p>Stupid me forgot to set <code>set CMAKE_PREFIX_PATH=%TORCH_HOME%</code> this early in the morning.</p>