akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Conda packaging with CUDA enabled / CPU only

username: Haydnspass
cooked: <p>Hi there,</p>
<p>I am working on a project that heavily relies on PyTorch which I now want to deploy as a conda package.<br>
I have some custom C++ and CUDA (equivalent but faster) code which I also want to include in my conda build.</p>
<p>At the moment everything works so far, but I haven’t been able to implement the cuda / cpuonly logic which pytorch uses. So my idea would be to just reuse the pytorch logic</p>
<pre><code class="lang-auto">conda install [myownbuild] cudatoolkit=10.1 -c [mychannel]
conda install [myownbuild] cpuonly -c [mychannel]
</code></pre>
<p>such that when pytorch is installed with the respective cudatoolkit I would want to use the cuda version of my own build and when the cpu only flag is used, then I would want to use the CPU only version of my build as well.</p>
<p>But how can I handle this in the conda meta.yaml?</p>
<p>Currently, I have a simple Linux switch that looks like</p>
<pre><code class="lang-auto">...
  host:
    - python {{ python }}
    - numpy {{ numpy }}
    - pybind11 &gt;=2.4
    - cudatoolkit-dev # [linux]
...
</code></pre>
<p>and kind of assumes that on the linux machine there is a cuda card. Is it possible to use the cpuonly / cudatoolkit flag somewhere there? Or could some tell me how this is done in pytorch?</p>
<p>Cheers,<br>
Lucas</p>

username: ptrblck
cooked: <p>Check out <a href="https://github.com/pytorch/builder">this repository</a>, which is also used to build the binaries.</p>
How to handle batch_size when using linear layer?

username: MehdiZouitine
cooked: <p>Hi everybody ! I’m currently building LSTM auto_encoder and i use linear layer.<br>
Linear layer need a flat vector in input but i got a batch_size so i dont know how to handle it. Let’s see the code to be more precise</p>
<p>In <strong>init</strong></p>
<pre><code class="lang-auto">INPUT_FC = self.sequence_len * self.hidden_dim

        self.fc = nn.Sequential(
            nn.Linear(in_features=INPUT_FC, out_features=self.hidden_dim // 2),
            nn.ReLU(inplace=False),
            nn.Linear(in_features=self.hidden_dim // 2, out_features=self.latent_dim),
        )
</code></pre>
<p>in forward method :</p>
<pre><code class="lang-auto"># SIZE(output) = (seq_len, batch_size, num_directions * hidden_dim) if batch_first=False
        # SIZE(output) = (batch_size,seq_len, num_directions * hidden_dim) if batch_first=True
        # here num_directions = 1 because it's times series (times is ordonned)

        # SIZE(h_embeded) = SIZE(c_embeded) = (batch_size,hidden_dim)

        # output = output.view(-1)
        # fc need a flat vector
        output = output.view(1, output.size(1) * output.size(2))

        lattent_vector = self.fc(output)
        # fc take seq_len*hidden_dim
</code></pre>
<p>Thanks you !</p>

username: ptrblck
cooked: <p><code>nn.Linear</code> expects the input to have the shape <code>[batch_size, *, nb_features]</code>, the tensor should not be completely flattened to a 1-dim tensor.</p>
<p>Usually you would use <code>out = out.view(out.size(0), -1)</code> before feeding the activations to the linear layer.<br>
However, I’m currently unsure, if you are using <code>batch_first=True</code> or not, so you might need to permute the dimensions before.</p>
Concatenate two Sequential blocks?

username: xksteven
cooked: <p>If I have two nn.Sequential blocks how can I make a new nn.Sequential block that is the concatenation of both of them?</p>
<p>block1 = nn.Sequential(stuff)<br>
block2 = nn.Sequential(other_stuff)</p>
<p>block3 = nn.Sequential (block1,block2) ?? &lt;- like this?</p>
<p>Assume that block1 can feed directly into block2.</p>

username: SpandanMadan
cooked: <p>Haven’t tried this, but should work.</p>
<pre><code>list_of_layers = list(block1.children())
list_of_layers.extend(list(block2.children()))
block3 = nn.Sequential (*list_of_layers)
</code></pre>
<p>Hope it helps!</p>

username: allenye0119
cooked: <p>Since <code>nn.Sequential</code> is a subclass of <code>nn.Module</code>, you can concatenate multiple instances of <code>nn.Sequential</code> as you described.</p>

username: xksteven
cooked: <p><a class="mention" href="/u/allenye0119">@allenye0119</a> Thanks!</p>

username: Neeraj_Kulkarni
cooked: <p>That worked, thank you!</p>

username: ThaiThien
cooked: <p>I don’t understand. Can you put some example code ?</p>

username: 3yanlis1bos
cooked: <p>Hi <a class="mention" href="/u/thaithien">@ThaiThien</a>,</p>
<p>Lets consider this model from <a href="https://github.com/leeyeehoo/CSRNet-pytorch" rel="nofollow noopener">https://github.com/leeyeehoo/CSRNet-pytorch</a></p>
<p>which has a forward function as following</p>
<pre><code class="lang-auto">def forward(self,x):
        x = self.frontend(x)
        x = self.backend(x)
        x = self.output_layer(x)
        return x
</code></pre>
<p>You can concatenate these three blocks (last one is a single layer for output) as below:</p>
<pre><code class="lang-auto">model_cat = torch.nn.Sequential(*model.frontend.children(),
                                *model.backend.children(),
                                model.output_layer)
</code></pre>
How to estimate necessary (GPU) memory for a model

username: ljj7975
cooked: <p>I have a model that throws <code>not enough memory on GPU</code> error upon training (coming from <code>.forward()</code>).</p>
<pre><code class="lang-auto">File "/home/.../.../run/train.py", line 131, in main
  output = model(data)
File "/home/.../anaconda3/envs/.../lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
  result = self.forward(*input, **kwargs)
File "/home/.../.../model/resnet.py", line 47, in forward
  x = self.layers[f"conv_{i}"](x)
File "/home/.../anaconda3/envs/.../lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
  result = self.forward(*input, **kwargs)
File "/home/.../anaconda3/envs/.../lib/python3.7/site-packages/torch/nn/modules/conv.py", line 345, in forward
  return self.conv2d_forward(input, self.weight)
File "/home/.../anaconda3/envs/.../lib/python3.7/site-packages/torch/nn/modules/conv.py", line 342, in conv2d_forward
  self.padding, self.dilation, self.groups)
RuntimeError: CUDA out of memory. Tried to allocate 14.00 MiB (GPU 0; 7.93 GiB total capacity; 7.35 GiB already allocated; 12.75 MiB free; 87.83 MiB cached)
</code></pre>
<p>When I checked the model size based on parameters it definitely fits into the memory and each batch size is also quite small that these cannot be the source of the exception.</p>
<p>(I was told from my friend that it might be coming from the activation tensors that are stored for the backward pass.)</p>
<p>In this post, I would like to ask the following questions.</p>
<ol>
<li>What does PyTorch allocate memory for other than model and data (especially during the training process)? I would like to know the exact cause of the exception.</li>
<li>Is there any way of estimating how much memory the model requires “prior to training” and “programmatically”?</li>
</ol>
<p>When I use term <em>memory</em>, it can simply be the number of float (tensor) because I can always estimate one metric from the other</p>
<p>Thank you for your time</p>

username: divyesh_rajpura
cooked: <p>I am not sure about the inbuilt memory usage by pytorch library. There is one library called <a href="https://pypi.org/project/pytorch-memlab/" rel="nofollow noopener">pytorch_memlab</a>, which can be used to inspect the GPU memory usage by each line of your code.</p>

username: ljj7975
cooked: <p>It looks like I have to call backward() to check the usage though…</p>
Is my timeDistributed Layer doing the same thing as a convolutional layer with a (1,1) kernal?

username: Joshua_Clancy
cooked: <p>So after using multiple convolutional layers on an image, I end up with (b, c, x, y). I am now wanting each of my spatial units with their channels to go through their own separate linear layers.</p>
<p>I believe I have accomplished this with a time-distributed layer via this code: (Please excuse my coding verbosity)</p>
<pre><code class="lang-auto">class TimeDistributed(nn.Module):
    def __init__(self, module):
        super(TimeDistributed, self).__init__()
        self.module = module

    def forward(self, x):
        # input shape =  batch, channel, h, w
        # transform into batch, timestep, channel
        sh = x.size()
        #isolate batch, channel, x and y sizes
        b = sh[0]
        c = sh[1]
        x = sh[2]
        y = sh[3]
        #move around and reshape
        x = x.permute(0,2,3,1).contiguous().view(b, x*y, c)
        #put it through the neural module. 
        x = self.module(x)
        # Reshape it back
        x = x.permute(0, 2, 1).contiguous()
        x = x.view((b, :, x, y)) 
        return x
</code></pre>
<p>With the neural module being a series of linear layers. I believe this should work as linear layers can take 3d input. (correct me if I am wrong).</p>
<p>Now that I have done this, however, I am wondering whether I could have accomplished the same thing via a conv layer with a (1,1) kernel. Is my thinking there correct? Are there any advantages/disadvantages between the two techniques?</p>

username: Joshua_Clancy
cooked: <p>Let future me answer past me: The answer is yes. A 1 by 1 convolution is essentially the same as doing a time distributed layer over the spatial dimensions. If you are in this situation I suggest the 1by1 conv because it is more efficient.</p>
Having trouble adding tensor core support to pytorch

username: deama
cooked: <p>So I’ve been using this repo for some fps interpolation which works pretty well. I recently got a new GPU (2060super) to speed it up, which it did, however I recently found it that the tensor cores aren’t being used and that you have to activate them somehow?</p>
<p>I went online and found out that you need to use some sort of Nvidia/Apex extension for pytorch to be able to use those tensor cores, however even though on their repo they say that you only have to add 3 lines of code, I am completely confused on where to begin. Could anyone please help me?</p>
<p>Here’s the repo I am talking about: <a href="https://github.com/MortenHannemose/pytorch-vfi-cft" rel="nofollow noopener">https://github.com/MortenHannemose/pytorch-vfi-cft</a><br>
Here’s the nvidia/apex repo: <a href="https://github.com/NVIDIA/apex" rel="nofollow noopener">https://github.com/NVIDIA/apex</a></p>
What does pytorch use under the hood for cpu?

username: ahsabali
cooked: <p>As pytorch uses cudnn under the hood for gpu, what does pytorch use under the hood for cpu?</p>

username: albanD
cooked: <p>Hi,</p>
<p>We have our own implementations for most things just like for cuda. And when possible, we use optimized libraries like cudnn or magma on GPU and mkldnn or mkl/openblas on CPU.</p>

username: ahsabali
cooked: <p>Thank you for your reply. This is helpful.</p>
<p>Is there a commonly preferred default choice out of these options for pytorch operations on CPU?</p>

username: albanD
cooked: <p>The default choice in the binaries we provide is mkl for CPU backend.</p>
Ignite metrics (recall, precision, etc) - thresholded_output_transform

username: xen0f0n
cooked: <p>I’m working on a binary task and I’m figuring out Recall, Precision, etc, as implemented by Ignite. As per <a href="https://pytorch.org/ignite/_modules/ignite/metrics/recall.html" rel="nofollow noopener">documentation</a>:</p>
<p><em>“In binary and multilabel cases, the elements of <code>y</code> and <code>y_pred</code> should have 0 or 1 values. Thresholding of predictions can be done as below:”</em></p>
<pre><code class="lang-auto">def thresholded_output_transform(output):
            y_pred, y = output
            y_pred = torch.round(y_pred)
            return y_pred, y
</code></pre>
<p>Am I correct to assume that  <code>output</code>  is what is returned by the model?<br>
I’m using BCEWithLogitsLoss, so I have to pass logits through a sigmoid function. I have to use a sigmoid here as well, right?</p>
<pre><code class="lang-auto">def thresholded_output_transform(output):
            y_pred, y = output 
            y_pred = torch.sigmoid(y_pred)
            y_pred = torch.clamp(y_pred, 0., 1.)
            y_pred = torch.round(y_pred)
            return y_pred, y
</code></pre>

username: vfdev-5
cooked: <p>Output is what is returned by evaluator’s <code>process_function</code> :  <a href="https://pytorch.org/ignite/metrics.html" class="inline-onebox" rel="noopener nofollow ugc">ignite.metrics — PyTorch-Ignite v0.4.13 Documentation</a></p>
<blockquote>
<p>I’m using BCEWithLogitsLoss, so I have to pass logits through a sigmoid function. I have to use a sigmoid here as well, right?</p>
</blockquote>
<p>Well, it depends on your choice how to define whether logit’s value is sufficient to be interpreted as 1.<br>
Yes, it is more convenient to pass to “probabilties” using sigmoid and then apply a thresholding (0.5) as you put to your version of <code>thresholded_output_transform</code>.</p>
<p>Here is another way to apply the thresholding:</p>
<pre data-code-wrap="python"><code class="lang-python">t = 0.55

def thresholded_output_transform(output):
    y_pred, y = output 
    y_pred = torch.sigmoid(y_pred)
    y_pred = (y_pred &gt; t).float()
    return y_pred, y
</code></pre>

username: xen0f0n
cooked: <p><a class="mention" href="/u/vfdev-5">@vfdev-5</a> nice! Thanks!</p>
Customizing activation backward() with variable threshhold value?

username: satubus
cooked: <p>Hey there,</p>
<p>first of all nice Framework!<br>
To the core problem, I got a non differentiable activation function at 0 which depends on a threshold value. The activation function maps tensor values onto 0 if their absolute value is below threshold alpha.<br>
I have implemented a custom class inheriting torch.Function with both static methods forward() and backward() both depend on alpha.</p>
<p>Since I want to run multiple experiments with different alpha values I’m looking for a convenient way to just pass the threshhold value and modify it. Any ideas?</p>
<p>Thanks in advance</p>
What is the possible reason for showing up a constant frequently in prediction?

username: Ereshkigal
cooked: <p>first i’m a Deep learning newbee and i’m not a english speaker,so if i say something strange please just tell me. i’m using LSTM to make a prediction about time series. When this problem occurs, I also search about it in many places and find some possible reasons, such as normalization. However, these solutions dont work at all . so I came to here for advice.</p>
<p>Problem symptoms: a constant appears frequently in prediction. If it ran more epochs, all prediction could be the same. This constant may change with the modification of the model parameters, but it will not change without modification of the parameters; in a single training, a constant will start to have a large frequency, then it will change to other constants with the training and remain unchanged at last.</p>
<p>General process: first, label the problem classification number and convert the data in the data set to the format of label sequence, then divide the sequence into ten sliding windows (v0 to V9 to predict V10; V1 to V10 to predict V11 and so on), put it into LSTM, embed it with torch.nn.embedding, then process it with softmax, output it, send it to cross entropy function to calculate loss . finally use the output and real value to Post back propagation.</p>
<p>Maybe there are some problems in terms and expressions. Please forgive me and give me some advice.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/e/be512843f5df9df2c86270f4eaf827a0c9da17f0.png" data-download-href="https://discuss.pytorch.org/uploads/default/be512843f5df9df2c86270f4eaf827a0c9da17f0" title="20200327000134"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/e/be512843f5df9df2c86270f4eaf827a0c9da17f0.png" alt="20200327000134" data-base62-sha1="r9CB4Z0RWU8MAO6i1nDHnXd26SA" width="641" height="500" data-dominant-color="3D464F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">20200327000134</span><span class="informations">728×567 23.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
How to apply "in" to tensor element-wise?

username: ab-10
cooked: <p>Is there an efficient way to apply Python’s <code>in</code> (<code>__contains__</code>)  elementwise to a <code>torch.tensor</code>?</p>
<p>This is the behaviour I’m looking for:</p>
<pre><code class="lang-auto">&gt;&gt; some_values = [3, 5, 8]
&gt;&gt; m = torch.randint(0, 10, (2,2))
&gt;&gt; m
[[0, 9],
[8, 5]]
&gt;&gt; m in some_values
[[False, False],
 [True, True]]
</code></pre>
<p>Thanks in advance!</p>

username: richard
cooked: <p><code>(m == some_values[0]) | (m == some_values[1]) | (m == some_values[2])</code> is one way to do it. I’m not sure how efficient it is.</p>

username: ab-10
cooked: <p>I ended up doing it this way:</p>
<p><code>r</code> # of rows in matrix<br>
<code>c</code> # of columns in matrix<br>
<code>n</code> # of elements in the collection you’re comparing to</p>
<pre><code class="lang-auto">m = torch.randint(0, 10, (r, c))
some_values = torch.randint(0, 10, (n))

new_m = m.unsqueeze(2).expand(r, c, n)

new_val = some_values.repeat(r, c).view(r, c, n)
mask = torch.where(new_m == new_val, torch.ones(r, c, n), torch.zeros(r, c, n))
output = mask.sum(2)
</code></pre>
<p>I’m unsure about how efficient it is as compared to your method though.</p>
Get data is slow

username: zpmmehrdad
cooked: <p>Hi everyone,</p>
<p>This is my part of my code but I have a problem that when I want to get data it takes ~90ms. where is the problem?</p>
<p>at::Tensor image_tensor = torch::from_blob(<br>
input_preproc.data, { 1, _img_h, _img_w, _img_d }, at::kFloat);</p>
<p>image_tensor = image_tensor.to(*_device);</p>
<p>image_tensor = image_tensor.permute({ 0, 3, 1, 2 });</p>
<p>image_tensor[0][2] =<br>
image_tensor[0][0].div(255).sub(_img_means[0]).div(_img_stds[0]);<br>
image_tensor[0][1] =<br>
image_tensor[0][1].div(255).sub(_img_means[1]).div(_img_stds[1]);<br>
image_tensor[0][0] =<br>
image_tensor[0][2].div(255).sub(_img_means[2]).div(_img_stds[2]);</p>
<p>std::vector<a>torch::jit::IValue</a> inputs;<br>
inputs.push_back(image_tensor);<br>
torch::Tensor logits_tensor = _module-&gt;forward(inputs).toTensor();<br>
torch::Tensor argmax_tensor = logits_tensor.argmax(1, false);</p>
<p>argmax_tensor = argmax_tensor.toType(at::kInt);</p>
<hr>
<p>// it takes about ~90ms<br>
auto begin = std::chrono::high_resolution_clock::now();<br>
<strong>argmax_tensor = argmax_tensor.to(torch::kCPU);</strong><br>
auto end = std::chrono::high_resolution_clock::now();</p>
<p>OS: Win 10<br>
GPU: GTX 1080Ti<br>
CUDA: 10.1</p>

username: albanD
cooked: <p>Hi,</p>
<p>Keep in mind that the cuda api is asynchronous. And only wait for ops to actually run when the new value is needed outside of the GPU. So here the copy will have to wait for all the ops above to finish before running. You want to synchronize the cuda device before gettting timings if you want to see the runtime of a single op.</p>

username: zpmmehrdad
cooked: <p>Thanks for the reply, Yes I know that but time on Ubuntu is less than 30. Should I change the code in Windows?</p>

username: albanD
cooked: <p>You should not rely on any of these timings if you don’t do proper sync. It is most likely due to a slower GPU or a faster CPU on your windows machine.</p>
Chunk + cat data assignment does not propogate

username: preddy5
cooked: <p>Hello,<br>
I am breaking a tensor into chunks so that I can individually change the requires_grad argument based on which values I want to optimize</p>
<pre><code class="lang-auto">ones = torch.ones(3)
chunks = torch.chunk(ones, 3, 0)
# using chunks as input to optimizer
k = torch.cat(chunks)
print(k)
Output: tensor([1., 1., 1.])
k[0]=2
print(k)
Output: tensor([2., 1., 1.])

print(chunks)
Output: (tensor([1.]), tensor([1.]), tensor([1.]))
</code></pre>
<p>I am doing this so that I don’t have to change most of my code, so I am first using the chunk function to break the tensor and then using cat function. Changing the values of variable ‘k’ doesn’t seem to reflect on variable ‘chunks’. I don’t understand where the flaw in my intuition is.</p>
<p>Thank you.</p>

username: richard
cooked: <p><code>torch.chunk</code> creates views of the original <code>ones</code> tensor. <code>torch.cat</code> does not create a view; it creates a branch new tensor by copying data. This is why modifying the output of <code>torch.cat</code> (k) doesn’t change <code>chunks</code>; their data storage is completely unrelated.</p>

username: preddy5
cooked: <p>Thank for the reply <a class="mention" href="/u/richard">@richard</a><br>
I am trying to understand the difference better</p>
<pre><code class="lang-auto">x = torch.cat([x1, x2, x3])
out = f(x)
</code></pre>
<p>would the gradients propage to x1, x2, x3 as expected or do they not get any gradients.</p>
<p>Also is there a good way of merging all the chunks together, so that I can perform operations on them as if they are a single tensor?</p>
<p>Thank you.</p>

username: richard
cooked: <p>The gradients will propagate as long as <code>x1, x2, ...</code> require grad.</p>
<blockquote>
<p>Also is there a good way of merging all the chunks together, so that I can perform operations on them as if they are a single tensor?</p>
</blockquote>
<p>You could use <code>x = torch.cat([x1, x2, x3])</code>, apply some in-place operations, and then split it back up via <code>y1, y2, y3 = x.chunk(3)</code>.</p>
Question about tensor's index

username: disguise
cooked: <p>I run the code:</p>
<pre><code class="lang-python">y = t[0, :] 
y += 1
print(y)
print(t[0, :]) 
</code></pre>
<p>the result is :</p>
<pre><code class="lang-python">tensor([5, 9, 9])
tensor([5, 9, 9])
</code></pre>
<p><code>y</code> and <code>t[0,:]</code> share memory. But after I changed the index, I run the new code:</p>
<pre><code class="lang-python">y = t[0, [1, 2]]
y += 1
print(y)
print(t[0, [1, 2]]) 
</code></pre>
<p>the result is:</p>
<pre><code class="lang-python">tensor([10, 10])
tensor([9, 9])
</code></pre>
<p><code>y</code> and <code>t[0, [1, 2]</code> don’t share memory。Do you know the reason? Any help would be appreciated, thank you.</p>

username: albanD
cooked: <p>Hi,</p>
<p>This is because advanced indexing methods (fancy things in brackets) try to return view but they cannot always do it. So you should not rely on it returning the same memory.</p>
<p>If you just want to change this part of t, you can use <code>t[0, [1, 2]] += 1</code> that will change t inplace properly.</p>
How to delete loaded model from torch.hub

username: Vitalik
cooked: <p>Hi, how can I find the directory where pre-trained model was loaded?</p>
<p>loaded like this:</p>
<pre><code class="lang-auto">en2ru = torch.hub.load('pytorch/fairseq', 'transformer.wmt19.en-ru.single_model', tokenizer='moses', bpe='fastbpe')
</code></pre>

username: yash1994
cooked: <p>Hi,</p>
<p>You find it at <code>.cache/torch/</code> in your home directory (for linux based systems). You can find documentation <a href="https://pytorch.org/docs/stable/hub.html" rel="nofollow noopener">here</a>: <code>default path is $TORCH_HOME/hub where environment variable $TORCH_HOME defaults to $XDG_CACHE_HOME/torch. $XDG_CACHE_HOME follows the X Design Group specification of the Linux filesytem layout, with a default value ~/.cache if the environment variable is not set.</code></p>

username: Vitalik
cooked: <p>Thanks! I found this directory, but I didn’t found the model there, so I removed directories, that had fairseq in the name <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to remove backbone network parametrs from model.parameters?

username: neouyghur
cooked: <p>Hi, my model includes a pre-trained network. Now, I am trying to use different learning rate for the backbone network. I can get parameters of the backbone network easily, but how can I get the rest of the parameters. I tried to use remove backbone network parameters from <code>model.parameters</code> but failed. Could you show me how to do that? Thanks.</p>

username: albanD
cooked: <p>Hi,</p>
<p>One way is to “hide” you backbone from the discovery by putting it in a data structure: <code>self.backbone = [my_backbone]</code>. And since <code>.parameters()</code> does not look into arbitrary data structure it won’t be found.</p>
<p>The other way if you don’t want the weights to be parameters anymore is to make sure the weights are not <code>nn.Parameter()</code> anymore. You can do so by:</p>
<pre><code class="lang-auto">weight_tensor = self.weight.detach() # .detach() to extract the Tensor out of the nn.Parameter
del self.weight
self.weight = weight_tensor
</code></pre>

username: neouyghur
cooked: <p><a class="mention" href="/u/alband">@albanD</a> Thanks for your answer. Can I create new parameter list via subtracting model.backbone.parameters from model.parameters through a for loop based comparison?</p>

username: albanD
cooked: <p>They contain the same objects so you can yes: <code>[p for p in model.parameters() if p not in model.backbone.parameters()]</code> or somthing similar.</p>
nn.Parameter inside class, but the module doesn't seem to contain the nn.Parameters

username: barakb
cooked: <p>I’m creating a class, details are not so important just the fact that in init there are conv layers and nn.Parameters:</p>
<pre><code class="lang-auto">class MyClass(nn.Module):
    def __init__(self, args):
        super(MyClass, self).__init__()
        nc = 100
        self.teta0 = nn.Parameter(torch.cuda.FloatTensor(1,nc),requires_grad=True)
        self.teta1 = nn.Parameter(torch.cuda.FloatTensor(1,nc),requires_grad=True)
        self.teta2 = nn.Parameter(torch.cuda.FloatTensor(1,nc),requires_grad=True)
        self.conv  = nn.Conv2d(1,3,3)
    def forward(self,a,b): 
        first  = self.teta2*(a-self.teta1)
         ....
        return loss
</code></pre>
<p>Now I’m instantiating the class and print</p>
<pre><code class="lang-auto">        self.model= MyClass(args)
        print(self.model)
</code></pre>
<p>I’m getting:</p>
<blockquote>
<p>MyClass (<br>
(conv): Conv2d(1, 3, kernel_size=(3, 3), stride=(1, 1))<br>
)</p>
</blockquote>
<p>It’s seems like the class is not consisting of those nn.Parameters.<br>
Just to makes things clear, self.conv is there because I had this problem and I wanted to see if the conv layer can be seen.<br>
Any idea what’s going on, where I’m mistaking and what can lead to such an error?<br>
Thanks!</p>

username: charan_Vjy
cooked: <p>Conv2d is a module.  In the source for nn.Conv2d, Conv2d inherits from _ConvNd which in turn inherits from Module. On the other hand, nn.Parameter inherits from torch.Tensor. From what I understand, this implies that the parameters, although learnable, are not part of the Model to begin with and any operation involving it is dynamically computed. This is why initializing  torch.ones in the <strong>init</strong> section does not add that variable to the model.</p>
Performance of model depends greatly on intial seed

username: JanoschMenke
cooked: <p>Hi,</p>
<p>so I am training a model and the  performance varies considerably when I use a different seed. I assume its is due to the weight initiation, so I was wondering what I could do to combat that issue.<br>
I did a hyper-parameter optimization and that is why I feel the learning rate is appropriate.</p>
<p>Maybe there are ways to change the weight initialization, below is the way I do it at the moment.<br>
The weight and weight-skip matrices are of same size.<br>
If you need more information I am happy to provide them</p>
<pre><code>def reset_parameters(self):
    stdv = 1. / math.sqrt(self.weight.size(1))
    self.weight.data.uniform_(0, stdv)
    
    stdv = 1. / math.sqrt(self.weightSkip.size(1))
    self.weightSkip.data.uniform_(0, stdv)
    
    if self.bias is not None:
        self.bias.data.uniform_(-stdv, stdv)</code></pre>

username: charan_Vjy
cooked: <ol>
<li>What is the different performance ranges, with and without initializations?</li>
<li>What type of models are you using? Generative models, say GAN’s, are very sensitive to initializations.</li>
<li>Empirically, I have observed that the using Batch Norm Layers tend to ease the dependency on a good initialization.</li>
<li>Xavier’s initialization can be used to prevent the exploding and vanishing gradients problem.</li>
</ol>

username: JanoschMenke
cooked: <p>Hi,</p>
<ol>
<li>
<p>I am not sure what is meant without initialization. But for example with one seed I get an AUC (BCE Loss of 0.48) of 0.88 and with a different one an AUC (BCE Loss of 0.41) of 0.91. So the loss is actually more concerning to me than the AUC.</p>
</li>
<li>
<p>It a Graph Convolution Network</p>
</li>
<li>
<p>That could be indeed a issue because I removed for some layers the batch-norm as it provided better results.</p>
</li>
<li>
<p>I will have a look at the Xavier init</p>
</li>
</ol>
<p>Thanks for the reply</p>
Difference between Tensor.clone() and Tensor.new_tensor()

username: LongPham
cooked: <p>Can you please explain a difference between Tensor.clone() and Tensor.new_tensor()? According to the documentation, Tensor.new_tensor(x) = x.clone().detach(). Additionally, according to <a href="https://discuss.pytorch.org/t/how-to-copy-a-variable-in-a-network-graph/1603">this post on the PyTorch forum</a> and <a href="https://pytorch.org/docs/stable/tensors.html" rel="nofollow noopener">this documentation page</a>, x.clone() still maintains a connection with the computation graph of the original tensor (namely x). However, I am new to PyTorch and don’t quite understand how x.clone() interacts with the computation graph of x.</p>

username: vmirly1
cooked: <p>So, s you said, <code>x.clone()</code> maintains the connection with the computation graph. That means, if you use the new cloned tensor, and derive the loss from the new one, the gradients of that loss can be computed all the way back even beyond the point where the new tensor was created. However, if you detach the new tensor, as it is done in the case of <code>.new_tensor()</code>, then the gradients will only be computed from loss backward up to that new tensor but not further than that.</p>
<p>I hope this helps!</p>

username: LongPham
cooked: <p>I see. It makes a lot of sense. Thank you very much.</p>

username: HAORAN_LI
cooked: <p>Hi, Vahid! Say if have a tensor ‘A’, then I use <code>B=A.clone()</code>; finally, I get my <code>loss=Lossfunc([g(B);A])</code> where ‘;’ denotes concatenation. Do the gradients of A include two parts where the first part is directly from the A in the concatenation and the second part is from B? The function <code>g(.)</code> will not affect A. So basically I mean there is a residual style connection in the computation graph. Is that right?</p>
Generate caffe2 LMDB file for imagenet 

username: chiang
cooked: <p>Hi there,</p>
<p>I want to generate caffe2 LMDB for imagenet. I follow the advice in <a href="https://github.com/facebookarchive/caffe2/issues/1755" rel="nofollow noopener">this thread</a>. Unfortunately, I cannot locate where the make_image_db is as I install pytorch 1.4 using conda as recommended on your website. I tried to build from source code but no luck at all. I wonder how I could use the script for the generation with conda installed pytorch?</p>
<p>Best wishes,</p>
<p>Chiang</p>
Connecting two models

username: mk_sherwani
cooked: <p>I am working on MR images to synthetic CT and my results are not very satisfactory based on the output. I am new to pytorch so I don’t have experience in it.</p>
<p>Now, I am planning to merge two models.</p>
<p>1st model that I was using to do my research takes the input MR and CT images to train the network and once trained it provide me the synthetic CT. Model is trained in a way that it should not loose the HU values while reconstructing the images because HU is the most important part for the medical CT images.</p>
<p>Now I am planning to implement super resolution over this network.</p>
<p>I would like to hear the suggestion from the experts that should I take the output of my network to the input of super resolution model to get much improved result or there can be some other better solution.</p>
What is proxy loss

username: lycaenidae
cooked: <p>What is the meaning of the ‘proxy loss’?</p>

username: ptrblck
cooked: <p>Where did you encounter this term and in which context?<br>
In the terms of ML/DL I would claim a proxy loss is a loss, which might be used instead of the “real” loss, as it’s either simpler to optimizer or implementable.<br>
E.g. instead of using an absolute loss function, you could use <code>nn.MSELoss</code> as a proxy, as it might be easier to train with it.</p>
<p>Let me know, if you’ve seen the term in another context.</p>

username: lycaenidae
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="74121">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Let me know, if you’ve seen the term in another context.</p>
</blockquote>
</aside>
<p>This is the explanation that I am looking for, it was somehow confused when I encountered for the first time in several papers. Now it makes sense, thanks a lot:)</p>
Result of trace() and script() produce the same output while they should be different

username: minhduc0711
cooked: <ol>
<li>Traced module</li>
</ol>
<pre><code class="lang-python">torch.manual_seed(25)
module = torch.jit.trace(net, torch.randn(1,3,5,5))
print(module.code)
module(torch.randn(1,3,5,5))[0, 0, 0]
</code></pre>
<p>And the output</p>
<pre><code class="lang-python"># module.code
def forward(self,
    input: Tensor) -&gt; Tensor:
  _0 = getattr(self.convs, "1")
  _1 = (getattr(self.convs, "0")).forward(input, )
  _2 = getattr(self.residuals, "0")
  _3 = (getattr(self.convs, "2")).forward((_0).forward(_1, ), )
  _4 = getattr(self.residuals, "2")
  _5 = (getattr(self.residuals, "1")).forward((_2).forward(_3, ), )
  _6 = getattr(self.upsample_convs, "1")
  _7 = (getattr(self.upsample_convs, "0")).forward((_4).forward(_5, ), )
  _8 = (self.final_conv).forward((_6).forward(_7, ), )
  return torch.sigmoid(_8)
# Output
tensor([0.3275, 0.4366, 0.2979, 0.2537, 0.4489, 0.4663, 0.4455, 0.4363],
       grad_fn=&lt;SelectBackward&gt;)
</code></pre>
<ol start="2">
<li>Scripted module: I only changed <code>torch.jit.trace()</code> to <code>torch.jit.script()</code>
</li>
</ol>
<pre><code class="lang-python"># module.code
def forward(self,
    x: Tensor) -&gt; Tensor:
  _0 = self.convs
  _1 = getattr(_0, "0")
  _2 = getattr(_0, "1")
  _3 = getattr(_0, "2")
  x0 = (_1).forward(x, )
  x1 = (_2).forward(x0, )
  x2 = (_3).forward(x1, )
  _4 = self.residuals
  _5 = getattr(_4, "0")
  _6 = getattr(_4, "1")
  _7 = getattr(_4, "2")
  x3 = (_5).forward(x2, )
  x4 = (_6).forward(x3, )
  x5 = (_7).forward(x4, )
  _8 = self.upsample_convs
  _9 = getattr(_8, "0")
  _10 = getattr(_8, "1")
  x6 = (_9).forward(x5, )
  x7 = (_10).forward(x6, )
  x8 = (self.final_conv).forward(x7, )
  return torch.sigmoid(x8)
# Output
tensor([0.3275, 0.4366, 0.2979, 0.2537, 0.4489, 0.4663, 0.4455, 0.4363],
       grad_fn=&lt;SelectBackward&gt;)
</code></pre>
<p>You can see the results <code>module.code</code> is totally different, as my <code>forward()</code> contains loops, but the module’s output is exactly the same.<br>
Why is this the case? Thanks in advance!</p>

username: driazati
cooked: <p>The two don’t necessarily produce the same IR (which is what you’re seeing, it corresponds directly to <code>module.graph</code>) since tracing only records tensor operations as they occur, while scripting will compile the code itself. Tracing will unroll loops based on your example input, so this is probably why you’re seeing the same results.</p>
<p>But if your model has no data dependent control flow (meaning that tracing will generate correct results), the output from the two will be identical (and it will also match the output from eager (no jit) PyTorch)</p>

username: minhduc0711
cooked: <p>you’re right, my model has no data dependent control flow, as I was only looping over a <code>nn.ModuleList</code></p>
Why we need collate_fn()?

username: silence
cooked: <p>The input of collate_fn is a list of training examples which are returned by <strong>getitem</strong>. Why we process data in collate_fn instead of <strong>getitem</strong>? I don’t know what the difference is.</p>

username: ptrblck
cooked: <p>You could use a custom <code>collate_fn</code> to create batches containing tensors with e.g. different shapes.<br>
The default collate function creates a batch by stacking the tensors in dim0. However, this is not possible, if the spatial size of each tensor is different, so that a custom collate function gives you more flexibility to create the batches for your use case.</p>

username: silence
cooked: <p>Yes, i have read some answers about that. But what confuse me is why not limit the tensor shape in <code>__getitem__()</code>.  For example, i can reshape a tensor by truncating or padding in <code>__getitem__()</code> to meet requirements. Maybe collate_fn is more flexible, but is there any further difference?</p>

username: ptrblck
cooked: <p>Creating tensors with the same shape would be the default approach and thus the default collate function will just work.<br>
However, a custom collate function allows you to work with more flexible approaches.<br>
Not providing this method would limit some use cases.</p>

username: silence
cooked: <p>Well, i haven’t met such a condition. Maybe i will use it in the future.<br>
Thanks for your help!</p>
Custom BatchNorm layer

username: zhryang
cooked: <p>I am writing BatchNorm layer, I want to know is there a forward and back propagation interface available in pytorch. I find interface of conv. This is conv code here:</p>
<pre><code class="lang-auto">from torch.autograd import Function
from torch.nn.grad import conv2d_input, conv2d_weight


class my_conv2d(Function):
    @staticmethod
    def forward(ctx, input, weight, bias=None, stride=1, padding=0, dilation=1, groups=1):
        ctx.bias = bias
        ctx.stride = stride
        ctx.padding= padding
        ctx.dilation = dilation
        ctx.groups = groups
        ctx.save_for_backward(input, weight)
        # print("{}-{}".format(current_batch, input.size()))
        ...


    @staticmethod
    def backward(ctx, grad_output):
        input, weight = ctx.saved_variables

        ...

        input_grad = weight_grad = None
        if ctx.needs_input_grad[0]:
            input_grad = conv2d_input(input.shape, weight, grad_output, ctx.stride, ctx.padding, ctx.dilation, ctx.groups)
        if ctx.needs_input_grad[1]:
            weight_grad = conv2d_weight(input, weight.shape, grad_output, ctx.stride, ctx.padding, ctx.dilation, ctx.groups)

        return input_grad, weight_grad, None, None, None, None, None
</code></pre>

username: ptrblck
cooked: <p>You can use <code>autograd.Function</code> for your custom functions as described <a href="https://pytorch.org/docs/stable/notes/extending.html">here</a>, so it’s not specific to a convolution use case. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: zhryang
cooked: <p>In this case, conv2d_input and conv2d_weight is provided by pytorch source code.Does pytorch also provide BatchNorm or relu related interfaces?</p>