akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to compute the diagonal of grad_x grad_y f(x,y; theta)?

Wei-Cheng_Chang: <p>Let say I have a function f(x,y; theta) with two d-dimensional input variables x and y, along with the model parameters theta.</p>
<p>Is it possible to use torch.autograd to efficiently compute the diagonal of partial_x partial_y f(x,y; theta)?</p>
<p>From the previously thread [<a href="https://discuss.pytorch.org/t/how-to-calculate-2nd-derivative-of-a-likelihood-function/15085">link</a>], I can easily get the diagonal of Hessian by doing two backward() operation, but that seems to only hold for the f(x; theta) case (i.e. one input variable, but not two).</p>
<p>Any thoughts? very appreciated.</p>
Faster for-loop for ensemble

sbuschjaeger: <p>I am working with ensembles of neural networks, roughly of the following form:</p>
<pre><code>class NNEnsemble():
    def __init__(self, n_estimators) :
        super().__init__()
        self.estimators = nn.ModuleList([ generate_model() for _ in range(self.n_estimators)])

    def forward(self, x):
        out = torch.stack([est(x) for est in self.estimators], dim=1)
        return torch.sum(out, dim=1)
</code></pre>
<p>Here, <code>generate_model()</code> generates a small base model (e.g. MLP 3 layers with 128 hidden neurons). As it turns out, the list-comprehension/for-loop during forward is rather slow. In fact, I only utilize &lt; 10% GPU for an ensemble with 100 networks of small size.  Is there any way to speed-up this operation, e.g. by using a “parallel” list comprehension?</p>
<p>I found one post about grouped convolution which mentions a similar structure, but there was no real solution besides waiting for cudnn support.</p>

albanD: <p>Hi,</p>
<p>Unfortunately if you do many small operations, it’s going to be fairly slow. The overhead of starting the work on the GPU might be larger than the time you spend actually computing.<br>
You can try to run on the CPU the small part of your model, increase the batch size to give more work to your gpu or find a way to agregate your estimators into a single one that performs one big operation.</p>
Non-deterministic Sorting

askliar: <p>Hi!</p>
<p>I have a following piece of code:</p>
<pre><code class="lang-auto">sorted_values, indices = torch.sort(score, dim=0, descending=True)

score[filt_idxs] = float("-inf")
filt_values, filt_indices = torch.sort(score, dim=0, descending=True)
</code></pre>
<p>I have manually examined <code>score</code> tensor and it only has a <strong>single</strong> <code>-inf</code> value after filtering. That would mean that <code>filt_indices</code> might shift by at most one up compared to <code>indices</code>.<br>
However, I see that in the cases where values of <code>score</code> tensor are the same, <code>indices</code> and <code>filt_indices</code> are completely different.</p>
<p>That behavior gets me thinking that sorting in PyTorch is non-deterministic. Is that so? If yes - how can I make it deterministic?</p>
<p>Extra details:</p>
<ul>
<li>All the operations are on CPU</li>
<li>PyTorch 1.1.0</li>
</ul>

albanD: <p>Hi,</p>
<p>Could you find a small <code>score</code> Tensor that reproduces this behavior to help us reproduce the issue?</p>

askliar: <p>Hi,<br>
Sure!</p>
<p>You can just try it with <code>score = np.ones(10)</code>.</p>

askliar: <p>Also, I have realized that my choice of wording was incorrect.<br>
It would be more proper to say that sorting in PyTorch is non-stable, which makes sense if internally sorting operation does quicksort.</p>
<p>However, it would be a really good feature if you could specify whether you need it to be stable or not by e.g. passing a flaf to sorting function. In this case it would internally run merge sort since it’s stable.</p>

singleroc: <p>Good proposal. I also meet a similar problem when translating PC side pytorch code to mobile side C++ code. In my scenario, counting sort is more efficient. However, the indices returned by pytorch is different with my counting sort when there are equal items (they all sort correctly). It would be nice if we could control the pytorch sort algorthim to keep result consistency.</p>

albanD: <p>Hi,</p>
<p>We would be happy to accept PR implementing new sorting algorithms  that can be chosen based on a flag (with the current behavior as default).</p>
Loading optimizer state-dict for different model

sbb: <p>Suppose I train model M1 with optimizer O1.  I save the state_dict of both.</p>
<p>Now I add a few layers to M1 to create model M2.  It is possible to load the weights of M1 to M2.</p>
<p>Is it possible to do the equivalent operation for the optimizer?  Ideally we would create optimizer O2 for model M2 and then load the state dict of O1 onto O2 to correspond to the parameters of M1.</p>
<p>Many thanks</p>
Apply different convolutions to a batch of tensors

juanmanpr: <p>Hey guys,</p>
<p>So I have a batch of convolutional filters and a batch of images. What is the best way to perform the per-element convolution so it is executed in parallel (without iterating through the batch indices).</p>
<p>Thanks!</p>

phan_phan: <p>So, you have filters in a tensor <code>w</code> of shape:<br>
<code>[batch_size, out_channels, in_channels, kernel_height, kernel_width]</code>,<br>
and images in a tensor <code>x</code> of shape:<br>
<code>[batch_size, in_channels, in_height, in_width]</code>,<br>
and you want an output of shape:<br>
<code>[batch_size, out_channels, out_height, out_width]</code>,<br>
where the <em>i_th</em> output is the convolution of the <em>i_th</em> image by the <em>i_th</em> filter ?</p>
<p>You can do it by merging the batch and channel dimensions together, and use a grouped convolution, with <code>groups=batch_size</code> :</p>
<pre><code class="lang-auto">o = torch.nn.functional.conv2d(
        x.view(1, batch_size*in_channels, x.size(2), x.size(3)),
        w.view(batch_size*out_channels, in_channels, w.size(3), w.size(4)),
        groups=batch_size)
 o = o.view(batch_size, out_channels, o.size(2), o.size(3))
</code></pre>
<p>It works ! But is it the best way to do it ? I don’t know.<br>
Let me know if this ends up faster.</p>

juanmanpr: <p>Hey phan_phan,</p>
<p>Thanks. Your solution is correct. It seems to be ever so slightly faster than simply iterating over the batch dimension.</p>
<p>Cheers</p>
When to stop training

kenneth_maanika: <p>Hi ! I am new to deep learning and have some questions about training networks. Hope someone can help me clear my doubts.</p>
<p>When i train my network i usually stop training when i see that the training loss plot have converged. However this always produces a validation loss plot that decreases and increases after a point. Meaning the network have over fitted right ? Should i stop training at that point when validation loss starts to increases even though training loss has not converged</p>
<p>Thanks in advance</p>

phan_phan: <p>You can save the state of your model regularly during training, in different files, so at the end you can choose which one to use, by looking at your losses plot.</p>
<p>To get the best results on data that is not in the training set, you’d want to use the model state saved at the lowest validation loss (just before the over-fitting starts to increase validation loss).</p>

kenneth_maanika: <p>Thank you for the advice <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>
Example of a vanilla nn.Transformer usage

aliutkus: <p>Hi There,</p>
<p>I cannot find some tutorial of the Transformer as it is implemeneted in pytorch 1.2.</p>
<p>I found <a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" rel="nofollow noopener">this one</a> but it looks like it’s some fancy stuff that uses only some part (encoder) and not the plain transformer.</p>
<p>Something like an example for beginners exist on how to use this ?</p>
<p>thanks for the amazing feature,</p>
<p>best</p>
Distributed error. module 'torch.distributed' has no attribute 'is_initialized'

congve1: <p>I’ve installed pytorch 1.0 on windows.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/3/321d3c2d53b59092e2330b46a65d29171bbb7508.png" data-download-href="https://discuss.pytorch.org/uploads/default/321d3c2d53b59092e2330b46a65d29171bbb7508" title="TIM%E6%88%AA%E5%9B%BE20181211225234"><img src="https://discuss.pytorch.org/uploads/default/original/2X/3/321d3c2d53b59092e2330b46a65d29171bbb7508.png" alt="TIM%E6%88%AA%E5%9B%BE20181211225234" data-base62-sha1="79kvzZFjwE8pMBBmIQWFtIWhqCk" width="690" height="88" data-dominant-color="080808"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">TIM%E6%88%AA%E5%9B%BE20181211225234</span><span class="informations">1287×165 8.67 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
When I try to use webcam demo provided by <a href="https://github.com/facebookresearch/maskrcnn-benchmark" rel="noopener nofollow ugc">maskrcnn-benchmark</a>. An error occured:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "webcam.py", line 80, in &lt;module&gt;
    main()
  File "webcam.py", line 64, in main
    min_image_size=args.min_image_size,
  File "G:\PyProjects\maskrcnn-benchmark\demo\predictor.py", line 115, in __init__
    _ = checkpointer.load(cfg.MODEL.WEIGHT)
  File "g:\pyprojects\maskrcnn-benchmark\maskrcnn_benchmark\utils\checkpoint.py", line 61, in load
    checkpoint = self._load_file(f)
  File "g:\pyprojects\maskrcnn-benchmark\maskrcnn_benchmark\utils\checkpoint.py", line 128, in _load_file
    cached_f = cache_url(f)
  File "g:\pyprojects\maskrcnn-benchmark\maskrcnn_benchmark\utils\model_zoo.py", line 44, in cache_url
    if not os.path.exists(cached_file) and is_main_process():
  File "g:\pyprojects\maskrcnn-benchmark\maskrcnn_benchmark\utils\comm.py", line 28, in is_main_process
    if not torch.distributed.is_initialized():
AttributeError: module 'torch.distributed' has no attribute 'is_initialized'
</code></pre>
<p>But when I checked the pytorch 1.0 document, the <code>torch.distributed</code> module does have the <code>is_initialized()</code> method.<br>
How to solve this problem</p>

jinfagang: <p>Same issue herehyrtjedtujtuktu</p>

pietern: <p>We don’t support Windows for torch.distributed, unfortunately.</p>
Inconsistent results when load the weights from JIT ScriptModule to nn.Module

Bingchen_Liu: <p>Hi</p>
<pre><code class="lang-auto">jit_net = torch.jit.load(saved_path)   # load the pre-trained network defined as a ScriptModule
nn_net = TheSameNet()    # this is the same network as jit_net but defined as a nn.Module
nn_net.load_state_dict(jit_net.state_dict())
</code></pre>
<p>I have a pre-trained ScriptModule and now I want to change some forward-pass function of it, therefore I define an exactly same nn.Module class and want to load the weights into this new network. I do this by the above code.<br>
However, when I successfully load the pre-trained weights in the ScriptModule model to the nn.Module model, the “nn_net” outputs different results as the “jit_net” for the same inputs. I would like to know if there is a proper way to transfer the weights between jit-scriptmodule and nn.module, or do I miss something, or if it is potentially a bug for the inconsistent output, or is it just not recommened to do so (transfering weights between ScriptModule and nn.Module).</p>

joe8086: <p>I also meet this kind inconsistent before.<br>
So I give up use ScriptModel, instead I build a parser to parse ScriptModel.graph IR.<br>
Then use the IR label to access state_dict<br>
But the IR is not public release, it still not a good idea to handle this issue (because IR may change between different version)<br>
I think this is a pytorch bug, need pytorch to fix it.</p>

Michael_Suo: <p>This is not expected, no, but could happen if there is a source of non-determinism in the model. If you could file a Github issue with a repro, that would be very helpful. Ideally a simple model that we can run to produce an inconsistency.</p>
Loss becoming nan after some time

shivam2298: <p>I have a total_ loss which is sum of -</p>
<ol>
<li>A BCELoss</li>
<li>A Crossentropy loss</li>
<li>A <strong>custom loss function</strong> for image gradient.</li>
</ol>
<p>The problem I am facing is that after 1st batch, some weights are updated to nan which results in all outputs as nan. If I remove the gradient loss, then it works fine.<br>
What I found out was the denominator in the gradient loss were becoming 0, which was causing the problem. To fix it, I replace all denominators( Gradient magnitude) which were 0 with 1. But doing this only computes numerical loss for first backprop, next results in nan.</p>
<p>code-</p>
<pre><code class="lang-auto">sobel_x = torch.tensor([[+1, 0, -1], [+2, 0, -2], [+1, 0, -1]], requires_grad=False,dtype = torch.float)
	sobel_y = torch.tensor([[+1, +2, +1], [0, 0, 0], [-1, -2, -1]], requires_grad=False,dtype = torch.float)
	if cuda:
		sobel_x,sobel_y = sobel_x.cuda(),sobel_y.cuda()
		boundary_mask = boundary_mask.cuda()
	sobel_x = sobel_x.view((1,1,3,3))
	sobel_y = sobel_y.view((1,1,3,3))
	
	
	#gradients in the x and y direction for both predictions and the target transparencies
	G_x_pred = F.conv2d(pred,sobel_x,padding = 1)
	G_y_pred = F.conv2d(pred,sobel_y,padding = 1)
	G_x_target = F.conv2d(target,sobel_x,padding = 1)
	G_y_target = F.conv2d(target,sobel_y,padding = 1)

	#magnitudes of the gradients
	M_pred = torch.sqrt(torch.pow(G_x_pred,2)+torch.pow(G_y_pred,2))
	M_target = torch.sqrt(torch.pow(G_x_target,2)+torch.pow(G_y_target,2))
	
	#taking care of nans
	M_pred = (M_pred==0.).float() + M_pred
	M_target = (M_target==0.).float() + M_target

	# Lcos = (1-v_pred*v_target)*Magnitude_pred
	Lcos = (1-torch.abs((G_x_pred/M_pred)*(G_x_target/M_target)+(G_y_pred/M_pred)*(G_y_target/M_target)))*M_pred
	
	#Lmag = max(lambda*M_target-M_pred,0)
	lambd = 1.5
	Lmag = lambd*M_target-M_pred
	Lmag[Lmag&lt;0] = 0

	gamma_1 = 0.5
	gamma_2 = 0.5

	#total gradient loss
	image_gradient_loss = (gamma_1*Lcos+gamma_2*Lmag)*boundary_mask
</code></pre>
<p>Can someone please help in tackling this situation.</p>

shivam2298: <p>I just checked, the gradients are becoming nan and the refine loss is 0. due to nan gradient, weights are nan and hence results in nan output. Why are gradients ‘nan’ when using refine loss because when I don’t use it, the gradients are fine.</p>

Kushaj: <p>In the denominator add epsilon, to prevent the denominator from becoming zero and gradient clipping can also help.</p>

suruoxi: <p>add epsilon in torch.sqrt</p>
Deprecation warnings for dispatch macros that don't appear to be deprecated

marcman411: <p>I extended PyTorch with a few custom C++/CUDA layers using PyTorch 1.0. I am now upgrading those layers to support PyTorch 1.2, and I can’t seem to figure out this deprecation warning:</p>
<pre><code class="lang-auto">ext_modules/include/nn/cuda/my_file.cuh: In lambda function:
ext_modules/include/nn/cuda/my_file.cuh:169:101: warning: ‘c10::ScalarType detail::scalar_type(const at::DeprecatedTypeProperties&amp;)’ is deprecated [-Wdeprecated-declarations]
/data/anaconda3/envs/my_lib/lib/python3.7/site-packages/torch/include/ATen/Dispatch.h:78:1: note: declared here
 inline at::ScalarType scalar_type(const at::DeprecatedTypeProperties &amp;t) {
</code></pre>
<p>It seems to be related to my usage of <code>AT_DISPATCH_FLOATING_TYPES</code>. For example, I get this warning from my reimplementation of <em>im2col</em>:</p>
<pre><code class="lang-auto">void Im2Col2DLauncher(at::Tensor data_im, const int64_t channels,
                      const int64_t height_im, const int64_t width_im,
                      const int64_t width_out, const int64_t width_col,
                      const int64_t kernel_h, const int64_t kernel_w,
                      const int64_t pad_h, const int64_t pad_w,
                      const int64_t stride_h, const int64_t stride_w,
                      const int64_t dilation_h, const int64_t dilation_w,
                      at::Tensor data_col) {
  const int64_t num_kernels = channels * width_col;
  const dim3 blocks((num_kernels + CUDA_NUM_THREADS - 1) / CUDA_NUM_THREADS);

  // Launch channels * width_col kernels, with each kernel responsible for
  // copying a the convolutions over a single channel.
  AT_DISPATCH_FLOATING_TYPES(
      data_col.type(), "Im2Col2DKernel", ([&amp;] {
        Im2Col2DKernel&lt;&lt;&lt;blocks, CUDA_NUM_THREADS&gt;&gt;&gt;(
            num_kernels, data_im.data&lt;scalar_t&gt;(), height_im, width_im,
            width_out, width_col, kernel_h, kernel_w, pad_h, pad_w, stride_h,
            stride_w, dilation_h, dilation_w, data_col.data&lt;scalar_t&gt;());
        CUDA_CHECK(cudaGetLastError())
      }));
}
</code></pre>
<p>The only related include I am using is <code>#include &lt;torch/extension.h&gt;</code>.</p>
<p>I looked at the <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/Dispatch.h" rel="nofollow noopener"><code>Dispatch.h</code></a> file but I don’t see <code>AT_DISPATCH_FLOATING_TYPES</code> listed as deprecated.</p>
<p>What’s going on? And is there a deprecation guide somewhere?</p>

marcman411: <p>It appears the issue lies in:</p>
<pre><code class="lang-auto">AT_DISPATCH_FLOATING_TYPES(
      data_col.type(), ...
</code></pre>
<p>The correct way to do this now is to use <code>.scalar_type()</code>, so it should read:</p>
<pre><code class="lang-auto">AT_DISPATCH_FLOATING_TYPES(
      data_col.scalar_type(), ...
</code></pre>

Ramin_Rasoulinezhad: <p>Thanks. it solved my problem.</p>
Problem with .detach()

Arhazf: <p>I am trying to have different activations in forward and backward paths. I wrote this code but, it seems that it does not work properly in the backward path.<br>
Do I have to include anything else?</p>
<pre><code>class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(28 * 28, 64)
        self.fc2 = nn.Linear(64, 10)
        nn.init.xavier_normal_(self.fc1.weight)
        nn.init.xavier_normal_(self.fc2.weight)

    def forward(self, x):
        x1 = F.relu(self.fc1(x))
        x_backward = x1
        x1[x1 &lt;= 0] = 0
        x1[x1&gt; 0] = 1
        x_forward = x1
        y1 = x_backward + (x_forward - x_backward).detach()
        y2 = self.fc2(y1)
        y3 = F.log_softmax(y2, dim=1)
        return y3


for epoch in range(epochs):
    total_train = 0
    correct_tr = 0
    for batch_idx, (data, label) in enumerate(train_set):
        data, label = Variable(data), Variable(label)
        data = data.view(-1, 28*28)
        optimizer.zero_grad()
        data = data.float()
        net_out = net(data)
        print(net_out.size())
        loss = criterion(net_out, label)
        loss.backward()
        optimizer.step()
        .....</code></pre>

ptrblck: <p>If you assign a tensor to another, you will use a reference and thus manipulate both, <code>x_backward</code> and <code>x1</code> using inplace operations.<br>
Try to call <code>x_backward = x1.clone()</code> and test the code again.</p>
<p>Double post from <a href="https://discuss.pytorch.org/t/binary-activation-function-with-pytorch/56674/3">here</a>.</p>

Arhazf: <p>Thank you. It works.</p>
Determing what models and tensors exist on a GPU?

ProGamerGov: <p>So, I have an issue with some code where GPU:0 always seems to have abnormally high usage (and it gets worse with bigger inputs). Is it possible to see what tensors and models exist on a particular GPU? And if so, how would I do that? The hope is that I could use this information to help determine the cause of the issue.</p>

Nikronic: <p>Hi,</p>
<pre><code class="lang-auto">print(tensor.device)
print(model.get_device())
</code></pre>
<p>Note that if your <code>model</code> and <code>tensor</code> are on GPU, you will get a output, otherwise it will through and error. They will return the <code>id</code> of the currently being used GPUs.</p>
<p>Bests<br>
Nik</p>
Printing weights in LSTM Networks

Rick_Sanchez: <p>Hi.<br>
I have problem with printing weights between inputs and first layer.</p>
<pre><code class="lang-auto">class LSTMClassifier(nn.Module):
    """Very simple implementation of LSTM-based time-series classifier."""

    def __init__(self, input_size, hidden_size, n_layers, output_size):
        super().__init__()
        self.hidden_size = hidden_size
        self.n_layers = n_layers
        self.rnn = nn.LSTM(input_size, hidden_size, n_layers, batch_first=True)
        self.fc = nn.Linear(hidden_size, output_size)
        self.batch_size = None
        self.hidden = None

    def forward(self, x):
        h0, c0 = self.init_hidden(x)
        out, (hn, cn) = self.rnn(x, (h0, c0))
        out = self.fc(out[:, -1, :])
        return out

    def init_hidden(self, x):
        h0 = torch.zeros(self.n_layers, x.size(0), self.hidden_size)
        c0 = torch.zeros(self.n_layers, x.size(0), self.hidden_size)
        return [t for t in (h0, c0)]

        return y
</code></pre>
<p>I know how to do this with feedforward Neural Network but with LSTM it’s not working.<br>
Thanks for help.</p>

albanD: <p>Hi,</p>
<p>Why can’t you add a print statement in your <code>forward()</code> function?<br>
Can you give an example of what you want to achieve?</p>

Rick_Sanchez: <p>How this print statement should looks like?<br>
I just wanna know what neural network see as important inputs referring to weights.</p>

albanD: <p>You can add <code>print(self.rnn.weight_ih)</code> for example. You can find <a href="https://github.com/pytorch/pytorch/blob/4cc16782f3d0561acce447e3471934478c84e436/torch/nn/modules/rnn.py#L740-L747">here</a> the definition of each weight and their names.</p>
<aside class="quote no-group" data-username="Rick_Sanchez" data-post="3" data-topic="56808">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rick_sanchez/48/10907_2.png" class="avatar"> Rick_Sanchez:</div>
<blockquote>
<p>I just wanna know what neural network see as important inputs referring to weights.</p>
</blockquote>
</aside>
<p>Not aure what that means <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"></p>

Rick_Sanchez: <aside class="quote no-group" data-username="albanD" data-post="4" data-topic="56808">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>Not aure what that means <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"></p>
</blockquote>
</aside>
<p>I’m just checking my theory <img src="https://discuss.pytorch.org/images/emoji/apple/laughing.png?v=12" title=":laughing:" class="emoji" alt=":laughing:" loading="lazy" width="20" height="20">.<br>
Thanks for help.</p>
How does conv2d apply kernels?

carlstronaut: <p>Hello!</p>
<p>I am following along PyTorchs Udacity tutorial and I’m working with CNNs right now.<br>
In one of the assignments, we’re taking a (3 x 32 x 32) image and running it through a CNN to classify the image. When defining the convolutional layers the first two layers looks like this:</p>
<pre><code class="lang-auto"># First input = (32 x 32 x 3)
self.conv1 = nn.Conv2d(3, 16, 3, padding=1)
# Second input = (16 x 16 x 16)
self.conv2 = nn.Conv2d(16, 32, 3, padding=1)
</code></pre>
<p>I’m curious about how PyTorch transforms the input data with depth 3 to a depth of 16, since 16 isn’t a multiple of three. In my mind intuitively, PyTorch would apply equally as manny kernels to each channel.  Does it use 5 kernels for each of the three channels, and then select a random channel to run an additional run?</p>
<p>Thanks!</p>

ptrblck: <p>In the default setup you are using, each kernel uses all input channels and creates an output channel. Have a look at <a href="http://cs231n.github.io/convolutional-networks/#conv" rel="nofollow noopener">CS231n</a> which describes this approach quite well.</p>

carlstronaut: <p>Ah, thanks alot.</p>
<p>If someone comes looking into this thread, the intuition is that if we define the convolutional layer as:</p>
<p><code>nn.Conv2d(3, 16, 5, padding=1)</code>,</p>
<p>we define 16 filters, which all have the dimension (5x5x3). I.e, the filters look at the entire depth of the input. The output would then be the output from every filter, stacked on top of each other.</p>
<p>Refer to the excellent link in ptrblck’s answer for further explanation.</p>
Does torch.cat() method have gradient?

Ma_John: <p>Greetings!<br>
I have three weight maps that’s output from a CONV layer. They are of the same size [B, C, H, W]<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/e/ae750a91298d8d99d3cb14ebf31470730f3ce185.png" data-download-href="https://discuss.pytorch.org/uploads/default/ae750a91298d8d99d3cb14ebf31470730f3ce185" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/e/ae750a91298d8d99d3cb14ebf31470730f3ce185.png" alt="image" data-base62-sha1="oTjQ3zpoGKDNWJSXfPufRCjyRaB" width="690" height="251" data-dominant-color="2C2E28"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">887×323 9.45 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
My goal is to employ a sparsemax(variant of softmax) normalization on every pixel of the three weight maps so that after normalization their values would sum to one and have a sparse distribution. But my implementation above doesn’t seem to work. So I wonder if torch.cat() method have gradient? Or is it something else that leads to the mistake?</p>

btimar: <p>Hi – torch.cat() does pass gradients, as you can verify with the following:</p>
<pre><code class="lang-auto">import torch

a = torch.ones(2, requires_grad=True)
b = torch.ones(3, requires_grad=True)
c = torch.cat((a, b))
output = c.sum()
output.backward()
print(a.grad) #tensor([1., 1.])
print(b.grad) #tensor([1., 1., 1.])
</code></pre>
<p>Could you be more specific about your problem?</p>
Export a LSTM with an embedding (unknow size input) layer

davebulaval: <p>Hi,<br>
I’m trying to export in ONNX a model which received a list of index of word for a embedding layer in my model. The problem is I need a dummy input for the export and my input size will vary.</p>
<p>How can I solved my problem ?</p>
<p>tks</p>

davebulaval: <p>I found a work around by padding all my sentence to a fixed length.</p>
Torch: invalid memory size -- maybe an overflow? at /opt/conda/conda-bld/pytorch-nightly_1549065556721/work/aten/src/TH/THGeneral.cpp:188

Gkv: <p>Hi all,<br>
After 2,3 epochs (sometimes before finishing first epoch also) my model gives the error, <em>Torch: invalid memory size – maybe an overflow? at /opt/conda/conda-bld/pytorch-nightly_1549065556721/work/aten/src/TH/THGeneral.cpp:188</em>.  Can anyone help me to resolve this?. Pytorch version: 1.0.0.dev20190201 on Ubuntu 18.04.<br>
Thanks</p>

ptrblck: <p>Do you see the same error using the latest stable release (or latest nightly build)?</p>

Gkv: <p>I did not update Pytorch. I will do the same and let you know the details.</p>

Gkv: <p>I have updated Pytorch. Now I am getting another error in my data loader, <em>RuntimeError: Trying to create tensor with negative dimension -2: [-2, 300]</em>.</p>

ptrblck: <p>Could you try to isolate which batch creates this issue or is it thrown in the first step of your <code>DataLoader</code> iteration?<br>
You could iterate your <code>Dataset</code> and check, which index is raising this error.<br>
Once you’ve isolated it, you could check what’s going wrong in the <code>__getitem__</code> (or where this exception is raised).</p>

Gkv: <p>Thanks. I resolved it. There was a small dimension error in my data loader.</p>
About nn.LSTM using with train_loader

jinujanu: <p>Hello everyone.</p>
<p>I’m currently learning to use <a href="https://pytorch.org/docs/stable/nn.html#lstm" rel="nofollow noopener">nn.LSTM</a> with pytorch and a bit confused with the description of the input of the function.</p>
<p>Basically I’m trying to feed my dataset matrix (M x N) as M timesteps, N numbers of features.<br>
Since the dataset is a matrix, I wanted to feed the dataset recursively into the LSTM network with Dataloader(utils.data.Dataset).</p>
<p>The point where i got confused was the size of input (seq_len, batch, input_size)</p>
<p>Let’s say I’m getting my data_loader with batch_size=10.<br>
In order to generate the train_loader with the right form, I had to make the previous size of (M x N) into the size including the sequence_length which could simply be transformed to (M/seq_len, seq_len, N).</p>
<p>Then the input size of my nn.LSTM would be like:<br>
(M/seq_len/batch_size, seq_len, N)</p>
<p>So, my main question comes:<br>
If i feed this data size into the LSTM model nn.LSTM(N, hidden_size),<br>
is the LSTM model automatically doing the “recursive operation” within the sequence of the whole batch?</p>
<p>I wanted to know the actual behavior that’s done within this LSTM layer… anyways the operation should be done recursively within both the sequence, And the batch right?</p>
<p>I’m not sure i made the questionaire clear because this is actually my current status working on this nn.LSTM(quite messed up…lol), hope someone could help me organizing the ideas.</p>
<p>Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
No_grad vs detach

palimboa: <p>Hi,</p>
<p>I have read this thread here carefully: <a href="https://discuss.pytorch.org/t/detach-no-grad-and-requires-grad/16915/18">question</a> but I am still confused about it.</p>
<p>Let’s say we have 3 operations:</p>
<pre><code class="lang-auto">x1 = input
x2 = op1(x1)
x3 = op2(x2)
x4 = op3(x3)
loss = lossfunction(x4)
</code></pre>
<p>What is the difference between:</p>
<pre><code class="lang-auto">
x1 = input
x2 = op1(x1)
with torch.no_grad()
    x3 = op2(x2)
x4 = op3(x3)
loss = lossfunction(x4)

</code></pre>
<p>and (using detach)</p>
<pre><code class="lang-auto">x1 = input
x2 = op1(x1)
x3 = op2(x2).detach()
x4 = op3(x3)
loss = lossfunction(x4)
</code></pre>
<p>Assuming all operations have learnable parameters, how does the gradient propagation, memory management and activations change?</p>

albanD: <p>Hi,</p>
<p>In this particular case, they will be quite similar.<br>
The only difference I guess is that in the first case, you will never even instantiate the state in op2. While you will in the second case and discard them later.<br>
If you do inplace operations in op2 though, there will be some difference.</p>
How to do sequence classification with nn.TransformerEncoder()?

whisht: <p>Hey, guys. I am doing a sequence classification task using <code>nn.TransformerEncoder()</code>. Whose pipeline is similar to <code>nn.LSTM()</code>.</p>
<p>I have tried several temporal features fusion methods:</p>
<ol>
<li>
<p>Selecting the final outputs as the representation of the whole sequence.</p>
</li>
<li>
<p>Using an affine transformation to fuse these features.</p>
</li>
<li>
<p>Classifying the sequence frame by frame, and then select the max values to be the category of the whole sequence.</p>
</li>
</ol>
<p>But, all these 3 methods got a terrible accuracy, only <strong>25%</strong> for 4 categories classification. While using <code>nn.LSTM</code> with the last hidden state, I can achieve <strong>83%</strong> accuracy easily. I tried plenty of hyperparameters of <code>nn.TransformerEncoder()</code>, but without any improvement for the accuracy.</p>
<p>Could you guys give me some practical advice? Thanks</p>

dhpollack: <p>I think that BERT uses the special <code>[CLS]</code> token to get the entire sentence representation of the sequence in one token.  Perhaps you can try something similar and then take this vector as your input.  For BERT, they generally put this token in the first position.</p>

Abhilash_Srivastava: <p>+1 <a class="mention" href="/u/dhpollack">@dhpollack</a><br>
From my experience, for classification tasks, usually the concatenation of last few encoder layers and then operating on this (using RNN or Linear) tend to give best results. You can also average or maxpool to reduce dimensions.<br>
If you’re planning to use a pre-trained model like BERT, the output representation of <code>[CLS]</code>  token is your best bet. For some reason <code>[CLS]</code> captures the meaning of the entire sequence (no theoretical backing to this).</p>

dhpollack: <p>The idea is that every token interacts with every other token.  So if you have a token that captures the meaning of the entire sequence without influencing or having meaning within the sequence itself.</p>
How do I parse parameters of BatchNorm2d?

han324solo: <p>I want to extract every parameters of certain BatchNorm2d layer from pretrained model, like epsilon, input size, momentum etc…</p>
<p>Then, I will feed the extracted parameters to my new BatchNorm2d layer from different model.(let say this is target model)</p>
<p>I don’t think torch.save&amp;load won’t work for this case cuz the pretrained model and target model have different architecture.</p>
<p>How do I parse parameters of BatchNorm2d?</p>

ptrblck: <p>You could get all attributes from the original batch norm layer simply by accessing them:</p>
<pre><code class="lang-python">bn = nn.BatchNorm2d(3)
print(bn.eps)
print(bn.momentum)
...
</code></pre>
Pytorch question about BiLSTM-CRF

lucky: <p>pytorch tutorial have a bilstm-crf example。But， it isn’t used minibatch。<br>
when i try to make a minibatch in it。I find that， CRF can’t be minibatch?<br>
And, CRF need run in cpu? it will be so slowly！<br>
aspect these，there are also some questiones below：</p>
<ol>
<li>how pytorch auto deal variable sequence length？padding a same length？but pytorch is dynamic right？</li>
<li>I don’t konw why，but pytorch is slowly so much and the uses of gpu util<br>
is little.</li>
</ol>
<p>may be all these question is caused by （batch and crf ）<br>
how can CRF read the batch and deal in gpu？<img src="//discuss.pytorch.org/images/emoji/apple/sob.png?v=5" title=":sob:" class="emoji" alt=":sob:"></p>

Hodaya_Binyamini: <p>Hi,<br>
I also have the problem to fit the model for the GPU.<br>
Did you fixed it?</p>
Custom Back-propagation gradient update

paganpasta: <p>Hi, I just started to use Pytorch and wanted to know if there’s any thing wrong with my approach.<br>
I’ll briefly explain the problem I have and how I have decided to solve it.</p>
<p>Neural network: M<br>
Data: A labels: a loss: l_a (Cross-entropy), gradient of loss w.r.t parameters: G_a<br>
Data: B labels: b loss: l_b (Cross-entropy), gradient of loss w.r.t parameters: G_b<br>
learning rate: lr</p>
<p>Algorithm I want to implement goes like this:</p>
<ol>
<li>Forward-pass A to obtain M(A)</li>
<li>Compute l_a and correspondingly G_a using M(A)</li>
<li>Forward-pass B to obtain M(B)</li>
<li>Compute l_b and correspondingly G_b using M(B)</li>
<li>Some computations based on G_a and G_b to obtain G_c</li>
<li>Update M_0 w.r.t G_c</li>
</ol>
<p>My solution:</p>
<pre><code class="lang-auto">G_a = [ ]
G_b = [ ]
M.zero_grad()
outputs = net(A)
loss_value_A = l_a(a, outputs)
loss_value_A.backward()
for f in M.parameters():
    G_a.append(f.grad.data)

M.zero_grad()
outputs = net(B)
loss_value_B = l_b(b, outputs)
loss_value_B.backward()
for f in M.parameters():
    G_b.append(f.grad.data)

G_c = some_computation(G_a, G_b)
M.zero_grad()
for f in M.parameters():
    f.data.sub_(lr * f.grad.data for f in G_c)
</code></pre>
<p>Thanks.</p>
Help with exporting model

lugookeke: <p>Hi guys i can’t seem to export my model to onnx properly as when i convert the onnx back to coreml, i get crashes when i run it on my iphone. Anyone know a way to do it preperly? its a drn segmentation model.</p>
<p>thanks</p>
<pre><code class="lang-auto">import argparse
import os
import sys
import numpy as np
import torch
import torchvision.transforms as transforms
from PIL import Image

from networks.drn_seg import DRNSeg
from utils.tools import *
from utils.visualize import *


if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "--input_path", required=True, help="the model input")
    parser.add_argument(
        "--dest_folder", required=True, help="folder to store the results")
    parser.add_argument(
        "--model_path", required=True, help="path to the drn model")
    parser.add_argument(
        "--gpu_id", default='0', help="the id of the gpu to run model on")
    parser.add_argument(
        "--no_crop",
        action="store_true",
        help="do not use a face detector, instead run on the full input image")
    args = parser.parse_args()

    img_path = args.input_path
    dest_folder = args.dest_folder
    model_path = args.model_path
    gpu_id = args.gpu_id

    # Loading the model
    if torch.cuda.is_available():
        device = 'cuda:{}'.format(gpu_id)
    else:
        device = 'cpu'

    model = DRNSeg(2, None)
    state_dict = torch.load(model_path, map_location=device)
    model.load_state_dict(state_dict['model'])
    model.to(device)
    model.eval()

    # Data preprocessing
    tf = transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize(
            mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
    ])

    im_w, im_h = Image.open(img_path).size
    if args.no_crop:
        face = Image.open(img_path).convert('RGB')
    else:
        faces = face_detection(img_path, verbose=False)
        if len(faces) == 0:
            print("no face detected by dlib, exiting")
            sys.exit()
        face, box = faces[0]
    face = resize_shorter_side(face, 400)[0]
    face_tens = tf(face).to(device)

    with torch.no_grad():
        flow = model(face_tens.unsqueeze(0))[0].cpu().numpy()
        flow = np.transpose(flow, (1, 2, 0))
        h, w, _ = flow.shape

    modified = face.resize((w, h), Image.BICUBIC)
    modified_np = np.asarray(modified)
    reverse_np = warp(modified_np, flow)
    reverse = Image.fromarray(reverse_np)

    # Saving the results
    modified.save(
        os.path.join(dest_folder, 'cropped_input.jpg'),
        quality=90)
    reverse.save(
        os.path.join(dest_folder, 'warped.jpg'),
        quality=90)
    flow_magn = np.sqrt(flow[:, :, 0]**2 + flow[:, :, 1]**2)
    save_heatmap_cv(
        modified_np, flow_magn,
        os.path.join(dest_folder, 'heatmap.jpg'))


</code></pre>
Redundant context in forward pass when some input tensors do not require grad

urital: <p>In the following example, I see that the memory consumption is very high, which clearly shows that the forward pass is saving all the tensors that are used in the multiplications, even though only one of the tensors requires grad.</p>
<pre><code class="lang-auto">gs=256
b = torch.randn((gs,gs), dtype=torch.float32)
b.requires_grad_()
a = torch.randn((gs,gs), dtype=torch.float32)
b_orig = b

start_mem_usage = memory_profiler.memory_usage(-1,0.1)

for i in range(1000):
    b = (b+1).mm(a)

l = torch.sum(b)
end_mem_usage = memory_profiler.memory_usage(-1,0.1)
print ("mem usage",end_mem_usage[0] - start_mem_usage[0])
</code></pre>
<p>this code will need ~285 MB, which means that the forward pass is saving 1000 tensors of 256x256 floats. Since ‘a’ doesn’t require gradient, I’m having a hard time understanding why the context needs more than the value of ‘a’, which never changes.</p>
<p>So I went ahead and implemented my own matmul for constant multiplier, and this time it doesn’t consume almost any memory. My question is why doesn’t torch understand that by itself, according to the grad_required attribute.</p>
<pre><code class="lang-auto">class myMatMulConstant(torch.autograd.Function):
    @staticmethod
    def forward(ctx,input,const_mat):
        ctx.save_for_backward(const_mat)
        return input.mm(const_mat)

    @staticmethod
    def backward(ctx,grad_output):
        const_mat = ctx.saved_tensors[0]
        grad_input = grad_output.mm(const_mat.t())
        return  grad_input, None

.
.
.
my_mm_const = myMatMulConstant.apply

for i in range(1000):
    b = my_mm_const(b+1,a)
</code></pre>
<p>The above implementation doesn’t consume 1000 tensors in memory, which makes sense.<br>
Thanks!</p>
Getting Jacobian of RNN

vdw: <p>In the paper <a href="https://arxiv.org/abs/1604.08275" rel="nofollow noopener">“Crafting Adversarial Input Sequences for Recurrent Neural Networks”</a>, the authors evaluate the importance of a word in sequence by looking at the gradients of the embeddings w.r.t. to the outcome. It essentially boils down to calculate the Jacobian of the unfolded representation of the RNN.</p>
<p>How should I go about to do the same with a trained PyTorch RNN model for simple classification? I can see to check <code>embedding.weight.grad</code>, but I struggle with the recursive nature. I assume/guess that I would have to check the gradient of the respective embedding of a word at each time step during <code>backward()</code>. Or am I completely on the wrong? I can vaguely follow the formulas in the paper but I don’t know how to implement this approach.</p>
Where could I find torch module's methods in source code?

michiboo: <p>Hi, I am new to pytorch and currently working on an issue, but I am having trouble to find torch.tril source code.<br>
It would be great if you can help me!</p>
<p>Additionally, you can also give tip and trick how to find function/method in unfamiliar code base that would great!</p>
<p>I would normally ask question in IM but pytorch’s slack are invitation only <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>
<p>Many thanks!</p>

albanD: <p>Hey !</p>
<p>You can PM <a class="mention" href="/u/smth">@smth</a> to get an invitation to the slack <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<p>Otherwise, for function implementation you will have to check the cpp code.<br>
The simplest way is usually to use the search from github on the repo.<br>
For tril, you’ll find a reference to <a href="https://github.com/pytorch/pytorch/blob/ade60f8a8deb1943ba8b9f0f249c58beab83d32b/aten/src/ATen/native/TensorFactories.cpp#L603" rel="nofollow noopener">this</a> function that is the cpu implementation.<br>
Unfortunately these is no standard way to look for functions at the moment (for historical reasons, they are not all at the same place) <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
Collecting garbage questions

blackbirdbarber: <p>I found:</p>
<pre><code class="lang-auto">import torch
import gc
for obj in gc.get_objects(): 
    try:
        if torch.is_tensor(obj) or (hasattr(obj, 'data') and torch.is_tensor(obj.data)):
            print(type(obj), obj.size())
    except: pass
</code></pre>
<p>Are these objects <code>gc.get_objects()</code> ready to be collected or they will not be collected unless we set them to None inside of the program?</p>
<p>Then I found:</p>
<pre><code class="lang-auto"># prints currently alive Tensors and Variables
import torch
import gc
for obj in gc.get_objects():
    if torch.is_tensor(obj) or (hasattr(obj, 'data') and torch.is_tensor(obj.data)):
        del obj
torch.cuda.empty_cache()
</code></pre>
<p>What will this code do?<br>
Is <code>del obj</code> same as <code>obj=None</code>.<br>
What is the difference between <code>torch.cuda.empty_cache()</code> and <code>gc.collect()</code>.</p>
What is the difference between .grad and ._grad?

Quan_Vuong: <p>In the most popular a3c pytorch implementation, there’s a function <code>ensure_shared_grads</code> that ensure the local and global shared optimizer share gradient.</p>
<p>In line <a href="https://github.com/ikostrikov/pytorch-a3c/blob/master/train.py#L18" rel="nofollow noopener">18</a>, <code>shared_param._grad = param.grad</code> is used to share the gradient of the local optimizer with the global optimizer. However, only <code>shared_param.grad</code> is checked to ensure the gradients are shared in subsequently backward pass.</p>
<p>What’s the difference between <code>.grad</code> and <code>._grad</code> ?</p>
<p>Thanks!</p>

hughperkins: <p><code>_grad</code> is an internal variable, that is writable from python. you’ll notice that <code>.grad</code> is read-only</p>

Yummy_Chen: <p>so, if I want change the gradient <strong>manually in optimizer</strong>, must I using the following program?</p>
<pre><code class="lang-auto">for group in meta_optim.param_groups:  # meta_optim is optimizer
     for p,g in zip(group['params'], grads):  # grads were calculated manually.
         if p.grad is not None:
             p._grad.data = g.data # rather than p.grad.data=g.data?
meta_optim.step()
</code></pre>

btimar: <p>Hey – according to <a href="https://pytorch.org/docs/stable/autograd.html#variable-deprecated" rel="nofollow noopener">these docs</a> Variable is deprecated, and it seems that you can write to the Tensor.grad attribute now. For example: (python 3.7, torch v1.2.0):</p>
<pre><code class="lang-auto">import torch

#case 1, an ordinary Tensor
a = torch.randn(1, requires_grad=True)
assert a.grad is None
a.grad = torch.ones_like(a)
a.grad.data = 2* torch.ones_like(a)
print(a.grad)  # prints tensor([2.])

# case 2, parameters in an nn.Module
class MyModel(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.layer = torch.nn.Linear(1,1)
    
model = MyModel()
for param in model.parameters():
    assert param.grad is None
    param.grad = torch.ones_like(param.data)
    param.grad.data = 2 * torch.ones_like(param.data)
    print(param.grad) #more 2's, no error


</code></pre>
Mapping layer class with its parameters

hashbrown: <p>I have a trained model and I want to do some surgery on it like merging layers into a fewer ones. For that, I need the following information for every layer in the model:</p>
<ol>
<li>layer class name - like nn.Conv2d, nn.BatchNorm2d or my custom classes.</li>
<li>layer definition information - like what stride, what padding, bias true or false etc.</li>
<li>layer parameters - like weights, bias, running mean &amp; variance etc.</li>
</ol>
<p>I have not been able to find a mechanism to get this consolidated information.</p>
<p>When I parse through the state dictionary, I only get a key and associated parameters. There is no way to find layer class/definition information from the key name. And when I use named_modules(), I get the layer information, but don’t get the key information to retrieve appropriate parameters from the state dictionary.</p>
<p>Appreciate  your inputs, thanks!</p>

albanD: <p>Hi,</p>
<p>When you go through <code>named_modules</code>, you have access to the parameters of each module. Why do you need to map it back to the state_dict?</p>

hashbrown: <p><a class="mention" href="/u/alband">@albanD</a>, that is a great suggestion. Let me try that.</p>
Individual learning rate for each sample

mital: <p>Hi,</p>
<p>I’m training CNNs on MNIST dataset and would like to prioritize learning some of the samples (they will be determined during the training) and thought a straightforward way to do this would be to set a learning rate (multiplier) for these “important” samples.</p>
<p>Another way to do this is maybe by duplicating these samples in training and validation sets, but, I think, with dynamic prioritization the code can get too big and complex.</p>
<p>What I’m looking for is basically like weighing in semantic segmentation but instead of weights being static and per class I want them to be dynamic and per sample.</p>

albanD: <p>Hi,</p>
<p>You can use a weighted loss to change the relative importance of your samples. Indeed, multiplying the loss for a single sample by 2 is the same as multiplying the gradients corresponding to this sample by 2.</p>

mital: <p>Thanks, I didn’t think of that, good idea.</p>