akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Whether distributed RPC supports collective communication?

Midhilesh: <p>I am currently studying distributed RPC for hybrid parallelism. From the documentation, I figured out RPC supports TensorPipe backend and it is a point-to-point communication. But for hybrid parallelism, I need all-to-all collective communication. Are there any ways for implementing hybrid parallelism with collective communication using distributed RPC?.</p>
<p>I kindly request anyone to provide a solution for this issue.</p>

agolynski: <p>cc Luca <a class="mention" href="/u/lcw">@lcw</a></p>
<p>I think the main usecase of tensorpipe is not collectives, you can use other solutions, e.g. NCCL, GLOO, UCC.</p>
<p>Luca, are there plans for tensorpipe to be a backend for such collectives?</p>

lcw: <p>Yes, correct, we currently don’t provide a way to do collectives on top of RPC/TensorPipe. The rationale is that the “native” collective libraries (NCCL, Gloo, MPI) are already doing a much better job at this, hence we’re not optimizing TensorPipe and RPC for that use case. However you should be able to combine RPC with the collective libraries very easily. Here is a tutorial showing how to do so with DDP, but if you prefer to use the “lower-level” API that should work too: <a href="https://pytorch.org/tutorials/advanced/rpc_ddp_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Combining Distributed DataParallel with Distributed RPC Framework — PyTorch Tutorials 1.8.1+cu102 documentation</a></p>
I want to figure out how Pytorch copy data from one GPU to another when the board of a server doesn't support P2P

111507: <p>Hi all!</p>
<p>I’m just wondering how Pytorch copy data from one GPU to another when the board of your server doesn’t support P2P.</p>
<p>It it clear that we can copy Tensor by using “.to()” function.</p>
<p>But, I want to understand the process in detail.</p>
<p>What I’ve found is that Pytorch use ‘cudaMemcpy’ or ‘cudaMemcpyAsync’ function at lower Level.</p>
<p>However, if the board of a server doesn’t support P2P, then it is impossible to use ‘cudaMemcpy’ and ‘cudaMemcpyAsync’ to copy data to another GPU.</p>
<p>Without UVA(Unified Virtual Addressing), It is unable to copy data from a GPU to another GPU by using ‘cudaMemcpy’ and ‘cudaMemcpyAsync’.</p>
<p>What made me crazy was this <a href="https://discuss.pytorch.org/t/pytorch-with-cuda-unified-memory/60783">Pytorch.forum link</a> saying Pytorch does not support UVA!</p>
<p>I would really appreciate it if you tell me how Pytorch ‘to()’ function work at the C language level on the Non-p2p board.</p>
<p>Thank you.</p>
Non gradient-trackable convolutions

spacemeerkat: <p>Hi all!</p>
<p>I’m wondering <strong>is it possible to perform convolutions during the training of neural networks in which a convolution kernel is detached from the gradient tree</strong>.</p>
<p>An example would be the autoencoder shown below:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/9/d9cf3cf246e1e1df36502b94add17d6ece50ee2f.png" data-download-href="https://discuss.pytorch.org/uploads/default/d9cf3cf246e1e1df36502b94add17d6ece50ee2f" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/9/d9cf3cf246e1e1df36502b94add17d6ece50ee2f.png" alt="image" data-base62-sha1="v4PCef6FU5drt5hR1SA0XWserF5" width="690" height="222" data-dominant-color="F1F2F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1272×410 8.25 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>In the example above, a convolution of the reconstructed image (exiting the decoder CNN subnet) is performed using a 2D kernel which is detached from the gradient tree.</p>
<p>In this way, the encodings (exiting the encoder CNN subnet) would be learned parameters/embeddings necessary to reconstruct an <em>unconvolved</em> image prior to the untracked convolution.</p>
<p>Forgetting use cases and why you may want to do this, my question is whether a torch.nn.ConvNd operation using a gradient-tree-detached kernel is something that the API allows, without breaking the gradient tree?</p>
<p>I know this is possible for simple operations such as multiplying/summing/subtracting/dividing parameters/weights by untracked tensors, but I wonder if it still holds for operations which use more complex operations rooted in the torch API, like convolutions.</p>
<p>Many thanks in advance for all your help!</p>

ptrblck: <p>Yes, that’s possible.<br>
You could either use the functional API with a kernel specified as a plain tensor (not an <code>nn.Parameter</code>):</p>
<pre><code class="lang-python">kernel = torch.randn(...)
out = F.conv2d(input, kernel, ...)
</code></pre>
<p>or set the <code>requires_grad</code> attribute of the <code>weight</code> (and <code>bias</code>) in the conv layer to <code>False</code>.</p>
Lost args when export ONNX model

MerlinLi: <p>I try to export a Decoder model of a Seq2Seq model. The definition of the class is as follow:</p>
<pre><code class="lang-auto">class Decoder(MarianPreTrainedModel):
    def __init__(self, config: MarianConfig):
        super().__init__(config)
        self.dropout = config.dropout
        self.layerdrop = config.decoder_layerdrop
        self.padding_idx = config.pad_token_id
        self.max_target_positions = config.max_position_embeddings

        self.embed_positions = MarianSinusoidalPositionalEmbedding(
            config.max_position_embeddings,
            config.d_model,
            self.padding_idx,
        )
        self.layers = nn.ModuleList([MarianDecoderLayer(config) for _ in range(config.decoder_layers)])
        self.init_weights()
    ....
    def forward(
        self,
        inputs_embeds: torch.Tensor,
        encoder_hidden_states: torch.Tensor,
        encoder_attention_mask: torch.Tensor,
        decoder_past_key_values: PAST_KEY_VALUES_TYPE = None,
        encoder_past_key_values: PAST_KEY_VALUES_TYPE = None,
    ):
    ....
</code></pre>
<p>The Module is modified from huggingface MarianMTModel.<br>
When I export the model to ONNX, the eported model lost the arg <code>encoder_hidden_states</code>.<br>
I have debug it in the export funtion, and I find the arg lost after this line:</p>
<pre><code class="lang-auto">params_dict = torch._C._jit_pass_onnx_eliminate_unused_items(graph, params_dict)
</code></pre>
<p>The 509th line in torch.onnx.utils.py<br>
Before the funtion, I print the graph:</p>
<pre><code class="lang-auto">graph(%decoder_input_embeds : Float(*, *, 512, strides=[1024, 512, 1], requires_grad=0, device=cpu),
      %encoder_hidden_states : Float(*, *, 512, strides=[3584, 512, 1], requires_grad=0, device=cpu),
      %encoder_attention_mask : Long(*, *, strides=[7, 1], requires_grad=0, device=cpu),
      %decoder_cache_values : Float(*, 8, *, 64, strides=[1536, 192, 64, 1], requires_grad=0, device=cpu),
      %encoder_cache_values : Float(*, 8, *, 64, strides=[3584, 448, 64, 1], requires_grad=0, device=cpu),
      %embed_positions.weight : Float(512, 512, strides=[512, 1], requires_grad=0, device=cpu),
      %layers.0.self_attn.k_proj.weight : Float(512, 512, strides=[512, 1], requires_grad=1, device=cpu),
</code></pre>
<p>After the function, I print the graph:</p>
<pre><code class="lang-auto">graph(%decoder_input_embeds : Float(*, *, 512, strides=[1024, 512, 1], requires_grad=0, device=cpu),
      %encoder_attention_mask : Long(*, *, strides=[7, 1], requires_grad=0, device=cpu),
      %decoder_cache_values : Float(*, 8, *, 64, strides=[1536, 192, 64, 1], requires_grad=0, device=cpu),
      %encoder_cache_values : Float(*, 8, *, 64, strides=[3584, 448, 64, 1], requires_grad=0, device=cpu),
      %embed_positions.weight : Float(512, 512, strides=[512, 1], requires_grad=0, device=cpu),
</code></pre>
<p>So, why the arg <code>encoder_hidden_states</code> lost? Some approaches to solve this?</p>
Is there a way to reindex a matrix?

wanjunhong0: <p>Let’s say a index matrix or embedding lookup matrix is not started from 0 to n.</p>
<pre><code class="lang-auto">a = torch.randint(10, 100, (10, 10))
old_index = torch.unique(a)
new_index = torch.arange(len(old_index))
</code></pre>
<p>is there way to map the new index to the matrix in Pytorch?</p>
<p>Something like this in Pandas</p>
<pre><code class="lang-auto">a.map(dict(zip(old_index, new_index)
</code></pre>
<p>or numpy</p>
<pre><code class="lang-auto">a = torch.randint (10, 100, (10, 10))
b = a.numpy()
old_index = np.unique(b)
new_index = np.arange(len(old_index))
index_map = dict(zip(old_index, new_index))

np.vectorize(index_map.get)(b)
</code></pre>

KFrank: <p>Hi Junhong!</p>
<aside class="quote no-group quote-modified" data-username="wanjunhong0" data-post="1" data-topic="118284" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wanjunhong0/48/37152_2.png" class="avatar"> wanjunhong0:</div>
<blockquote>
<p>is there way to map the new index to the matrix?</p>
</blockquote>
</aside>
<p>If I understand your use case, <code>.scatter_()</code> should work:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.7.1'
&gt;&gt;&gt; _ = torch.manual_seed (2021)
&gt;&gt;&gt; a = torch.randint (10, 100, (10, 10))
&gt;&gt;&gt; old_index = torch.unique (a)
&gt;&gt;&gt; new_index = torch.arange (len (old_index))
&gt;&gt;&gt; imap = -torch.ones (a.numel()).long()
&gt;&gt;&gt; imap.scatter_ (0, old_index, new_index)
tensor([-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,  0,  1, -1, -1,  2, -1, -1,
         3,  4,  5, -1,  6, -1,  7,  8,  9, -1, -1, 10, -1, 11, -1, 12, 13, 14,
        -1, 15, -1, -1, 16, 17, 18, 19, -1, 20, 21, 22, 23, -1, 24, 25, 26, 27,
        -1, 28, 29, 30, 31, 32, 33, 34, -1, 35, 36, 37, 38, 39, 40, 41, 42, 43,
        44, -1, 45, -1, 46, 47, 48, 49, -1, 50, -1, -1, 51, -1, -1, 52, 53, 54,
        -1, 55, 56, -1, -1, 57, -1, 58, -1, 59])
&gt;&gt;&gt; old_index[50]
tensor(81)
&gt;&gt;&gt; imap[old_index[50]]
tensor(50)
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

wanjunhong0: <p>Hi Frank,</p>
<p>I can’t quite follow you. What I want is to map index matrix a.<br>
This is how to do it in numpy, but I just wonder is there a way to do it in Pytorch？</p>
<pre><code class="lang-auto">a = torch.randint (10, 100, (10, 10))
b = a.numpy()
old_index = np.unique(b)
new_index = np.arange(len(old_index))
index_map = dict(zip(old_index, new_index))

np.vectorize(index_map.get)(b)
</code></pre>
How to train only middle component?

oasjd7: <pre><code class="lang-auto"># train encoder and classifier.
out = encoder(input)
pred = classifier(out)
ce_loss = ce(pred, target)
total_loss += ce_loss

# In this train phase, I want to train only encoder_2
out = encoder(input.detach())
out_2 = encoder_2(out)
pred_2 = classifier(out_2.detach())
ce_loss2 = ce(pred_2, target)
total_loss += ce_loss2

optimizer.zero_grad()
total_loss.backward()
optimizer.step()

</code></pre>
<p><code>Flow: Encoder1 -&gt; Encoder2 -&gt; Classifier.</code></p>
<p>I want to train <code>Encoder1</code> and <code>Classifier</code> with only <code>ce_loss</code> and train <code>Encoder2</code> solely with <code>ce_loss2</code>.<br>
In this case, how to freeze <code>encoder</code> and <code>classifier</code> when I train <code>Encoder2</code> ?</p>
<p>I don’t know which one should I use  [<code>detach</code>, <code>no_grad()</code>, <code>param.requires_grad = grad_on</code>] ?</p>

crcrpar: <p>Hi,</p>
<p>I think there are a few related threads</p>
<ul>
<li><a href="https://discuss.pytorch.org/t/update-only-sub-elements-of-weights/29101">Update only sub-elements of weights - autograd - PyTorch Forums</a></li>
<li><a href="https://discuss.pytorch.org/t/passing-a-subset-of-the-parameters-to-an-optimizer-equivalent-to-setting-requires-grad-of-subset-only-to-true/42866">Passing a subset of the parameters to an optimizer equivalent to setting requires_grad of subset only to True? - autograd - PyTorch Forums</a></li>
</ul>
How to disactivate a layer at evaluation

Yasser_Dahou: <p>Hi,</p>
<p>I am training a ResNet18 with a graph neural network on top of it, I want to discard the GAT layer at evaluation, can someone help me with this?</p>
<pre><code>def forward(self, x, lin=0, lout=5):
    out = x
    if lin &lt; 1 and lout &gt; -1:
        out = self.conv1(out)
        out = self.bn1(out)
        out = F.relu(out)
    if lin &lt; 2 and lout &gt; 0:
        out = self.layer1(out)
    if lin &lt; 3 and lout &gt; 1:
        out = self.layer2(out)
    if lin &lt; 4 and lout &gt; 2:
        out = self.layer3(out)
    if lin &lt; 5 and lout &gt; 3:
        
         #This part
        out = out.view(out.size(0), -1)
        edges = self.topology(self.knn(out),512)
        graph_data = (out,edges,edges)
        out,edge,attention = self.gat_net(graph_data)
        out = out.squeeze()
        out = torch.reshape(out,(512,256,8,8))
        # to here,  I don't want it to perform at evaluation
        out = self.layer4(out)
    if lout &gt; 4:
        #out = F.avg_pool2d(out,2)
        #res = out
        out = F.adaptive_avg_pool2d(out, (1, 1))
        out = out.view(out.size(0), -1)
        out = self.linear(out)

    return out
</code></pre>
<p>Thanks</p>

ptrblck: <p>You could add a condition using the internal <code>self.training</code> argument:</p>
<pre><code class="lang-python">        out = torch.reshape(out,(512,256,8,8))
        # to here,  I don't want it to perform at evaluation
        if self.training: # will only be called during training
            out = self.layer4(out)
</code></pre>
<p>The <code>self.training</code> argument is set to <code>True</code> or <code>False</code> by calling <code>model.train()</code> and <code>model.eval()</code>, respectively.</p>
Need feature maps of ResNet50

jS5t3r: <p>I want to get the feature maps of each input image from each layer after an activation function. The underlying model is ResNet50 (<a href="https://github.com/weiaicunzai/pytorch-cifar100/blob/master/models/resnet.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch-cifar100/resnet.py at master · weiaicunzai/pytorch-cifar100 · GitHub</a>)</p>
<p>In contrast, the VGG Net (<a href="https://github.com/weiaicunzai/pytorch-cifar100/blob/master/models/vgg.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch-cifar100/vgg.py at master · weiaicunzai/pytorch-cifar100 · GitHub</a>) has a <code>self.features</code> variable where the feature map extraction is easy.</p>
<p>I cannot use this <code>self.features</code> variable, because I also need to extract features within the <code>nn.Sequential</code> block, which is not possible with  <code>self.features</code>  on my point of view.</p>
<p>After googling, I have two possibilities:</p>
<ol>
<li>Hooks</li>
<li>Iterating through the graph by using <code>model.children()</code> as here <a href="https://www.programmersought.com/article/23176813233/" class="inline-onebox" rel="noopener nofollow ugc">Use Pytorch and Matplotlib to visualize the features of convolutional neural networks - Programmer Sought</a>
</li>
</ol>
<p>Can someone tell me, what is the best way to the get the feature maps of a ResNet50? I need these features for further computations.</p>

ptrblck: <p>I usually use forward hooks as described <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">here</a>, which can store the intermediate activations. You could then pass these activations to further processing.</p>
How to predict the values between 0 and 1 except using MSE loss?

oasjd7: <p>I have target labels which is between 0 and 1.<br>
(ex. [0.2, 0.3, 0.0, 0.4])</p>
<p>This is a kind of regression problem, but I can’t use MSE loss because of some constraints.<br>
Is there any way to predict the values between 0 and 1 ?</p>

ptrblck: <p>Yes, <code>nn.BCEWithLogitsLoss</code> for example takes targets in the range <code>[0, 1]</code> and could thus be used.<br>
This use case would be applied for “soft” targets, which are between the two binary target classes 0 and 1.</p>
Plotting tensors using plt

Merwa_Mehmood: <p>I am trying to plot a tensor in which EIGEN_VALUE when added to the latent variable generates the image.but i am getting this output<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/6/0672e2eb69c9194c1f4a1417e70bab27d16d083b.png" alt="Screenshot (135)" data-base62-sha1="V30u1cf8AE2NjEHiGoZKwsGR3Z" width="373" height="218"><br>
import matplotlib.pyplot as plt</p>
<p>#<span class="hashtag-raw">#Latent</span> code<br>
code_1 = torch.randn(1, generator.z_space_dim).cuda()<br>
code_1 = generator.layer0.pixel_norm(code_1)<br>
code_1.shape</p>
<h2><a name="small-alpha-leads-to-good-results-1" class="anchor" href="#small-alpha-leads-to-good-results-1"></a>small alpha leads to good results</h2>
<p>alpha_1=0.0000005</p>
<p>CODE =code_1.cpu().detach()<br>
CODE.shape<br>
EIGEN_VALUE= np.matrix(u1)<br>
EIGEN_T = torch.from_numpy(n)</p>
<p>FINAL = CODE + alpha_1 * EIGEN_T</p>
<p>#<span class="hashtag-raw">#ploting</span><br>
plt.figure(figsize=(3,3))<br>
plt.imshow(FINAL,cmap=‘gray’)</p>

ptrblck: <p>What would be the expected output, i.e. do you assume that the dimensions are wrong or any other issue?</p>
Is there a way to call Pytorch model inside Cuda C kernel

Emerald01: <p>I have a use case like this, it is like an reinforcement learning rollout that the neural network is in charge of producing part of the data d1, and the rest of the data s1 will be created accordingly by observing d1 (simple example, maybe s1 = a_function(d1)), and so on so forth, right now I have a Cuda C kernel that can take care of s1 generation if d1 is given, meaning that this <code>a_function</code> has been implemented in Cuda C, something like this</p>
<pre><code class="lang-auto">(d0, s0) ==&gt; nn model ==&gt; d1 ==&gt; Cuda C create s1 = a_function(d1) ==&gt; (d1, s1) ==&gt; nn model .....
</code></pre>
<p>You can see that in order to make this work, so far what I can do is every time when I need to <code>create s</code> I have to call the Cuda C kernel, and the code is as a result switching between <code>nn model</code> and <code>Cuda C</code> kernel. This seems not efficient.</p>
<p>Ideally, if Cuda C could call <code>nn model</code> directly, e.g., pass the model pointer or something, then the entire loop above can be sitting inside one Cuda C kernel. The a for loop in a Cuda C kernel can work out everything, but I am not sure if this is possible.</p>

ptrblck: <p>I don’t fully understand the use case.<br>
PyTorch modules are not single CUDA kernels, but are implemented in e.g. Python, call into the C++ backend, and finally dispatch to the corresponding CUDA kernels.<br>
“Calling nn model directly” in a CUDA kernel would assume that “model” is a single kernel.<br>
Could you explain your use case a bit more?</p>
Adding typing to __call__ of nn.Module?

hughperkins: <p>Hi, I want autocomplete and type checking working when calling into a customer nn.Module. Of course any parameters and types on <code>forward</code> don’t get mirrored to <code>__call__</code>. I’m wondering whether simply definining <code>__call__</code>, with the correct parameter names and types, and calling <code>super().__call__</code> from that is a legitimate approach to this, or whether there might be some issues sometimes, and what those issue could be?</p>
<pre><code>import torch
from torch import nn


class Foo(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.h = nn.Linear(1, 1)

    def __call__(self, param1: torch.Tensor) -&gt; torch.Tensor:
        return super().__call__(param1)

    def forward(self, param1: torch.Tensor):
        print('Foo.forward')
        param1 = self.h(param1)
        return param1


foo = Foo()
outputs = foo(param1=torch.rand(1))
print('outputs', outputs)
</code></pre>
<p>This code runs correctly, type-checks correctly etc. Thoughts?</p>

ptrblck: <p>I can’t think of a breaking use case at the moment, as hooks, <code>nn.DataParallel</code>, etc. should all work in your approach. This is of course not a valid verification, so let me think about some more use cases.</p>
JNI does not support GPU?

ajmssc: <p>When using JNI from Java, it seems not possible to allocate tensor objects to the GPU, and pytorch JNI will always allocate the tensors to CPU.<br>
It’s also not possible to move the model to GPU memory, or if the traced model uses a GPU, it’s not possible to move it back to CPU.</p>
<p>Can someone confirm that my understanding is correct?</p>

saudet: <p>It’s possible to that using the C++ API, which we can now use from Java with the JavaCPP Presets for PyTorch:</p><aside class="onebox githubfolder">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">
      <a href="https://github.com/bytedeco/javacpp-presets/tree/master/pytorch" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars.githubusercontent.com/u/5856852?s=400&amp;amp;v=4" class="thumbnail onebox-full-image" width="60" height="60">

<h3><a href="https://github.com/bytedeco/javacpp-presets/tree/master/pytorch" target="_blank" rel="noopener nofollow ugc">bytedeco/javacpp-presets</a></h3>

<p><a href="https://github.com/bytedeco/javacpp-presets/tree/master/pytorch" target="_blank" rel="noopener nofollow ugc">master/pytorch</a></p>

  <p><span class="label1">The missing Java distribution of native C++ libraries</span></p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Does Pytorch support training on android devices ( with java codes)

Shuo_Zhang: <p>Wonder if pytorch support training on android devices to perform federated learning in practice</p>

IvanKobzarev: <p>At the moment we do not support training in pytorch mobile. We are looking on it. Stay tuned.</p>

saudet: <p>Regarding Java code, as long as we can build the C++ API for Android, we can use that from Java with the JavaCPP Presets for PyTorch:</p><aside class="onebox githubfolder">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">
      <a href="https://github.com/bytedeco/javacpp-presets/tree/master/pytorch" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars.githubusercontent.com/u/5856852?s=400&amp;amp;v=4" class="thumbnail onebox-full-image" width="60" height="60">

<h3><a href="https://github.com/bytedeco/javacpp-presets/tree/master/pytorch" target="_blank" rel="noopener nofollow ugc">bytedeco/javacpp-presets</a></h3>

<p><a href="https://github.com/bytedeco/javacpp-presets/tree/master/pytorch" target="_blank" rel="noopener nofollow ugc">master/pytorch</a></p>

  <p><span class="label1">The missing Java distribution of native C++ libraries</span></p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
How can I use dot-product attention on two cnn features?

A11en0: <p>there are two features from two CNN, and they share the same parameters, in my case, its shape of &lt;128, 764&gt;. I want to add dot-product attention on them, how can I implement them in PyTorch.</p>
<p>below picture contains my attention score function:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/4/54159aacd715d0b213d04111908895074e203746.png" alt="image" data-base62-sha1="bZQorHiz6X72lGDJDAR04gKB0r4" width="625" height="174"></p>
<p>and W denotes the learning parameters.</p>

A11en0: <pre><code>    def attention_cnn(self, c, q):
        ......
        alpha = F.softmax(u, -1)  # batch_size, concept_seq_len
        
        return alpha
</code></pre>
<p>framework maybe like this, alpha is the attention weight.</p>
LSTM runtime shape error in evaluation - training is fine

user12233: <p>I’m able to successfully train a LSTM model but when I tried to evaluate it it throws a runtime shape error <code>RuntimeError: shape '[3, 1, 5000] is invalid for input of size 5000</code> on the last batch.  My test data is 4776 samples and each sample has 5000 features.</p>
<pre><code>with torch.no_grad():
n_correct = 0
n_samples = 0
for x_batch, y_batch in test_loader:
    x_batch = x_batch.reshape(batch_size,  1, input_dim).to(device)
    y_batch = y_batch.to(device)
    outputs = model(x_batch)
    # max returns (value ,index)
    _, predicted = torch.max(outputs.data, 1)
    n_samples += y_batch.size(0)
    n_correct += (predicted == y_batch).sum().item()
    print(f'Testing accuracy: ' + str( 100.0 * n_correct / n_samples) + " samples tested: " + str(n_samples))

acc = 100.0 * n_correct / n_samples
print(f'Accuracy of the network on the 4776 test samples: {acc} %')
</code></pre>
<p>Below is the last output I get from my print statement:</p>
<pre><code>What is the batch size: torch.Size([3, 1, 5000])
Testing accuracy: 96.5011523151058 samples tested: 4773
</code></pre>
<p>I’m confused as to how I could train the model to completion using the same dimensions but it fails on the last batch in the evaluation.  According to the error is seems that only one sample is in that last batch.</p>

god_sp33d: <p>I guess it is because the total number of samples is not divisible by the batch_size. So, when you are creating the DataLoader there is this argument drop_last. This is by default set to False. Try changing it to True.</p>

user12233: <aside class="quote no-group" data-username="god_sp33d" data-post="2" data-topic="118351">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/e495f1/48.png" class="avatar"> god_sp33d:</div>
<blockquote>
<p>drop_last</p>
</blockquote>
</aside>
<p>Thanks for the help!  It did run to completion without the error but the test dataset has 4776 samples and that is divisible by the batch size (3).</p>
Inserting and popping to a Tensor?

Sam_Lerman: <p>Let’s say I have two tensors:</p>
<pre><code class="lang-auto">A = torch.empty([N, M]).to(device)
B = torch.empty([batch_size, M]).to(device)
</code></pre>
<p>The desired effect I want to achieve is this:</p>
<p><code>A = torch.cat((B.to(device), A[:-batch_size])).to(device)</code></p>
<p>In plain English, I just want to insert B into the front of A but keep A the same size.</p>
<p>However, when I do it with the following code:</p>
<pre><code class="lang-auto">A[batch_size:].data = A[:-batch_size].data.to(device)
A[batch_size:].data = B.data.to(device)
</code></pre>
<p>Now I get a CUDA error:</p>
<p><code>RuntimeError: CUDA error: device-side assert triggered</code></p>
<p>The assert it’s referring seems to be:</p>
<p><code>Assertion </code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; “index out of bounds”<code> failed.</code></p>

god_sp33d: <p>I don’t think I got your question properly. But is this what you are trying?</p>
<pre><code class="lang-auto">a = torch.empty(10,2).to(device)
b = torch.empty(5,2).to(device)
batch_size = 5
a[:batch_size, :] = b
</code></pre>
<p>Here, the device can be any “cpu” or “cuda”. numbers 10,5,2 are random examples for demonstration (instead of using N, batch_size, M, but the order remains same).</p>

Sam_Lerman: <p>Close, but no, I’m trying to implement a kind of queue. So first in, first out. Your 4th line of code overwrites the top of <code>a</code>, but I want to shift <code>a</code> so that the top is preserved and insert <code>b</code> before it, if that makes sense.</p>

Sam_Lerman: <p>Oh but I see your point! Yes, I would do</p>
<pre><code class="lang-auto">A[batch_size:].data = A[:-batch_size].data.to(device)
A[:batch_size].data = B.data.to(device)
</code></pre>
<p>I missed that that colon was on the wrong side.</p>

god_sp33d: <p>yeah, is it solved then ?</p>

Sam_Lerman: <p>Never mind, still getting the same CUDA error, which I don’t get when I do:</p>
<p><code>A = torch.cat((B.to(device), A[:-batch_size])).to(device)</code></p>

Sam_Lerman: <p>I changed it a bit and now I’m getting:</p>
<pre><code class="lang-auto">A[batch_size:] = A[:-batch_size].to(device)
</code></pre>
<p><code>RuntimeError: unsupported operation: some elements of the input tensor and the written-to tensor refer to a single memory location. Please clone() the tensor before performing the operation.</code></p>
<p>But that’s exactly what’s shown being done here: <a href="https://stackoverflow.com/questions/51761806/is-it-possible-to-create-a-fifo-queue-with-pytorch" class="inline-onebox" rel="noopener nofollow ugc">python - Is it possible to create a FIFO queue with pyTorch? - Stack Overflow</a></p>
How to use dataloader with large dataset?

juhyung: <p>Hi,</p>
<p>I have 300G size h5 file that has images, texts.</p>
<p>and I’m having trouble with dataloader. for example, in the code,</p>
<pre><code class="lang-python">class loader(Dataset):
    def __init__():
        self.file = load(my_300G_file)
    def __getitem(self, index):
        return self.file[index]
</code></pre>
<p>Will I get memory error? because I load 300g file? (my ram is 8G)<br>
If so, how can i load dataset?</p>

ptrblck: <p>If you try to load the whole file at once, your system will run out of memory.<br>
You could use pandas to load slices:</p>
<pre><code class="lang-auto">df = pd.DataFrame(np.random.rand(100, 1))
df.to_hdf('file.h5', 'df', format='table')
pd.read_hdf('file.h5', start=0, stop=1)
</code></pre>
<p>You could use the <code>index</code> to set the <code>start</code> and <code>stop</code> values accordingly.</p>

juhyung: <p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Thanks, so the code I wrote will not work because of memory right??<br>
Then, Should I use the pandas load method when I define dataset class?</p>
<pre><code class="lang-python">class loader(Dataset):
    def __init__():
        df = pd.DataFrame(np.random.rand(100, 1))
        df.to_hdf('file.h5', 'df', format='table')
        pd.read_hdf('file.h5', start=0, stop=1)
</code></pre>
<p>like this?</p>

ptrblck: <p>No, I just created a dummy dataframe and saved it to Disc so that I could load it with <code>pd.read_hdf</code>.<br>
You should add the read line to your <code>__getitem__</code> method and store the file path in <code>__init__</code>.</p>

juhyung: <p>I see,<br>
then if i just store the file path in <code>__init__</code> and call indexed h5 in <code>__getitem__</code> like <code>[0:10]</code><br>
then computer will just assign <code>[0:10]</code> data memory not the whole h5 file.<br>
am i right?</p>

ptrblck: <p>Yes, that’s the plan. You would need to use the method I’ve posted as you can’t directly index the h5 file.</p>

anuradhawick: <p>Hi, I have a similar scenario. I analyse genomic data and each sequence can be represented in a vector of about 50-1000 dimensions. Larger the better. So in gist;<br>
I generate features using a different pipeline and gives a text file, vector per line.</p>
<p>In this case, do I have to save vectors to an h5 file somehow? Any help would be greatly appreciated.</p>
How to print the computed gradient values for a model pytorch?

Fsajad94: <p>Hello everyone,<br>
I want to know the value of the gradient in each iteration of the algorithm, but I do not have any idea. Could you help me?</p>
<p>import matplotlib.pyplot as plt<br>
import numpy as np</p>
<p>import torch<br>
import torch.optim as optim<br>
import torch.nn as nn</p>
<p>n=500<br>
d=2<br>
np.random.seed(42)<br>
x=np.random.randn(n,d)<br>
epsilon=np.random.randn(n,1)<br>
b=np.ones([1,1])<br>
w=2<em>np.ones([1,d])<br>
x0=np.ones([n,1])<br>
y=b+np.dot(x,np.transpose(w))+.1</em>epsilon<br>
idx=np.arange(n)<br>
np.random.shuffle(idx)<br>
idx_train=idx[:int(.8<em>n)]<br>
idx_val=idx[int(.8</em>n):]<br>
x_train,y_train=x[idx_train],y[idx_train]<br>
x_val,y_val=x[idx_val],y[idx_val]</p>
<p>device = ‘cuda’ if torch.cuda.is_available() else ‘cpu’<br>
x_train_tensor = torch.as_tensor(x_train).float().to(device)<br>
y_train_tensor = torch.as_tensor(y_train).float().to(device)<br>
torch.manual_seed(13)<br>
model = nn.Sequential(nn.Linear(d, 1)).to(device)<br>
losses=[]<br>
n_epochs=1<br>
lr=.1<br>
loss_fn = nn.MSELoss(reduction=‘mean’)<br>
optimizer = optim.SGD(model.parameters(), lr=lr)<br>
for epoch in range(n_epochs):<br>
model.train()<br>
yhat = model(x_train_tensor)<br>
loss = loss_fn(yhat, y_train_tensor)<br>
t=loss.item()<br>
losses.append(t)<br>
loss.backward()<br>
optimizer.step()<br>
optimizer.zero_grad()</p>

god_sp33d: <p>Can you see if this helps? <a href="https://discuss.pytorch.org/t/how-to-print-the-computed-gradient-values-for-a-network/34179">here</a></p>
CNN: array dimensions from convolution make no sense to me

newtopytorch: <p>Hi. I have a tensor which is made up of 8 matrices, and each matrix is 20x20 numbers. I am trying to do a convolution on this matrix but I don’t understand the output dimensions. Here is what I have so far (the x is my tensor):</p>
<blockquote>
<p>class CNN(nn.Module):</p>
<pre><code>def __init__(self, n_filters, filter_sizes):

    super(CNN, self).__init__()

    self.conv0 = nn.Conv2d(in_channels=1, out_channels=n_filters[0], kernel_size=filter_sizes[0])

def forward(self, x, lengths=None):
    x = x.unsqueeze(1)

    map0_conv = F.relu(self.conv0(x))
    map0_avg = torch.mean(map0_conv, dim=1)
</code></pre>
</blockquote>
<p>My filter size is 2x2, and the stride is 1. Therefore, for each matrix I need 19*19 = 361 filters. Thus, starting with an 8x20x20 tensor, I end up with the dimensions of self.conv0(x) being 8x361x19x19.</p>
<p>I don’t know what this result means. It’s causing me problems in the next step, the torch.mean. What I’m trying to do, is to take a 8x20x20 tensor, apply a convolution to it and then get the average of each convoluted matrix to end up with one vector containing 8 entries. Clearly I’m doing it wrong but I don’t know how to fix it.</p>

god_sp33d: <p>Hi,</p>
<p>So, when you are doing x.unsqueeze(1), you are basically passing an input size as (8,1,20,20). Now, the output dimensions of your conv operation depends on the number of out_channels you pass in.<br>
so, let’s say you pass in out_channels=1, then you end up with an output size of (8,1,19,19). [19 because you have a kernel of size 2x2 and stride as 1]. In short, the output size (8, x, 19, 19) where x is a variable depending on the number of out_channels you pass.</p>
<p>Finally, for the mean to be a vector of 8 entries, all you need to do is calculate the average across axis=1,2,3.</p>
<pre><code class="lang-auto">map0_avg = torch.mean(map0_conv, dim=[1,2,3])
</code></pre>
<p>the results would be a vector of size 8.</p>
Scatter concatenation?

XT_Li: <p>Updates:</p>
<p>I came with something like this:</p>
<pre><code class="lang-auto">import torch
from torch.nn.utils.rnn import pad_sequence

idx = torch.tensor([[0, 0, 1, 1], [0, 0, 0, 1], [2, 2, 3, 3], [2, 2, 2, 2]])
feat = torch.tensor([[11, 12, 21, 22], [13, 14, 15, 23], [31, 32, 41, 42], [33, 34, 35, 36]])

# turn index into one hot
onehot = torch.zeros(4, 4, 4)
onehot.scatter_(0, idx.view(1, 4, 4), 1)

# extract features for each cluster
onehot = onehot.view(4,-1)
feat = feat.view(1, -1)
sub = torch.masked_select(feat, onehot == 1)

# padding
cluster_size = torch.sum(onehot, dim=1).long().numpy().tolist()
clusters = torch.split(sub, cluster_size)
padded_feat = pad_sequence(clusters, batch_first=True)
print(padded_feat)
</code></pre>
<p>It seems working, although it’s impossible to be batched.</p>
<hr>
<p>Original post:</p>
<p>I don’t know if scatter concatenation is the correct name, but I would like to do the following operation:</p>
<p>Suppose I have an index matrix Idx and a feature matrix F:</p>
<pre><code class="lang-auto">Idx = [[0, 0, 1, 1], [0, 0, 0, 1], [2, 2, 3, 3], [2, 2, 2, 2]]
F = [[a1,a2,b1,b2], [a3,a4,a5,b3], [c1,c2,d1,d2], [c3,c4,c5,c6]]
</code></pre>
<p>I would like to take the features of each cluster defined in the index and put them in a row with padding wherever necessary, so the result looks like:</p>
<pre><code class="lang-auto">out = [[a1,a2,a3,a4,a5,0], [b1,b2,0,0,0,0], [c1,c2,c3,c4,c5,c6], [d1,d2,0,0,0,0]]
</code></pre>
<p>It’s like a <code>scatter</code> operation, but without the reduction. The order of each row in the output doesn’t matter, i.e. <code>[a1,a2,a3,a4,a5,0]</code> or <code>[a2,a1,a4,a3,a5,0]</code> are both Okay.</p>
<p>I suppose I can for loop and reconstruct each row in the output:</p>
<pre><code class="lang-auto">out = torch.zeros(4, 6)
for i in range(4):
    data = F[Idx==I]
    out[:, :data.shape[0]] = data
</code></pre>
<p>but I am wondering if there’s more efficient way to do this. Any suggestion would be appreciated.</p>
RNN for Video Classification

PCPJ: <p>I am trying to train a SqueezeNet with ConvLSTM cells to perform video classification on the 20BN-Jester dataset.<br>
I never played with a recurrent model, so I am a little bit confused. I am starting my experiments with only the firsts 100 videos from the dataset, just to see if the model is being able to overfit this 100 videos, but even that I am being able to do.<br>
My batches have shape (bs, seq_len 3, H, W). I am reshaping them to (bs* seq_len, 3, H, W), feeding through the squeeze net encoder, them reshaping to (bs, seq_len 512, H, W) so I can feed it to the convLSTM cell, reshaping back to (bs*seq_len, 512, H, W) to feed through the squeeze net classifier.</p>
<p>The implementation of convLSTM cell I am using is this one: <a href="https://github.com/ndrplz/ConvLSTM_pytorch/blob/master/convlstm.py" rel="noopener nofollow ugc">https://github.com/ndrplz/ConvLSTM_pytorch/blob/master/convlstm.py</a><br>
Here is the code of my model:</p>
<pre><code class="lang-auto">class SqueezeNet(nn.Module): 
    def __init__(self, n_classes=27):
        super(SqueezeNet, self).__init__()
        
        self.n_classes = n_classes

        sq = torchvision.models.squeezenet.squeezenet1_0(pretrained=False)
        self.sq_feaures = sq.features
        self.convLSTM = ConvLSTM(512, [512], [(3,3)], 1, True, True, False) # one convLSTF cell, with input size of 512, output size of 512, and 3x3 kernels.
        self.sq_classifier = sq.classifier
        self.sq_classifier[1] = nn.Conv2d(512, n_classes, kernel_size=(1, 1), stride=(1, 1)) 

        self.init_layers()

    def forward(self, images):
        """
        Forward propagation.
        :param images: images, a tensor of dimensions  (batch_size, seq_len, 3, H, W)
        :return: classes prdictions, a tensor of dimensions (N, T, n_classes)
        """
        b, s, c, h, w = images.shape
        images = images.view(b*s, c, h, w)
        
        out = self.sq_feaures(images) # (b*s, 512, H, W)
        _,_,h,w = out.shape
        out = self.convLSTM(out.view(b, s, 512, h, w))[0]
        out = out.view(b*s, 512, h, w) # (b*s, 512, H, W)
        out = self.sq_classifier(out) # (b*s, n_classes, 1, 1)
        return out.view(b, s, self.n_classes)

    def init_layers(self):
        for name, param in self.named_parameters():
            if name.endswith(".weight"):
                nn.init.xavier_uniform_(param)
            if name.endswith(".bias"):
                nn.init.constant_(param, 0.0)

</code></pre>
<p>Since the videos from Jester dataset haven’t the same number of frames (35 to 40), I am appending some copies of the last frame of each video at the end, so every video in the dataset has the same length. I am performing one prediction to each frame of each video, but I already tried one prediction for each video, and my loss just don’t decreases. This leads me to think that I am doing something really wrong, or the net work would be able to overfit such a small dataset (100 videos)<br>
Someone can see what I am doing wrong? Or have any hint?</p>
Optimizing the backward computational graph

ttoosi: <p>Hi,</p>
<p>I am trying to perform optimization on the backward computational graph. Put another way, I want to treat the backward computational graph as a forward path. This way, I would have two .grad for each parameter, one calculated by the regular forwad loss and the other for a backward loss.<br>
Here is a pseudo-code:</p>
<pre><code class="lang-auto">output1 = model(input)
loss1 = criterion(output1, targets1)
loss1.backward()

output2 = backward_comp_graph(output1.detach())
loss2 = criterion(output2, targets2)
loss2.backward()

</code></pre>
<p>I would appreciate your suggestions on what implementation strategy is the nicest!</p>
<p>Thanks,<br>
ttoosi</p>
How to manually obtain the minus log-likelihood?

felipeary91: <p>I’m implementing a VAE and I want to obtain the minus log-likelihood manually (not using an existing function). The given equation is <img src="https://discuss.pytorch.org/uploads/default/original/3X/8/1/81526da0c60e65962bcf7f57e17e2481ef3befcd.png" alt="Screenshot from 2021-04-15 22-24-06" data-base62-sha1="is2c6Z8s3DiQdSDwSf5dz30UplP" width="371" height="50">. Another equation for it (the one implemented below) is available at <a href="https://www.statlect.com/glossary/log-likelihood" class="inline-onebox" rel="noopener nofollow ugc">Log-likelihood</a> (couldn’t attach another picture). Been stuck in this for a couple of days now, don’t know where my code is wrong.</p>
<pre><code class="lang-auto">def loss_loglik(y_mean, y_logvar, x):
    out_1 = (x.size()[2]*x.size()[3] / 2) * np.log(2 * np.pi)
    out_2 = (x.size()[2]*x.size()[3] / 2) * torch.log(y_logvar.exp())
    x_diff = x - y_mean
    out_3 = torch.sum(x_diff.pow(2)) / (2 * y_logvar.exp())
    loss = out_1 + out_2 + out_3

</code></pre>
<p>The shape of the three arguments is (batch_size, 1, 28, 28).</p>
How to skip layer without weights

jun_suk_ha: <pre><code class="lang-auto">    def __init__(self):
        super(DNet, self).__init__()

        
        
        self.conv2_1 = nn.Conv2d(1, 2, 3, 1, 1)
        self.relu_1 = nn.ReLU()
        self.maxpool_1 = nn.MaxPool1d(2, 2)
        self.conv2_2 = nn.Conv2d(2, 3, 1, 1, 4)
        self.relu_2 = nn.ReLU()
        self.maxpool_2 = nn.MaxPool1d(2, 2)
        self.conv2_3 = nn.Conv2d(2, 3, 1, 0, 8)
        self.fc = nn.Linear(8, 1)

        self._weight_init()

    def _weight_init(self):
        for module in self.children():
            print(module.weight)
            if (module != nn.ReLU()):
#             module.weight = nn.init.kaiming_uniform()
                nn.init.kaiming_uniform_(module.weight)
                module.bias.data.fill_(0.0)
</code></pre>
<p>While looping, I want to skip my ReLU layer. How do I do that? I also tried</p>
<pre><code class="lang-auto">if module.weight is not None
</code></pre>

sio277: <p>Oh, I misunderstood. Try something like this:</p>
<pre><code class="lang-auto">def __init__(self):
    super(DNet, self).__init__()
    self.conv2_1 = nn.Conv2d(1, 2, 3, 1, 1)
    self.relu_1 = nn.ReLU()
    self.maxpool_1 = nn.MaxPool1d(2, 2)
    self.conv2_2 = nn.Conv2d(2, 3, 1, 1, 4)
    self.relu_2 = nn.ReLU()
    self.maxpool_2 = nn.MaxPool1d(2, 2)
    self.conv2_3 = nn.Conv2d(2, 3, 1, 0, 8)
    self.fc = nn.Linear(8, 1)

    self.apply(self._weight_init)

def _weight_init(self, m):
    if not isinstance(m, nn.ReLU):
        (your init code)
</code></pre>

sio277: <p>FYI, since you have other layers without parameters (such as <code>nn.MaxPool1d</code>), this would be better:</p>
<pre><code class="lang-auto">def _weight_init(self, m):
    if isinstance(m, (nn.Conv2d, nn.Linear)):
        (your init code)
</code></pre>
DataParallel on modules with dynamically overwritten forwards

tmflgp: <p>Hi. When I use DataParallel on a module whose <code>forward</code> has been overwritten at the instance level, and the new <code>forward</code> calls the original one, it errors out with <code>RuntimeError: arguments are located on different GPUS</code>. Here’s a minimal example:</p>
<pre><code class="lang-auto">def forward_wrapper(self, x):
    return self._forward(x)

net = nn.Linear(10, 10)
net._forward = net.forward
net.forward = MethodType(forward_wrapper, net)

net = nn.DataParallel(net).cuda()
x = torch.randn(8, 10).cuda()
net(x)
</code></pre>
<p>The following achieves the same thing and runs fine:</p>
<pre><code class="lang-auto">class ForwardWrapper(nn.Module):
    def __init__(self, module):
        self.module = module
    def forward(self, x):
        return self.module.forward(x)

net = nn.Linear(10, 10)
net = ForwardWrapper(net)

net = nn.DataParallel(net).cuda()
x = torch.randn(8, 10).cuda()
net(x)
</code></pre>
<p>So I’m not blocked by this. But I have no idea why one should work and not the other, and no idea what I would have to do if I perversely insisted on doing things the first way.</p>

razvanc92: <p>I experience the same problem. Did you managed to solve it?</p>

jsalbert1: <p>Did any of you manage to solve it?</p>
Split one LSTM model into two LSTM. Should it perform better?

Nick_ishere: <p>Dear experienced friends, I am trying to split one LSTM model into three sub-models, and I hope it could somehow improve the model performance. Suppose the shape of our original data is <code>[100, 7, 6]</code>.</p>
<ul>
<li>Firstly, I split data into 2 sub-dataset as <code>[100, 7, 3]</code>.</li>
<li>Then I feed them into two mini-LSTM models and concatenate the outputs.</li>
<li>Finally, I feed the combination of outputs as the input to the final LSTM model.</li>
</ul>
<p>The code is shown below. After I get the loss, I just backpropagate it to tune all the parameters in three models. May I ask two questions:</p>
<ul>
<li>I initialize the <code>h0_3, c0_3</code> as zero for the final LSTM model. Should I use the <code>h0,c0</code> from previous mini-models (<code>hn_1, cn_1</code>, <code>hn_2, cn_2</code>)?</li>
<li>If I just backpropagate the loss and tune all the parameters in three models, is this the same as tuning the parameters in three connected LSTM layers in sequential models? Like</li>
</ul>
<pre><code class="lang-auto">model = Sequential()
model.add(LSTM(n1, return_sequences=True))
model.add(LSTM(n2, return_sequences=True))
model.add(LSTM(n3, return_sequences=False))
</code></pre>
<p>Any suggestions would be appreciated. Thank you so much in advance!</p>
<pre><code class="lang-auto"># (input_size,hidden_size,num_layers)
lstm_1 = nn.LSTM(3, 10, 2, batch_first=True)
lstm_2 = nn.LSTM(3, 10, 2, batch_first=True)
lstm_3 = nn.LSTM(20, 20, 2, batch_first=True)

# (batch_size, seq length, fea_number)
x = torch.randn(100, 7, 6)

# split into 2 --&gt; (100,7,3)
x1,x2 = torch.split(x, 3, dim=-1)

# init h0,c0 (num_layers*num_directions, batch, hidden_size)
# model 1
h0_1 = torch.randn(2, 100, 10)
c0_1 = torch.randn(2, 100, 10)
output_1,(hn_1, cn_1) = lstm_1(x1, (h0_1, c0_1))

# model 2
h0_2 = torch.randn(2, 100, 10)
c0_2 = torch.randn(2, 100, 10)
output_2, (hn_2, cn_2) = lstm_2(x2, (h0_2, c0_2))

# concat two outputs
x3 = torch.cat((output_1, output_2),dim=-1)
h0_3 = torch.randn(2, 100, 20)
c0_3 = torch.randn(2, 100, 20)

output_3, (hn_3, cn_3) = lstm_3(x3, (h0_3, c0_3))
</code></pre>
MSE ignore_index

Diego999: <p>Hi,</p>
<p>I’m working on rating prediction, more especially on sub-rating. People tend to not rate every proposed aspects. In order to alleviate the training, I would like to know if there exists a way to mimic efficiently the parameter <em>ignore_index</em> or <em>CrossEntropyLoss</em> or <em>NLLLoss</em> but using instead <em>MSELoss</em>.</p>
<p>Thanks for your answer</p>

wrk226: <p>I know it is late, but I am also looking for the same answer. Here is what I got, hope it helps others who got the same question in the future.</p>
<pre><code class="lang-auto">    def mse_loss(self, input, target, ignored_index, reduction):
        mask = target == ignored_index
        out = (input[~mask]-target[~mask])**2
        if reduction == "mean":
            return out.mean()
        elif reduction == "None":
            return out
</code></pre>