akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Working on pickle file with custom data

username: eranb
cooked: <p>I create custom data with torchvision datasets and I have a pickle file that I want to load.<br>
The size of the pickle file is 79600x1x30 (batch, class, len of the vector).<br>
I work with the <strong>getitem</strong> function, I tried to divide this pickle file, but it didn’t work.<br>
There is some function that slices the pickle file or takes a batch from him?</p>

username: ptrblck
cooked: <p>I assume you are loading a tensor or e.g. numpy array from the <code>pickle</code> file?<br>
If so, what kind of error are you seeing in the <code>__getitem__</code> by indexing this object?</p>

username: eranb
cooked: <p>The size of my pickle file is 19700x4<br>
I’m trying to work with 32-batch, but it does not divide the size of the pickle (row), so it takes all the rows, e.g: (32, 19700, 4). and it needs to be (32,4).<br>
I don’t know where I’m wrong I think I need to define the sampler.</p>
<p>code:<br>
class ROboDataset(Dataset):<br>
def <strong>init</strong>(self, root, path, train=True, transform=None):<br>
self.root = root<br>
self.path = path<br>
self.train = train<br>
self.transform = transform</p>
<pre><code>def __getitem__(self, index):
    root = self.root
    path = self.path
    data = pickle.load(file=open(os.path.join(root, path), "rb"))


    features = data[0]
    target = data[1]

    if self.transform is not None:
        features = self.transform(features)

    return features, target

def __len__(self):
    data = pickle.load(file=open(os.path.join(self.root, self.path), "rb"))
    return len(data[0])</code></pre>

username: ptrblck
cooked: <p>In your code snippet you are not using the <code>index</code>, but load the complete <code>pickle</code> file and split it into the <code>features</code> and <code>target</code> tensors. I assume you could load the file in the <code>__init__</code> and index it in the <code>__getitem__</code> using the passed <code>index</code> value.</p>

username: eranb
cooked: <p>Yes, I tried not to load it on <strong>init</strong>, but I see there is no other option.<br>
Thanks.</p>
Difference in saved and loaded state dict

username: Asim_Jalwana
cooked: <p>Hi,</p>
<p>I was just wondering, there is a small difference whenever we save state dictionary of a model and reload it back. Can you please point if I am doing some mistake or if this is an insignificant issue?</p>
<p>Regards,<br>
P.S: I have pasted sample code below</p>
<p>from torchvision import models<br>
import torch</p>
<p>model = models.resnet50(pretrained=True)<br>
model.eval()</p>
<p>stateDict = model.state_dict()<br>
torch.save(model.state_dict(), “test.pth”)<br>
reloadDict = torch.load(“test.pth”)</p>
<p>diff = stateDict.items() - reloadDict.items()</p>

username: ptrblck
cooked: <p>If I’m not mistaken, subtracting <code>dict.items()</code> would try to remove duplicated entries as seen here:</p>
<pre><code class="lang-python">d1 = {'a': 1, 'b': 2, 'c': 10}
d2 = {'a': 3, 'b': 4}
print(d1.items() - d2.items())
&gt; {('a', 1), ('b', 2), ('c', 10)}

d1 = {'a': 1, 'b': 2, 'c': 10}
d2 = {'a': 3, 'b': 4, 'c': 10}
print(d1.items() - d2.items())
&gt; {('a', 1), ('b', 2)}
</code></pre>
<p>so compare the values via:</p>
<pre><code class="lang-python">model = models.resnet50(pretrained=True)
model.eval()

stateDict = model.state_dict()
torch.save(model.state_dict(), 'test.pth')
reloadDict = torch.load('test.pth')

diff = stateDict.items() - reloadDict.items()
len(diff)

for key in stateDict:
    paramA = stateDict[key]
    paramB = reloadDict[key]
    print(key, (paramA - paramB).abs().max())
</code></pre>
<p>which will show a zero difference.</p>

username: Asim_Jalwana
cooked: <p>Thanks a bunch, I made that silly assumption, will be careful next time in asking question!<br>
Thanks a heap!</p>
How to compute the distance between two word embedding?

username: Sine
cooked: <p>Hi, everyone. I am trying to train a model to output an approximate word embedding. I want to make it close the original word embedding.</p>
<p>Which loss function should I use?  Is it MSE Loss?</p>
<p>Thanks.</p>

username: pascal_notsawo
cooked: <p>If I understand you have a pre-trained word embedding model. And you have your own model that you want to train so that it produces representations very close to the one produced by the first one.<br>
Is that it?</p>

username: Versus
cooked: <p>Check <a href="https://pytorch.org/docs/stable/generated/torch.nn.CosineSimilarity.html" class="inline-onebox" rel="noopener nofollow ugc">CosineSimilarity — PyTorch 1.7.0 documentation</a></p>

username: jbschlosser
cooked: <p>Expanding on <a class="mention" href="/u/versus">@Versus</a>’s answer, the loss function form is <a href="https://pytorch.org/docs/stable/generated/torch.nn.CosineEmbeddingLoss.html" rel="noopener nofollow ugc">nn.CosineEmbeddingLoss</a>.</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.TripletMarginLoss.html" rel="noopener nofollow ugc">nn.TripletMarginLoss</a> may be helpful to you as well.</p>

username: Sine
cooked: <p>That’s right <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>

username: Sine
cooked: <aside class="quote no-group" data-username="Versus" data-post="3" data-topic="109453" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/versus/48/33590_2.png" class="avatar"> Versus:</div>
<blockquote>
<p>Check <a href="https://pytorch.org/docs/stable/generated/torch.nn.CosineSimilarity.html" rel="noopener nofollow ugc">CosineSimilarity — PyTorch 1.7.0 documentation</a></p>
</blockquote>
</aside>
<p>Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=12" title=":grinning:" class="emoji" alt=":grinning:" loading="lazy" width="20" height="20"></p>

username: Sine
cooked: <aside class="quote no-group" data-username="jbschlosser" data-post="4" data-topic="109453" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jbschlosser/48/33084_2.png" class="avatar"> jbschlosser:</div>
<blockquote>
<p>Expanding on <a class="mention" href="/u/versus">@Versus</a>’s answer, the loss function form is <a href="https://pytorch.org/docs/stable/generated/torch.nn.CosineEmbeddingLoss.html" rel="noopener nofollow ugc">nn.CosineEmbeddingLoss</a>.</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.TripletMarginLoss.html" rel="noopener nofollow ugc">nn.TripletMarginLoss</a> may be helpful to you as well.</p>
</blockquote>
</aside>
<p>Sorry, I don’t  know how to set the y label of nn.CosineEmbeddingLoss in my case. Is it -1?</p>
Selective Augmentation/Modifying DataLoader

username: zanussbaum
cooked: <p>Hi all,</p>
<p>Few questions. First, we want to compute some metrics during training and after each epoch, take the 10% and only apply augmentation to those examples from the dataset. If we have a custom dataset, is it best to subclass the DataLoader class on top of a Dataset class? What’s the best way to be able to change which examples we will be augmenting epoch to epoch?</p>

username: ptrblck
cooked: <p>You could use an internal attribute in your custom <code>Dataset</code> to tag the samples, which should be augmented, and use it in the <code>__getitem__</code> as a condition. At the beginning you could leave it empty and manipulate it after each epoch via directly accessing the internal dataset through the <code>DataLoader</code>.<br>
This approach should work, if you access the <code>loader.dataset</code> after each epoch is done and if <code>persistent_workers=False</code> is used.</p>

username: zanussbaum
cooked: <p>Thanks for this! I think the solution that we went with is similar to this <a href="https://discuss.pytorch.org/t/dynamic-dataloaders-for-on-the-fly-modifications/78870">answer you posted</a> where we create the dataLoader on the fly each time with minimal slowdowns <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to tie a tensor to a model's device without being in state_dict

username: robz
cooked: <p>I would like to add a tensor to a model so that <code>model.to(device)</code> also moves the tensor to the device. <code>register_buffer</code> seems to do this; however, I don’t want the tensor to be inside the model’s <code>state_dict</code>. Is there a method to do this?</p>
<p>My specific application is that I have a model that uses a transformer encoder. For convenience, it creates a casual mask and precomputes a positional encoding matrix in its constructor which is used in <code>forward</code>. I want the mask and positional encoding to move to the device that the model is moved to, but when I save the model to disk with <code>torch.save(model.state_dict())</code> I don’t want those tensors wasting space.</p>

username: ptrblck
cooked: <p>You could use <code>self.register_buffer</code> and set the <code>persistent</code> argument to <code>False</code>.<br>
From the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.register_buffer">docs</a>:</p>
<blockquote>
<p><strong>persistent</strong> (<a href="https://docs.python.org/3/library/functions.html#bool"><em>bool</em></a>) – whether the buffer is part of this module’s <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.state_dict"><code>state_dict</code></a>.</p>
</blockquote>

username: robz
cooked: <p>Not sure how I missed that. Thanks!</p>
Why the **forward** function is never be called

username: anhco989
cooked: <p>As the example below, I’ve never seen <strong>forward</strong> function called, but when the Network class is initialized, <strong>forward</strong> function still run?<br>
Can someone help me to explain this?</p>
<pre><code class="lang-auto">class Network(nn.Module):
    def __init__(self):
        super(Network, self).__init__()
        
    def forward(self, x):
        return x
</code></pre>

username: Nelyo
cooked: <p>The forward function gets called when you call it. “Network class is initialized?” You mean you instantiate your class “Network”. This will only call your constructor. You think that will lead to an async process? Which is not happen. I’m not sure if I understand you correctly. You might want to share more code.</p>

username: SumanthRH
cooked: <p><a class="mention" href="/u/anhco989">@anhco989</a> When the <code>Network</code> class is instantiated (Ex: <code>net = Network()</code>), all the statements within <code>__init__</code> are executed (the constructor). Later, when you run your network on some batch of data, you write <code>output = net(x)</code>, which invokes the <code>__call__</code> method. Your <code>Network</code> class simply inherits the  <code>__call__</code> method of the <code>nn.Module</code> class. It is here where the <code>forward</code> method is called.</p>

username: anhco989
cooked: <p>It really makes sense. Thank you</p>
Hardware Requirements for GPU PyTorch

username: Ed_Ed
cooked: <p>Hi there</p>
<p>I have a Dell T5400 (from 2006 vintage) with 32GB RAM , 2 X Xeon, SSD and a pair of GTX 1080 Ti cards. It works well for DL with TensorFlow, but requires me to build TF from scratch due to the lack of AVX support with my processors.</p>
<p>Does PyTorch require AVX capabilities? I cannot find this information anywhere, not anything regarding Compute Capability requirements for Nvidia cards.</p>
<p>Any pointers appreciated!</p>
<p>I am hoping either there is a compatible wheel but I will compile myself if need be.</p>

username: Ed_Ed
cooked: <p>OK found the answer.</p>
<p>Torch (quite amazingly) can auto-detect the instruction set and execute around this. Meaning my lack of AVX is not an issue. If only Tensorflow did the same. It was a joy to get cifar10 running on multiple GPU within minutes.</p>
<p>Thanks!</p>
How to concatenate tensors with dimension mismatch?

username: ankitvad
cooked: <p>I have 2 tensors:</p>
<pre><code class="lang-auto">a = [Batch x T x Dim]
b = [Batch x Dim]
</code></pre>
<p>I want to merge the output somehow by using repeat/cat so that the output is: <code>[Batch x T x (2.Dim)]</code></p>
<p>Example:</p>
<pre><code class="lang-auto">a: 
tensor([[[-0.2837,  0.1196,  0.5304, -1.1025],
         [-1.3174,  1.4882, -2.0489,  0.1606]],

        [[-1.1204,  2.4822, -0.6366, -0.9069],
         [ 0.3157, -1.7497, -1.0131,  1.5231]]])
B: 2, T: 2, Dim: 4

b:
tensor([[ 0.6773, -0.8552,  0.3827, -0.4008],
        [ 1.6068, -0.2427, -0.9487,  1.5622]])
B:2, Dim:4
</code></pre>
<p>I’d like to get something like this:</p>
<pre><code class="lang-auto">tensor([[[-0.2837,  0.1196,  0.5304, -1.1025, 0.6773, -0.8552,  0.3827, -0.4008],
         [-1.3174,  1.4882, -2.0489,  0.1606, 0.6773, -0.8552,  0.3827, -0.4008]],

        [[-1.1204,  2.4822, -0.6366, -0.9069, 1.6068, -0.2427, -0.9487,  1.5622],
         [ 0.3157, -1.7497, -1.0131,  1.5231, 1.6068, -0.2427, -0.9487,  1.5622]]])
</code></pre>
<p>So basically append the B x Dim to each batch element… And make sure that the whole thing is still differentiable finally.</p>

username: InnovArul
cooked: <p>This might work for you.</p>
<pre><code class="lang-auto">import torch
a = torch.randn(5,2,4)
b = torch.randn(5,4)
c = torch.cat([x, y.unsqueeze(1).repeat(1, x.shape[1], 1)], dim=2)
</code></pre>
PyTorch Tutorial: Loss does not decrease

username: PhysicsIsFun
cooked: <p>Greetings everyone,</p>
<p>I am now to PyTorch and to this board, and I hope I can get a little help here and there <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I just started doing the PyTorch 60 minutes Blitz tutorial, and I noticed that in one of their examples, after calling an SGD optimizer, the loss of a pretrained model did not decrease:</p>
<pre><code class="lang-auto">import numpy as np
import torch, torchvision

# %% Downloading stuff
model = torchvision.models.resnet18(pretrained=True)

# %% Rest
data = torch.rand(1, 3, 64, 64)
labels = torch.rand(1, 1000)

predic = model(data)
loss = (predic - labels).sum()
print(loss)  # loss negative
loss.backward()

optim = torch.optim.SGD(model.parameters(), lr=1e-3, momentum=0.9)
optim.step()
predic = model(data)
loss = (predic - labels).sum()
print(loss)  # loss even more negative, i.e. larger abs value
</code></pre>
<p>This was basically done here:<br>
<a href="https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html</a><br>
Should the absolute loss not decrease after a single optimization step when having only one data point? Does it have to do with the fact that the model is pretrained?</p>
<p>Best,<br>
PhysicsIsFun</p>

username: pascal_notsawo
cooked: <p>This may be due to the fact that you have used a bad loss function, such as <a href="https://discuss.pytorch.org/t/pytorch-model-not-converging/107363/4">pytorch-model-not-converging</a>.</p>

username: PhysicsIsFun
cooked: <p>Well I used what they used, which means the tutorial is not that great.<br>
Anyway, when squaring the (predic - loss), I get the mean-squared-error, and then the loss decreases just fine. So this seemed to be the problem.</p>

username: pascal_notsawo
cooked: <p>Are you making a classification? A regression? Or something else?</p>

username: PhysicsIsFun
cooked: <p>It’s what they did. Classification with real output (sigmoidal?) and 1000 labels.<br>
But seems to work now. Cheers!</p>
[Attention Mechanism] Is there a way of expanding a tensor on its diagonal axis?

username: meifish
cooked: <p>Hi, I am looking for a solution to achieve the effect of expanding a 3x4 tensor on its diagonal axis and fill the new diagonal with 1. The original tensor is like a weighting map with each column sum up to 1:</p>
<pre><code class="lang-auto">[[0.5,  0.8,  0.7,  0.3],
 [0.2,  0.1,  0.2,  0.4],
 [0.3,  0.1,  0.1,  0.3]]
</code></pre>
<p>and the target matrix (<code>y</code>) I want to expand to looks like:</p>
<pre><code class="lang-auto">[[1,    0.8,  0.7,  0.3],
 [0.5,  1,    0.2,  0.4],
 [0.2,  0.1,  1,    0.3],
 [0.3,  0.1,  0.1,  1   ]]
</code></pre>
<p>and the underlying reason that I want to achieve this is because I have an input tensor <code>x</code> with <code>4x4</code> dimension, and I would like to produce a <code>4x4</code> matrix by assigning the weightings obtained from <code>y</code> to the apply to the input tensor <code>x</code> (like the attention mechanism), so the resulting <code>4x4</code> is like this in pseudo:</p>
<pre><code class="lang-auto">[
 [ Y11 * X_row1 + Y21 * X_row2 + Y31 * X_row3 + Y41 * X_row4 ],
 [ Y12 * X_row1 + Y22 * X_row2 + Y32 * X_row3 + Y44 * X_row4 ]
 [ Y13 * X_row1 + Y23 * X_row2 + Y33 * X_row3 + Y43 * X_row4 ],
 [ Y14 * X_row1 + Y24 * X_row2 + Y34 * X_row3 + Y44 * X_row4 ]
]
</code></pre>
<p>which can be illustrated as:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/6/b6efe30c3f6455ac7c823f3d742ce8c240325081.png" data-download-href="https://discuss.pytorch.org/uploads/default/b6efe30c3f6455ac7c823f3d742ce8c240325081" title="Screenshot from 2021-01-20 15-16-31"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/6/b6efe30c3f6455ac7c823f3d742ce8c240325081_2_690x332.png" alt="Screenshot from 2021-01-20 15-16-31" data-base62-sha1="q6kR0pJSgLR09zYMkFDHg0CEL8R" width="690" height="332" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/6/b6efe30c3f6455ac7c823f3d742ce8c240325081_2_690x332.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/6/b6efe30c3f6455ac7c823f3d742ce8c240325081_2_1035x498.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/6/b6efe30c3f6455ac7c823f3d742ce8c240325081.png 2x" data-dominant-color="E7F4E8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-01-20 15-16-31</span><span class="informations">1126×542 38.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I appreciate any thoughts which related to (1) how to expand the tensor on the diagonal axis, or thoughts regarding (2) a better way of achieving the resulting attention mechanism with an easier way which won’t requires the expansion of Y.</p>
<p>Thanks in advanced!</p>
Concatenate two databases with pytorch

username: randinoo
cooked: <p>when I concatened two databases , this error is displayed</p>
<pre><code class="lang-auto">TypeError: expected Tensor as element 0 in argument 0, but got numpy.ndarray
</code></pre>
<p>help me</p>

username: ptrblck
cooked: <p>The error message explains that <code>torch.tensor</code>s are expected while you are trying to pass numpy arrays.<br>
You can transform arrays to tensors via <code>torch.from_numpy</code>.</p>

username: randinoo
cooked: <p>thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>  <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:"></p>
Best loss function for special classification

username: JDE65
cooked: <p>Hello,<br>
I build a Conv1D model to classify items into 6 categories (from 0 to 5).<br>
I use therefore nn.CrossEntropyLoss that penalizes the errors with high probabilities but also the low probabilities associated to right answers. And that is what I was looking for.<br>
But CrossEntropy considers the classes as independent, while I’d like reduce the loss for an error where model predicts 0 (cat) while it is a dog (1) and increase the loss when the model predicts 0 (cat) when the true answer is class 5 (fish).<br>
Is there a way to embed the class proximity in the loss computation of a classification problem ?<br>
Thanks in advance for any help !</p>
Can't quantize Linear + Relu

username: ignatius
cooked: <p>When running this code I can’t fuse Linear + ReLU, though the documentation says that it is possible (<a href="https://pytorch.org/docs/stable/_modules/torch/quantization/fuse_modules.html#fuse_modules" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/_modules/torch/quantization/fuse_modules.html#fuse_modules</a>)</p>
<pre><code class="lang-auto">class MyModule(nn.Module):
    def __init__(self):
        super().__init__()
        self.quant = torch.quantization.QuantStub()
        self.linear = nn.Linear(32, 64)
        self.relu = nn.ReLU()
        self.dequant = torch.quantization.DeQuantStub()

    def forward(self, input):
        y = self.quant(input)
        y = self.linear(input)
        y = self.relu(y)
        y = self.dequant(y)
        return y

    def fuse_model(self):
      torch.quantization.fuse_modules(self, modules_to_fuse=[[self.linear, self.relu]], inplace=True)

print("Create model...")
model_ = MyModule()
print("Create model... [OK]")
in_ = torch.ones(32, 32)
print("Forward pass...")
y = model_(in_)
print("Forward pass... [OK]")

print("Fusing model... ")
model_.eval()
model_.fuse_model()
print("Fusing model... [OK]")
</code></pre>
<p>I got the following error:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
ModuleAttributeError                      Traceback (most recent call last)
&lt;ipython-input-38-7ffe2cc10aa7&gt; in &lt;module&gt;()
     24 
     25 model_.eval()
---&gt; 26 model_.fuse_model()
     27 

4 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __getattr__(self, name)
    777                 return modules[name]
    778         raise ModuleAttributeError("'{}' object has no attribute '{}'".format(
--&gt; 779             type(self).__name__, name))
    780 
    781     def __setattr__(self, name: str, value: Union[Tensor, 'Module']) -&gt; None:

ModuleAttributeError: 'Linear' object has no attribute 'split'
</code></pre>
<ul>
<li>Torch version: 1.7.0+cu101</li>
<li>Torch vision version: 0.8.1+cu101</li>
</ul>

username: jerryzh168
cooked: <p>I think you need the following:</p>
<pre><code class="lang-python">def fuse_model(self):
      torch.quantization.fuse_modules(self, modules_to_fuse=[["linear", "relu"]], inplace=True)
</code></pre>

username: ignatius
cooked: <p>For sure… thanks a lot!</p>
Torchaudio.transforms.griffinlim output to tensorboard audio

username: aRI0U
cooked: <p>Hi,<br>
I am trying to implement a VAE on audio and want to listen to the reconstructed audio via tensorboard. The input and output of my network is a spectrogram (computed with <code>torchaudio.transforms.Spectrogram</code>) so I assume I should use <code>torchaudio.transforms.GriffinLim</code> to get a listenable waveform.<br>
However, when I send the output of <code>torchaudio.transforms.GriffinLim</code> to TensorBoard, I get <code>warning: audio amplitude out of range, auto clipped.</code> and there is no “audio” tab in TensorBoard.</p>
<p>It seems that GriffinLim outputs a waveform with values between -2 and 2, but I do not know what kind of audio format is supported by TensorBoard so I do not know what kind of transform I should perform after applying GriffinLim.<br>
Does anyone have an idea?</p>
Model Parallelism for HuggingFace Transformers

username: Saichandra_Pandraju
cooked: <p>Hi,</p>
<p>I’m implementing <a href="https://huggingface.co/transformers/model_doc/deberta.html" rel="noopener nofollow ugc">Microsoft’s DeBERTa from HuggingFace</a> in PyTorch.<br>
Can I implement PyTorch’s <a href="https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html" rel="noopener nofollow ugc">Model Parallelism</a> with this HuggingFace transformer?<br>
If yes, is there any documentation regarding this ?</p>
Freeze some layers EfficientNet

username: VirginieBfd
cooked: <p>Dear all,</p>
<p>I would like to freeze not just the last fully connected layer of EfficientNet-b0 but also some of the previous block to apply transfer learning to a fairly different domain.</p>
<p>This is something I have been able to achieve with ResNet18:</p>
<pre><code class="lang-auto">  cntr=0
        for name, child in model.named_children():
            cntr+=1
            if cntr &lt; lt:
                print(f'{name} not trainable')
                for param in child.parameters():
                    param.requires_grad = False
            else:
                print(f'{name} trainable')
</code></pre>
<p>But found complex with EfficientNet!</p>
Balance dataset with continous values for student/teacher

username: wetrhhj
cooked: <p>Hello,</p>
<p>I want to train a student network and therefore collected all the probability outputs of my teacher (~ 41.000) to create my dataset for the student. The distribution (Binned in 100 bins) looks like this (I already ploted the weights for the WeightedRandomSampler as curve):</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/4/6472f3a7961df46b24cfb21668d8ec6d78168703.png" alt="scores_sampler_before" data-base62-sha1="ekC1RF2z8VyVdWbyq1rGkuiO8gP" width="480" height="360"></p>
<p>The weights are obtained by:</p>
<pre><code class="lang-auto">batch_size = 100
weights = 1. / dataset.targets.bincount().double()
sampler = torch.utils.data.sampler.WeightedRandomSampler(weights=weights, num_samples=len(dataset), replacement=True)

kwargs = {'num_workers': 1, 'pin_memory': True} if device == 'cuda' else {}
dataloader = torch.utils.data.DataLoader(dataset, batch_size=batch_size, sampler=sampler, **kwargs)
</code></pre>
<p>After running through the whole dataloader I receive the following distribution: See next post</p>
<p>The amount of samples seems okay, but why are there so huge gaps in between? For the amount of samples I expected these to be more filled.</p>
<p>Thank you in advance!</p>

username: wetrhhj
cooked: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/b/2bc6369ba50b7e97743a1d97aa1914af6e221c7a.png" alt="scores_sampler_after" data-base62-sha1="6ffctLGcUbHSm8ATojvIbndRlh0" width="480" height="360"></p>
What's tag of a IVAUE of input type is c10::optional<Tensor>

username: leslie-fang-intel
cooked: <p>Refer to this test file: backend_fallback_test.cpp, When I am using the torch::CppFunction::makeFromBoxedFunction API, one of the input argument’s type is c10::optional. When it’s warpper by IVAUE, what’s the tag?</p>

username: leslie-fang-intel
cooked: <p>Hi <a class="mention" href="/u/ezyang">@ezyang</a>, could you kindly help to look into this question?</p>

username: glaringlee
cooked: <p><a class="mention" href="/u/leslie-fang-intel">@leslie-fang-intel</a><br>
I guess you meant wrapped by IVALUE?<br>
Can you give me an example, I will see whether I can help here.</p>

username: leslie-fang-intel
cooked: <p>Thanks Glaring. Sorry for the later reply.<br>
I want to handle this kind of case:</p>
<pre><code class="lang-auto">void generic_wrapper_fallback(const c10::OperatorHandle&amp; op, torch::jit::Stack* stack) {
  ...
}
</code></pre>
<p>One of input argument inside in jit::stack could be the type of c10::optional.<br>
How can I pick the argument with this type out?</p>

username: glaringlee
cooked: <p><a class="mention" href="/u/leslie-fang-intel">@leslie-fang-intel</a><br>
Sry, missed this message.<br>
Can you test<br>
torch::jit::pop(*stack).to&lt;c10::optional&lt;your_type&gt;&gt;()?<br>
If it doesn’t work, please change the category to jit, there will be jit people there help you.</p>
Char-CNN with sequences of different lengths in the batch

username: alsuhr
cooked: <p>Hi,</p>
<p>I am implementing a char-CNN over words in sentences in a hierarchical fashion (each word goes through the char CNN; the output of this is concatenated with a word embedding; this is then put through an RNN over the entire sentence). My CNN (Conv1d) has a kernel size / stride / padding such that the output of it has the same $N$ and $L$, so I am treating the outputs as one per character (the length $L$ is the number of characters).</p>
<p>I want to get the char-level word representation by doing a max-pool over the characters for a particular word, but in batch, the words don’t have the same length, and I don’t want to do a max pool over the padding characters. Kind of how with the RNN module you can have it ignore the padding in the batch by giving it the sequence lengths. Any idea how to do this?</p>
<p>I think I could iterate over all of the token sequences and do a max pool for each individually, with the kernel size being the length of the token. But this seems inefficient. Is there another easy way to do this, or example code somewhere (all the examples I found seemed to ignore this problem and just do a max-pool over valid and invalid characters in the batch)?</p>
<p>Thanks</p>
<p>edit: I found <a href="https://pytorch.org/docs/stable/generated/torch.nn.AdaptiveMaxPool1d.html" rel="noopener nofollow ugc">Adaptive Max Pool</a>, which might be related, but it doesn’t specify what the input/output sizes should be or whether it’s relevant for my use case.</p>
Need help in creating dataset class for multilabel object detection in a single image

username: IamSparky
cooked: <p>Can some please help me in modifying this single class object detection dataset class to multiple class object detection dataset</p>
<pre><code class="lang-auto">import os
import numpy as np
import torch
from PIL import Image

class object_detection_dataset_class(Dataset):
    def __init__(self, csv_file_path):
        self.train_df = pd.read_csv(csv_file_path)
        self.image_ids = self.train_df['image_id'].unique()
        
    def __len__(self):
        return len(self.image_ids.shape[0])
    
    def __getitem__(self, index):
        image_id = self.image_ids[index]
        bboxes = self.train_df[self.train_df['image_id'] == image_id]
        
        image = np.array(Image.open('__image_file_path__'+ self.image_id[index] +'.jpg')) 
        image = Image.fromarray(image).convert('RGB')
        image /= 255.0
       
        boxes = bboxes[['xmin', 'ymin', 'xmax', 'ymax']]
        area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])

        boxes = torch.as_tensor(boxes, dtype = torch.float32)
        area = torch.as_tensor(area, dtype = torch.float32)

        labels = torch.ones((bboxes.shape[0], ), dtype = torch.int64)
        is_crowd = torch.ones((bboxes.shape[0], ), dtype = torch.int64)

        target = {}
        target["boxes"] = boxes
        target["labels"] = labels
        target["masks"] = masks
        target["image_id"] = image_id
        target["area"] = area
        target["iscrowd"] = iscrowd

        image = torchvision.transforms.ToTensor()(image)
        return image, target
</code></pre>
<pre><code class="lang-auto"># initializing the dataset
train_dataset = object_detection_dataset_class("______csv file path for training data___________")

def collate_fn(batch):
     return tuple(zip(*batch)

training_data_loader = Dataloader(train_dataset, batch_size = 2, shuffle= True, num_workers= 2, collate_fn = collate_fn)
</code></pre>
<pre><code class="lang-auto"># for dry run testing of dataloader to see the images loaded inside of pytorch dataloader
from matplotlib import pyplot as plt

images, targets  = next(iter(training_data_loader)) 
images = list(image.to(device) for image in images) 
targets = [{k: v.to(device) for k,v in t.items()} for t in targets]  

boxes = targets[0]['boxes'].cpu().numpy().astype(np.int32)
img = images[0].permute(1, 2, 0).cpu().numpy()
fig, ax = plt.subplots(1,1, figsize(12, 6))

for box in boxes:
    cv2.rectangle(image,
                          (box[0], box[1]),
                          (box[2], box[3])
                          (220, 0, 0), 
                          1)

ax.set_axis_off()
ax.imshow(img)
</code></pre>

username: IamSparky
cooked: <p>Anyone please help me on this?</p>
Problem happens when I load the pytorch_pretrained_bert

username: yknUpUp
cooked: <p>ssh://zpf@10.12.41.181:22/home/zpf/.virtualenvs/pytorch/bin/python -u /home/zpf/Desktop/zpf/ED_NEEL/train.py<br>
Traceback (most recent call last):<br>
File “/home/zpf/Desktop/zpf/ED_NEEL/train.py”, line 146, in <br>
train(logger, False)<br>
File “/home/zpf/Desktop/zpf/ED_NEEL/train.py”, line 66, in train<br>
model = gcn_bert0(t=0.4, adj_file=‘data/god_adj.pkl’).cuda()<br>
File “/data/zpf/ED_NEEL/model.py”, line 178, in gcn_bert0<br>
model = BertModel.from_pretrained(’./bert_pretrain’)<br>
File “/home/zpf/.virtualenvs/pytorch/lib/python3.7/site-packages/pytorch_pretrained_bert/modeling.py”, line 603, in from_pretrained<br>
state_dict = torch.load(weights_path, map_location=‘cpu’)<br>
File “/home/zpf/.virtualenvs/pytorch/lib/python3.7/site-packages/torch/serialization.py”, line 585, in load<br>
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)<br>
File “/home/zpf/.virtualenvs/pytorch/lib/python3.7/site-packages/torch/serialization.py”, line 772, in _legacy_load<br>
deserialized_objects[key]._set_from_file(f, offset, f_should_read_directly)<br>
RuntimeError: storage has wrong size: expected 3981035933985103798 got 768</p>
<p>I download the pretrained model from the web and put the model in ./bert_pretrained.How can I solve it？Thanks a lot！</p>

username: yknUpUp
cooked: <p>this is because i use the ftp to directly push the pretrained model to the GPU server.<br>
Now I compressed it firstly,then push it.The problem is fixed. <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>
Turning a one-hot to padded indices tensor

username: Naveh_Porat
cooked: <p>I have a binary input to my model that I sample from some Bernoulli distribution:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; batch_size = 3
&gt;&gt;&gt; weights = torch.tensor([0.1111, 0.2669, 0.5122, 0.2045, 0.0224]).repeat((batch_size, 1))
&gt;&gt;&gt; batch = torch.bernoulli(weights)
&gt;&gt;&gt; batch
tensor([[1., 0., 0., 1., 1.],
        [0., 1., 0., 0., 0.],
        [0., 1., 1., 0., 0.]])
</code></pre>
<p>I want to get a padded sequence of embeddings for every sample in my batch so I can run it through a transformer encoder. For that, I first need to turn <code>batch</code> into a tensor of indices, then pad it, and finally run it through an embedding layer.<br>
The closest I’ve come is to use <code>tensor.nonzero()</code> but I don’t know how to pad it and reshape it back to <code>batch_size, max_seq_len</code>.</p>
<p>Desired output:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; func(batch, padding_value=-1)
tensor([[ 0.,  3.,  4.],
        [ 1., -1., -1.],
        [ 1.,  2., -1.]])
</code></pre>

username: Naveh_Porat
cooked: <p>In the meantime, I’ve solved this by iterating over the batch. Still curious if there exists a more elegant way.</p>
<pre><code class="lang-auto">&gt;&gt;&gt;def to_indices(batch, padding_value=-1):
   ...:     indices = []
   ...:     for sample in batch:
   ...:         indices.append(sample.nonzero(as_tuple=True)[0])
   ...:     return nn.utils.rnn.pad_sequence(indices, batch_first=True, padding_value=padding_value)
&gt;&gt;&gt; batch
tensor([[1., 0., 0., 1., 1.],
        [0., 1., 0., 0., 0.],
        [0., 1., 1., 0., 0.]])
&gt;&gt;&gt; to_indices(batch)
tensor([[ 0,  3,  4],
        [ 1, -1, -1],
        [ 1,  2, -1]])
</code></pre>
Difference between Adam and AdamW in pytorch

username: AOZMH
cooked: <p>There are a few <a href="https://discuss.pytorch.org/t/difference-between-adam-and-adamw-implementation/99392">discussions</a> on the difference between Adam(weight_decay=0.01) and AdamW() which point out that the implementation of weight decay in AdamW is the decoupled weight decay, different from the raw regularization of Adam.</p>
<p>However, I consulted the official <a>documentation</a> of Adam &amp; AdamW and noticed that the implementation of weight-decay in Adam also followed the <a href="https://arxiv.org/abs/1711.05101" rel="noopener nofollow ugc">Decoupled Weight Decay Regularization</a>, which is the same for AdamW. Does that mean that currently, Adam &amp; AdamW are the same w.r.t. weight decay?</p>

username: ptrblck
cooked: <p>This issue should be solved in <a href="https://github.com/pytorch/pytorch/pull/50464">this PR</a>.</p>

username: AOZMH
cooked: <p>Clear now, many thanks!</p>
How to use k channels in CNN for k FC Layers

username: spadel
cooked: <p>I have an encoder, which outputs a tensor with shape <code>(bn, c * k, 32, 32)</code>. I now want produce k means with shape <code>(bn, k, 1, 2)</code>. So the means are 2-dim coordinates. To do so, I want to use k FC Layers, while for each mean k_i I only want to use c channels.</p>
<p>So my idea is, that I reshape the encoder output <code>out</code> to a 5d tensor with shape <code>(bn, k, c, 32, 32)</code>. Then I can use the flattened <code>out[:, 0]</code> … <code>out[:, k]</code> as input for the k linear layers.</p>
<p>The trivial solution would be to define the linear layers manually:</p>
<pre><code class="lang-auto">self.fc0 = nn.Linear(c * 32 * 32, 2)
...
self.fck = nn.Linear(c * 32 * 32, 2)
</code></pre>
<p>Then I could define the forward pass for each mean as follows:</p>
<pre><code>mean_0 = self.fc0(out[:, 0].reshape(bn, -1))
...
mean_k = self.fck(out[:, k].reshape(bn, -1))
</code></pre>
<p>Is there a more efficient way to do that?</p>
<p>Edit: To give a little more information about the background: I want to find the keypoints/landmarks of the input image. The idea is, that I assign each keypoint k a number of feature maps c. That’s, why the output of the encoder has the shape <code>(bn, k, c, 32, 32)</code> . Now I want those c feature maps to predict its keypoint, which is a 2-dim coordinate. Essentially, I want a separate fc layer for each keypoint.</p>

username: spadel
cooked: <p>I have come to this solution using the <code>group</code> function of <code>nn.Conv2d</code>:</p>
<pre><code class="lang-auto">bn = 1; k = 5; c = 3
x = torch.rand(bn, k*c, 32, 32)
m = nn.Conv2d(in_channels=c*k, out_channels=2*k, kernel_size=32, groups=k)
</code></pre>
DropBlock implementation example

username: jarvico
cooked: <p>Hi,</p>
<p>I hava a basic classifier fo predict MNIST digits as below:</p>
<pre><code class="lang-auto">class LeNet_dropout(nn.Module):
    def __init__(self):
        super(LeNet_dropout, self).__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=3)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=3)
        self.fc1 = nn.Linear(2880, 128)
        self.fc2 = nn.Linear(128, 10)
        self.drop_layer = nn.Dropout(p=0.5)

    def last_hidden_layer_output(self, x):

        x = F.relu(self.conv1(x))
        x = F.max_pool2d(self.drop_layer(F.relu(self.conv2(x))), 2)
        x = x.view(-1, 2880)
        x = self.drop_layer(F.relu(self.fc1(x)))
        return x

    def forward(self, x):

        x = self.last_hidden_layer_output(x)
        x = self.fc2(x)
        return x
</code></pre>
<p>When I want to enable dropout, I simple pass my model to enable_dropout method as below:</p>
<pre><code class="lang-auto">def enable_dropout(model):
    for m in model.modules():
        if m.__class__.__name__.startswith('Dropout'):
            m.train()
</code></pre>
<p>My question is: I want to use “DropBlock” in the second Conv layer instead of normal dropout. I am not sure if there is a built-in pytorch functionality for DropBlock operation.</p>
<p>Can somebody help me on this?</p>
<p>Best Regards…</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="jarvico" data-post="1" data-topic="107986">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jarvico/48/28509_2.png" class="avatar"> jarvico:</div>
<blockquote>
<p>I am not sure if there is a built-in pytorch functionality for DropBlock operation.</p>
</blockquote>
</aside>
<p>There seem to be a few <a href="https://www.google.com/search?q=dropblock+pytorch&amp;rlz=1C1GCEA_enGB845GB845&amp;oq=dropblock+pytorch&amp;aqs=chrome..69i57j69i60.2844j0j7&amp;sourceid=chrome&amp;ie=UTF-8">PyTorch implementations</a> on GitHub.</p>

username: jarvico
cooked: <p>Thanks for your help!</p>
Autograd: Automatic differentiation of matrix / batched matrix in pytorch

username: jmaronas
cooked: <p>Hi,</p>
<p>I have been recently interested in understanding the more practical part of automatic differentiation. However, I have realized that a naive implementation of several operations using basic operations can be inefficient instead of directly coding up the derivative.</p>
<p>So for example the matrix product:</p>
<p>a = Wx</p>
<p>is in essence a combination of multiplication and summations. Hence we could in principle apply reverse-mode autodiff  to compute the required derivatives. This would for sure make the graph very big and ineficient; when the gradient of a w.r.t x is simply W, hence computing the chain rule can be done direclty hardcoding the derivative of this matrix product.</p>
<p>My question is: does autograd consider this for common operations as the one I have exemplified? I guess so but would like to be certain about how a state of the art toolkit handle this things.</p>
<p>The straightforward question here is: How does it handle batched matrix operations? Is it a hardcoded gradient or it basically expand/copy the matrix involved? (I guess is the later).</p>
<p>Any resource to get this information?</p>
<p>Thank you</p>
Post-Processing between two PyTorch layers

username: raphael410
cooked: <h3>Question</h3>
<p>Hello,<br>
I am using a pytorch model (<a href="https://github.com/MhLiao/MaskTextSpotterV3" class="inline-onebox" rel="noopener nofollow ugc">GitHub - MhLiao/MaskTextSpotterV3: The code of "Mask TextSpotter v3: Segmentation Proposal Network for Robust Scene Text Spotting"</a>) and I wanted to export it to onnx. I already asked on onnx github forum, and they suggested to ask the question here.</p>
<p>The issue is, we have some post-processing between some layers. For example, we perform post-processing on the outputs of the RPN boxes before feeding the proposals to the heads. I am having then a lot of warnings, to give you an example, we want to convert tensors to numpy array to have image computation using opencv, triggering then : <code>TracerWarning: Converting a tensor to a NumPy array might cause the trace to be incorrect.</code>. That case can be found in the class <code>SEGPostProcessor(torch.nn.Module)</code> in the repository I cited above.</p>
<p>Or another example, sometimes we have to iterate over a Tensor, triggering <code>RuntimeWarning: Iterating over a tensor might cause the trace to be incorrect.</code></p>
<p>Then the conversion to onnx is giving many warnings, and eventually the onnx runtime fails without surprise.<br>
Here’s the runtime failure log :</p>
<pre><code class="lang-auto">2021-01-19 13:41:54.381600130 [E:onnxruntime:, sequential_executor.cc:333 Execute] Non-zero status code returned while running Gather node. Name:'' Status Message: indices element out of data bounds, idx=94375546868464 must be within the inclusive range [-1,0]
Traceback (most recent call last):
  File "tools_onnx/test_net_onnx.py", line 263, in &lt;module&gt;
    main()
  File "tools_onnx/test_net_onnx.py", line 233, in main
    ort_outs = ort_session.run(None, ort_inputs)
  File "/home/ubuntu/anaconda3/envs/masktextspotter37/lib/python3.7/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 124, in run
    return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Non-zero status code returned while running Gather node. Name:'' Status Message: indices element out of data bounds, idx=94375546868464 must be within the inclusive range [-1,0]

</code></pre>
<p>One of my idea was to split the model into many sub-models, and make the post-processing between these models, but it’s very fastidious and I am convinced there is a better solution.</p>
<p>If you have any information about the good way to deal with these kind of steps, it would be really helpful. Thanks in advance.</p>
Error while appending tensors to a list

username: Bhavishya_Pandit
cooked: <p>I want to append 2 tensors of different size in a list and later I want to put all of it into dataloader.<br>
x_train is a numpy array of 277 numpy arrays<br>
and y_train is a numpy array of 277 scalars</p>
<pre><code class="lang-auto">image_transforms = {
    "train": transforms.Compose([transforms.ToPILImage(),
        transforms.Resize((224, 224)),
        transforms.ToTensor(),
        transforms.Normalize([0.5, 0.5, 0.5],
                             [0.5, 0.5, 0.5])
    ]),
    "test": transforms.Compose([
        transforms.Resize((224, 224)),
        transforms.ToTensor(),
        transforms.Normalize([0.5, 0.5, 0.5],
                             [0.5, 0.5, 0.5])
    ])
}
</code></pre>
<pre><code class="lang-auto">print(x_train[1].shape)
trainset=[]
testset=[]
train=image_transforms['train']
for i in range(x_train.shape[0]):
    trainset.append([torch.tensor(train(x_train[i]),torch.tensor(y_train))])
trainset
</code></pre>
<p><strong>Error</strong></p>
<pre><code class="lang-auto">TypeError                                Traceback (most recent call last)
&lt;ipython-input-26-66fc15feb658&gt; in &lt;module&gt;
      4 train=image_transforms['train']
      5 for i in range(x_train.shape[0]):
----&gt; 6     trainset.append([torch.tensor(train(x_train[i]),torch.tensor(y_train))])
      7 trainset

TypeError: tensor() takes 1 positional argument but 2 were given
</code></pre>

username: InternalInstance
cooked: <p>Probably your dataset returns 2 values. But to create a tensor you need to pass only 1 value.</p>

username: Bhavishya_Pandit
cooked: <p>x_train is an array of arrays which are nothing but pixel values of images.<br>
Total arrays in x_train are 277 and y_train is an array of 277 scalars<br>
It returns one value each.</p>

username: ptrblck
cooked: <p>The error is raise, as you are trying to pass multiple numpy arrays/scalars to <code>torch.tensor</code> as seen here:</p>
<pre><code class="lang-python">torch.tensor(np.random.randn(1), np.random.randn(2))
&gt; TypeError: tensor() takes 1 positional argument but 2 were given
</code></pre>
<p>Transform the numpy arrays to tensors first via <code>torch.from_numpy</code> and use <code>torch.cat</code> or <code>torch.stack</code> to create the single tensor.</p>
Batch-wise complex operation

username: Joseph_Amigo
cooked: <p>I’ve got a 4 dimensional torch tensor parameter defined like this :</p>
<pre><code class="lang-auto">nn.parameter.Parameter(data=torch.Tensor((13,13,13,13)), requires_grad=True)
</code></pre>
<p>and four tensors with dims (batch_size,13) (or one tensor with dims (batch_size,4,13)). I’d like to get a tensor with dims (batch_size) equal to the formula at the end of this picture :<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/9/e9680960d358029645d87cebbea7a37d67411221.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/e9680960d358029645d87cebbea7a37d67411221" title="true_pict"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e9680960d358029645d87cebbea7a37d67411221_2_666x500.jpeg" alt="true_pict" data-base62-sha1="xiO9bcfTZosm1EUe69yfFwu6QIV" width="666" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e9680960d358029645d87cebbea7a37d67411221_2_666x500.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/e/9/e9680960d358029645d87cebbea7a37d67411221.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/9/e9680960d358029645d87cebbea7a37d67411221.jpeg 2x" data-dominant-color="B4ACA0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">true_pict</span><span class="informations">750×563 88.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>If A is a tensor of dims 3, then I manage to do it with :</p>
<pre><code class="lang-auto">torch.bmm(torch.unsqueeze(z,2),torch.bmm(torch.unsqueeze(y,1),torch.transpose(torch.matmul(x,A),0,1))).sum(axis=2).sum(axis=1)
</code></pre>
<p>But if A is a tensor of dims 4, I have no idea of how to perform it with torch functions.</p>