akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Extract feature map pixels

username: solsol
cooked: <p>Hi,<br>
I wanna transform feature-map [batch, ch, n, m] divided by grids into pixel-based.<br>
Any help, please?</p>

username: ptrblck
cooked: <p>Could you explain your use case a bit and also what you mean by “pixel-based”?</p>

username: solsol
cooked: <p>def attention(keyA, keyB, value, mask=None, dropout=None):</p>
<pre><code>d_k = keyA.size(-1)
scores = torch.matmul(keyA, keyB.transpose(-2, -1)) \
         / math.sqrt(d_k)
if mask is not None:
    scores = scores.masked_fill(mask == 0, -1e9)
p_attn = F.softmax(scores, dim = -1)

return torch.matmul(p_attn, value), p_attn
</code></pre>
<p>This is a function computes attention for two feature-maps input. Actually I want to transfer the grid feature maps into denser one and not divided regions to n*m. To consider attention pixel-based not region-based. Thanks.</p>
About transfer learning (Please! help me !)

username: Sena_Andromeda
cooked: <p>Recently I have been working on a footprint image classification project. And I have just read the transfer learning tutorial from the official documentaion from the pytorch website(<a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="noopener nofollow ugc">here</a>).<br>
All the footprint images are like this :<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/a/0afaf5040482d518a4acb6605f260f5d953c6565.png" alt="image" data-base62-sha1="1z8rLkdJULmrOwGNcHa7RJyT0MZ" width="315" height="190"><br>
The dataset has 20 persons’ footprint images and each person has 10 footprint images.<br>
I tried the scripts from the tutorial mentioned before. And I have only changed the part for data transforming into this:</p>
<blockquote>
<p>data_transforms = {<br>
‘train’: transforms.Compose([<br>
<span class="hashtag-raw">#transforms</span>.RandomResizedCrop(224),<br>
transforms.Resize(256),<br>
transforms.CenterCrop(224),<br>
transforms.RandomHorizontalFlip(),<br>
<span class="hashtag-raw">#transforms</span>.RandomVerticalFlip(),<br>
transforms.RandomRotation(degrees=15),<br>
transforms.RandomRotation(degrees=30),<br>
transforms.ToTensor(),<br>
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])<br>
]),<br>
‘val’: transforms.Compose([<br>
transforms.Resize(256),<br>
transforms.CenterCrop(224),<br>
transforms.ToTensor(),<br>
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])<br>
]),<br>
}</p>
</blockquote>
<p>So, here is the result that has been puzzling me for pretty much time now.<br>
When I use this:<br>
<strong>model_ft = models.resnet50(pretrained=True)</strong><br>
<strong>num_ftrs = model_ft.fc.in_features</strong><br>
<strong>model_ft.fc = nn.Linear(num_ftrs, 20)</strong><br>
I could only get the acc of about 60%.But when I didnt change the fc layer like this:<br>
<strong>model_ft = se_resnet50(pretrained = True)</strong><br>
I could even get the result of acc reaching even 97.5%.<br>
Here are my questions:<br>
1.When I changed the fc layer it means it could only output 20 probabilities(Since I have a dataset of 20 persons).But , why is that I got worse results if I changed the fc layer???<br>
2.I have been considering using the SEnet(se-resnet50 <a href="https://github.com/Cadene/pretrained-models.pytorch" rel="noopener nofollow ugc">here</a>) to improve my result(from 97.5%).But I just got the acc of about 80% which is worse than resnet50. How could it be worse?<br>
3.If I want to imporve my results(from 97.5%), what I am supposed to do ??</p>
<p>Thanks in advance. I would appreciate all kinds of help.</p>

username: ptrblck
cooked: <p>The initialization of your new layer might not be suitable for the use case (or any other hyperparamter to train the new layer).</p>
Cuda illegal memory error

username: Hady_Pranoto
cooked: <p>I have cuda illegal memory error. It happen randomly aftter 7000 until 15000 iteration… im confuce about this error.  I was try to make all operation</p>
<p>Like</p>
<p>Torch.ones</p>
<p>Torch zeros</p>
<p>Into variable</p>
<p>It reduce the number of error</p>
<p>How i solve this problem?</p>

username: ptrblck
cooked: <p>Could you post the stack trace please?</p>

username: Hady_Pranoto
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/2/52ea1b3f22ab267f83ba8d7d204f3700fd7c5da7.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/52ea1b3f22ab267f83ba8d7d204f3700fd7c5da7" title="20200301_084806"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/2/52ea1b3f22ab267f83ba8d7d204f3700fd7c5da7_2_666x500.jpeg" alt="20200301_084806" data-base62-sha1="bPuIQljnHbtK37sJ8ez7IwSHp31" width="666" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/2/52ea1b3f22ab267f83ba8d7d204f3700fd7c5da7_2_666x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/5/2/52ea1b3f22ab267f83ba8d7d204f3700fd7c5da7_2_999x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/2/52ea1b3f22ab267f83ba8d7d204f3700fd7c5da7_2_1332x1000.jpeg 2x" data-dominant-color="7C6562"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">20200301_084806</span><span class="informations">2304×1728 1.45 MB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>this is my error,</p>

username: ptrblck
cooked: <p>Could you post a minimal code snippet to reproduce this issue?</p>
Patches correspond to specific locations

username: Frida
cooked: <p>Does anyone know how to divide images into patches following specific patch middle point? for example, take only patches at the points [1,2], [3,8] etc’?</p>

username: ptrblck
cooked: <p>You could directly index the tensor with the corresponding indices creating the window.<br>
Would that work or what is your use case exactly? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Frida
cooked: <p>Thanks a lot for the answer!<br>
Basically, I have a tensor of B,2,H,W and set of B,N*N pairs of (X,Y), which are the middle of the patch, do you see a way to do it efficiently? With indices we should do a for loop, right?</p>
How to do ensemble when model returns tuple instead of tensor?

username: mobassir94
cooked: <p>as i am onto some  multiclass classification problem,i need my model to return x1,x2,x3 for<br>
grapheme_root,vowel_diacritic and consonant_diacritic class<br>
so my ResNet implementation returns a tuple instead of tensor:<br>
`x1 = self.fc1(x)</p>
<p>x2 = self.fc2(x)</p>
<p>x3 = self.fc3(x)</p>
<p>return x1,x2,x3`</p>
<p>i tried ensemble using this code :</p>
<p>`<br>
num_features = 2000</p>
<p>class MyEnsemble(nn.Module):</p>
<pre><code>def __init__(self, model, model1):

    super(MyEnsemble, self).__init__()
    
    self.model = model

    self.model1 = model1

    # vowel_diacritic

    self.fc1 = nn.Linear(num_features,11)

    # grapheme_root

    self.fc2 = nn.Linear(num_features,168)

    # consonant_diacritic

    self.fc3 = nn.Linear(num_features,7)

    
def forward(self, x):
    x1 = self.model(x.clone()) 
    x1 = x1.view(x1.size(0), -1)
    x2 = self.model1(x)
    x2 = x2.view(x2.size(0), -1)
    x = torch.cat((x1, x2), dim=1)
    x1 = self.fc1(x)
    x2 = self.fc2(x)
    x3 = self.fc3(x)
    return x1,x2,x3
`
</code></pre>
<p>but i get the error : <strong>TypeError: expected Tensor as element 0 in argument 0, but got tuple</strong></p>
<p>how do i modify ensembler code to get rid of this error? thank you</p>
Why can a 1*n and a n*1 tensor be added together?

username: Vikings
cooked: <p>I’m a beginner of PyTorch and when I trying to run the following code:</p>
<blockquote>
<p>x=torch.tensor([[5,3],[5,3]])<br>
z1=x.view(1,4)<br>
z2=x.view(4,1)<br>
print(z1+z2)</p>
</blockquote>
<p>I got:</p>
<blockquote>
<p>tensor([[10,  8, 10,  8],<br>
[ 8,  6,  8,  6],<br>
[10,  8, 10,  8],<br>
[ 8,  6,  8,  6]])</p>
</blockquote>
<p>Why doesn’t the compiler throw an error? Why two tensors with different dimensions can be added together?</p>
<p>Thank you!</p>

username: ptrblck
cooked: <p>This behavior is called broadcasting and tries to stay as close as possible to numpy’s implementation, which is described <a href="https://docs.scipy.org/doc/numpy/user/basics.broadcasting.html">here</a>.</p>

username: Vikings
cooked: <p>Got it.<br>
Thank you for your help!</p>
SubsetRandomSampler with having same batch every epoch?

username: Mohamed_Ragab
cooked: <p>I train on 10% percentage of my data. I use subset randomsampler but it seems that it select different subsets each epoch, How can I fix the same subset each epoch</p>
<p>subset_idx = torch.arange(1000) # 10%<br>
my_sampler=SubsetRandomSampler(subset_idx)<br>
train_dl = DataLoader(MyDataset(X_train_Si,y_train_Si), batch_size=hparams[‘batch_size’],sampler=my_sampler,drop_last=False)</p>

username: ptrblck
cooked: <p>This shouldn’t be the case, as <code>SubsetRandomSampler</code> just uses the permuted indices as seen <a href="https://github.com/pytorch/pytorch/blob/973371139475834c44d118ba0256d97b070980a0/torch/utils/data/sampler.py#L124">here</a>.</p>

username: Mohamed_Ragab
cooked: <p>I agree with you, but I selected single batch and printed the values of this batch each epoch and they weren’t similar, that’s why I was surprised.</p>

username: ptrblck
cooked: <p>Were you printing the data tensors of each batch and are you using any random transformations?<br>
How did you verify that the subsets differ?</p>

username: Mohamed_Ragab
cooked: <p>I don’t use any random transformation. First the batch has single tensor and for each epoch I do this:<br>
a,b=iter(Pr_train_dl).next()<br>
print(a)</p>

username: ptrblck
cooked: <p>I’m not sure to understand the use case completely.<br>
If you are calling this line of core once per epoch, you would only get a single batch.<br>
On the other hand, if you are calling it in each iteration, you are recreating the <code>Iterator</code> every time.</p>
<p>Could you try to iterate the <code>DataLoader</code> directly via:</p>
<pre><code class="lang-python">for data, target in loader:
    ...
</code></pre>
<p>and check the results?</p>

username: Mohamed_Ragab
cooked: <p>Thanks, it works, I think that the I was using for extracting single batch was taking single batch from whole data and not restricted to the subset sampler, when I tried :for data, target in loader: it is the same each epoch</p>
How to unfold tensor properly to split a large tensor into blocks?

username: Yuerno
cooked: <p>Hi all, so this is a follow-up to a very similar question I had recently, except specifically relating to the unfolding function, since I can’t seem to figure out how to use it properly to split a large tensor into smaller blocks (as if one were running a 3D filter across the tensor). Hopefully this drawing below better clarifies what I’m trying to achieve:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/a/2a77468d546b2166e96e12b519df9496bfd50741.png" alt="Untitled" data-base62-sha1="63FBetBVFmBwObFUdTdcDTWJoEp" width="282" height="365"></p>
<p>First off, I’m generating a 4x4x4x3 tensor of 3D coordinate points using the following code:</p>
<pre><code class="lang-auto">shape = (4,)*3
size = shape[0] * shape[1] * shape[2]

pxs = torch.linspace(-1, 1, shape[0])
pys = torch.linspace(-1, 1, shape[1])
pzs = torch.linspace(-1, 1, shape[2])

pxs = pxs.view(-1, 1, 1).expand(*shape).contiguous().view(size)
pys = pys.view(1, -1, 1).expand(*shape).contiguous().view(size)
pzs = pzs.view(1, 1, -1).expand(*shape).contiguous().view(size)
points = torch.stack([pxs, pys, pzs], dim=1)

points = torch.reshape(points, (eval_dim, eval_dim, eval_dim, 3))
</code></pre>
<p>Next, I’m using the following unfolding operations to split this tensor into eight 2x2x2 blocks:</p>
<pre><code class="lang-auto">patches = rotated_grids_3d.unfold(2, 2, 2).unfold(1, 2, 2).unfold(0, 2, 2)
patches = patches.contiguous().view(-1, 2, 2, 2, 3)
</code></pre>
<p>However, no matter the different unfolding orderings I try, I can’t seem to get the blocks to be nearby points. For example, 2x2x2 block should be the back, top-left corner of the cube, with coordinate points as follows:</p>
<pre><code class="lang-auto">print(rotated_grids_4_3d[0, 0, 0, :])
print(rotated_grids_4_3d[0, 0, 1, :])
print(rotated_grids_4_3d[0, 1, 0, :])
print(rotated_grids_4_3d[0, 1, 1, :])
print(rotated_grids_4_3d[1, 0, 0, :])
print(rotated_grids_4_3d[1, 0, 1, :])
print(rotated_grids_4_3d[1, 1, 0, :])
print(rotated_grids_4_3d[1, 1, 1, :])
</code></pre>
<p>Then the next block would be two points moved over, etc. etc. However, my unfolding results seem to give arbitrarily located points that don’t themselves form cubes, and there’s sometimes repeating points, and I haven’t been able to figure out what I’m doing wrong. Any help would be appreciated!</p>
How to backward one self-defined block while fixing other layers?

username: Lausanne
cooked: <p>Hi, I want to add a new module into the pretrained backbone, e.g. ResNet-101.(The module is added in block-level) Since ResNet-101 already has pretrained weight, while the new module still needs to be trained. My question is , can I set requires_grad=False in all layers excluding the new module in backbone? I am not sure the gradient can be passed normally in my case ?<br>
A simplified version is one net which consists of A-B-C layers. For now, I set A and C layers’ requires_grad=False. Can B normally accept the backward gradient ?<br>
Appreciate for anyone who can help me!</p>

username: ptrblck
cooked: <p>Yes, that should work. Autograd will make sure to propagate the gradient to previous modules/tensors, if necessary.</p>

username: Lausanne
cooked: <p>Thank you very much!</p>
How to reset variables' values in nn.Modules?

username: silent567
cooked: <p>Hi, I want to implement cross-validation efficiently. Therefore, I want to build one model(/neural network) and then reset the variables randomly for each fold in the cross-validation.</p>
<p>Is there a way to reset variables’ values just the same as creating a new model?</p>
<p>Some example codes are</p>
<pre><code class="lang-auto">class Toy(torch.nn.Module):
	def __init__(self):
		super(Toy,self).__init__()
		self.fc = torch.nn.Linear(3,4)
	def forward(self,x):
		return self.fc(x)
		
toy = Toy()
cross_validation_acc = []
for fold_num in range(5):
	train(toy)
	cross_validation_acc.append(test(toy))
	reset_variables(toy)
print(np.mean(cross_validation_acc))
</code></pre>
<p>Thank you very much!</p>

username: justusschock
cooked: <p>You could simply write an initialization function, which could look like this:</p>
<pre><code class="lang-auto">def reset_parameters(self):
    for m in self.modules():
            if isinstance(m, nn.Conv2d):
                nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')
            elif isinstance(m, nn.BatchNorm2d):
                nn.init.constant_(m.weight, 1)
                nn.init.constant_(m.bias, 0)
</code></pre>
<p>and then call <code>toy.reset_parameters()</code> at the beginning of each fold.</p>
<blockquote>
<p><strong>Note</strong>: The example has been taken from <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py" rel="noopener nofollow ugc">the torchvision github repo</a> and does not contain an initialization for <code>torch.nn.Linear</code>. You would have to adapt this for your usecase.</p>
</blockquote>

username: silent567
cooked: <p>Great. I will also search the default initialization methods for other modules. Thank you very much for your help!</p>

username: justusschock
cooked: <p>Great! But if you don’t use SGD without momentum as optimizer, you would also have to reinitialize your optimizer’s <code>state_dict</code> for comparable results.</p>

username: silent567
cooked: <p>So, it’s like to firstly record the <code>Optimizer.state_dict()</code> and then use <code>Optimizer.load_state_dict()</code> to reset?</p>

username: justusschock
cooked: <p>yes, this should work.</p>

username: episodeyang
cooked: <blockquote>
<p>So, it’s like to firstly record the  <code>Optimizer.state_dict()</code>  and then use  <code>Optimizer.load_state_dict()</code>  to reset?</p>
</blockquote>
<p>This is really good.</p>
Tensorborad error: AttributeError: 'Tensor' object has no attribute 'param_init_net'

username: Debasish_Jyotishi
cooked: <hr>
<p>AttributeError                            Traceback (most recent call last)<br>
 in <br>
1 tb=SummaryWriter<br>
2 graph_data=next(iter(train_loader))[‘data’]<br>
----&gt; 3 tb.add_graph(model1,graph_data)<br>
4 tb.close()</p>
<p>~/anaconda3/lib/python3.7/site-packages/torch/utils/tensorboard/writer.py in add_graph(self, model, input_to_model, verbose)<br>
695             else:<br>
696                 # Handles cnn.CNNModelHelper, model_helper.ModelHelper<br>
–&gt; 697                 current_graph = model_to_graph_def(model)<br>
698             event = event_pb2.Event(<br>
699                 graph_def=current_graph.SerializeToString())</p>
<p>~/anaconda3/lib/python3.7/site-packages/torch/utils/tensorboard/_caffe2_graph.py in model_to_graph_def(model, **kwargs)<br>
772             **kwargs. See _operators_to_graph_def for detailed **kwargs.<br>
773     ‘’’<br>
–&gt; 774     nets = [model.param_init_net, <a href="http://model.net" rel="nofollow noopener">model.net</a>]<br>
775     return nets_to_graph_def(nets, **kwargs)<br>
776</p>
<p>AttributeError: ‘Tensor’ object has no attribute ‘param_init_net’</p>

username: JiaojiaoFan
cooked: <p>Hey, I met this problem because I used writer.add_graph(some_tensor, input)</p>
PyTorch LSTM has nan for MSELoss

username: shamoons
cooked: <p>My model is:</p>
<pre><code class="lang-auto">class BaselineModel(nn.Module):
    def __init__(self, feature_dim=5, hidden_size=5, num_layers=2, batch_size=32):
        super(BaselineModel, self).__init__()
        self.num_layers = num_layers
        self.hidden_size = hidden_size

        self.lstm = nn.LSTM(input_size=feature_dim,
                            hidden_size=hidden_size, num_layers=num_layers)

    def forward(self, x, hidden):
        lstm_out, hidden = self.lstm(x, hidden)
        return lstm_out, hidden

    def init_hidden(self, batch_size):
        hidden = Variable(next(self.parameters()).data.new(
            self.num_layers, batch_size, self.hidden_size))
        cell = Variable(next(self.parameters()).data.new(
            self.num_layers, batch_size, self.hidden_size))
        return (hidden, cell)
</code></pre>
<p>Training looks like:</p>
<pre><code class="lang-auto">
    train_loader = torch.utils.data.DataLoader(
        train_set, batch_size=BATCH_SIZE, shuffle=True, **params)

    model = BaselineModel(batch_size=BATCH_SIZE)
    optimizer = optim.Adam(model.parameters(), lr=0.01, weight_decay=0.0001)
    loss_fn = torch.nn.MSELoss(reduction='sum')

    for epoch in range(250):

        # hidden = (torch.zeros(2, 13, 5),
        #           torch.zeros(2, 13, 5))
        # model.hidden = hidden
        for i, data in enumerate(train_loader):
            hidden = model.init_hidden(13)
            inputs = data[0]
            outputs = data[1]

            print('inputs',  inputs.size())
            # print('outputs', outputs.size())

            # optimizer.zero_grad()
            model.zero_grad()

            # print('inputs', inputs)
            pred, hidden = model(inputs, hidden)

            loss = loss_fn(pred, outputs)

            loss.backward()
            torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
            optimizer.step()

            print('Epoch: ', epoch, '\ti: ', i, '\tLoss: ', loss)
</code></pre>
<p>I have gradient clipping set already, which seems to be the recommended solution. But after even the first step, I get:</p>
<pre><code class="lang-auto">Epoch:  0       i:  0   Loss:  tensor(nan, grad_fn=&lt;MseLossBackward&gt;)
</code></pre>
<p>If anyone is interested in magic internet points, I also posted <a href="https://stackoverflow.com/questions/60476943/pytorch-lstm-has-nan-for-mseloss" rel="nofollow noopener">here</a></p>
How to change manually the size of a layer without breaking autograd?

username: Zen
cooked: <p>I want to change the size of the tensors <code>weight</code> and <code>bias</code> in a <code>torch.nn.Linear</code> layer. Even if I set <code>weight.grad = None</code> and <code>bias.grad = None</code>, the backward pass returns an error. A possible way to solve the problem would be an entire re-initialization of the computational graph, but I don’t know how to do it. However, here is a minimal code:</p>

<pre><code>import torch

l = torch.nn.Linear(5, 4)
x = torch.randn(1, 5)
y = l(x).pow(2).mean()
y.backward()

l.weight.data = torch.randn(3, 5)
l.weight.grad = None
l.bias.data = torch.randn(3)
l.bias.grad = None
z = l(x).pow(2).mean()
print(z)
z.backward()
</code></pre>
<p>But I get the error:</p>

<pre><code>RuntimeError: Function ExpandBackward returned an invalid gradient at index 0 - expected shape [4] but got [3]
</code></pre>
<p>Somehow, it seems that autograd keeps intermediary computations, and use it to recompute the gradients, hence a necessary re-initialization of the whole computational graph.<br>
Since many objects in my code contain references to the layers I use, I don’t want to re-create a layer each time I am deleting a row or a column in it.</p>

username: tom
cooked: <p>At the risk of being blunt: <a href="https://pytorch.org/blog/pytorch-0_4_0-migration-guide/#what-about-data" rel="nofollow noopener">Using  <code>.data</code> has been a bad idea for more than half a year</a>.</p>
<p>When you try to assign <code>.weight</code> and <code>.bias</code>, PyTorch will tell you exactly what to do, i.e. have a <code>nn.Parameter</code> to assign (I know, because I tried with a tensor before that):</p>
<pre><code class="lang-python">l = torch.nn.Linear(5, 4)
x = torch.randn(1, 5)
y = l(x).pow(2).mean()
y.backward()

l.weight = torch.nn.Parameter(torch.randn(3, 5))
l.bias = torch.nn.Parameter(torch.randn(3))
z = l(x).pow(2).mean()
print(z)
z.backward()
</code></pre>
<p>Best regard and stay clear of <code>.data</code></p>
<p>Thomas</p>

username: Zen
cooked: <p>Ok. This means that I cannot store the reference to a Parameter object if its size is likely to change: a change of size means reassignment of the Parameter object (and not only the .data).<br>
For information, I reused old pieces of code that worked in the older versions of PyTorch.</p>

username: Ge0rges
cooked: <p>How would you do this in 0.3.0 before using .data was a bad idea? I’m encountering a similar challenge <a href="https://discuss.pytorch.org/t/pytorch-duplicate-a-neuron-in-a-layer-and-change-size/71539">here</a>.</p>
Try to train a neural network's derivative function

username: JiaojiaoFan
cooked: <p>Hi!<br>
I wonder anyone has tried training a NN’s derivative function using pytorch. I have seen someone is using tensorflow to do <a href="https://github.com/wiseodd/generative-models/blob/master/GAN/improved_wasserstein_gan/wgan_gp_tensorflow.py" rel="nofollow noopener">this</a><br>
For example, in the supervise learning. The input and labels keep unchanged, the real output is got from the autograd from the neural network’s output to the neural network’s input?</p>
Running model train cycle inside a function

username: blidger
cooked: <p>Hi! I am trying to define a python function with learning rate as the only parameter, which would instantiate a pytorch model, train it using an optimizer with the passed learning rate, and then print out training statistics. My code looks like this:</p>
<pre><code class="lang-auto">def get_lr_performance(loc_lr):
    loc_model = nn.Sequential(
    #some modules
    )
    loc_opt = torch.optim.SGD(loc_model.parameters(), lr=loc_lr)
    train_model(loc_model, loc_opt, train_loader)
</code></pre>
<p>where train_model is defined as follows:</p>
<pre><code class="lang-auto">def train(model, optimizer, dataloader): 
    model.train()
    stats = defaultdict(list)
    for x, y in dataloader:
        optimizer.zero_grad()
        output = model(x)
        loss = F.nll_loss(output, y, reduction='sum')
        loss.backward()
        optimizer.step()
        # save the statistics of interest to stats
    return stats

def train_model(model, optimizer, train_loader, epochs=7):
    stats = defaultdict(list)
    for epoch in range(epochs):
        train_stats = train(model, opt, train_loader)
        plot_stats(train_stats) # a basic plotting function using plt
</code></pre>
<p>The training stats do not improve during training when calling get_lr_performance(loc_lr) at any loc_lr. However, executing</p>
<pre><code class="lang-auto">loc_model = nn.Sequential(
#some modules
)
loc_opt = torch.optim.SGD(loc_model.parameters(), lr=loc_lr)
train_model(loc_model, loc_opt, train_loader)
</code></pre>
<p>without the function trains the model as expected. I suppose this issue is somehow related to the pytorch optimizer (im)mutability, but I am not sure. Is there a way to make the function work properly?</p>

username: ptrblck
cooked: <p>You are passing <code>opt</code> to <code>train</code>, while <code>train_model</code> gets <code>optimizer</code> passed in.<br>
Could <code>opt</code> refer to some global optimizer?</p>

username: blidger
cooked: <p>Hi, thank you for reply!<br>
Yes, I actually had a global optimizer under the “opt” variable. Changing from</p>
<pre><code class="lang-auto">train_stats = train(model, opt, train_loader)
</code></pre>
<p>to</p>
<pre><code class="lang-auto">train_stats = train(model, optimizer, train_loader)
</code></pre>
<p>in train_model() fixed the issue.</p>
Extracting image patches given a coordinate list without looping

username: maunz
cooked: <p>Hi!<br>
I have an image with a long list of image indices. For each index-pair I would like to extract a 4x4 patch around that coordinate. At the moment I’ve solved it by looping (as can be seen in the code below).</p>
<p>I’m trying to speed up the calculations and was wondering if I could achieve the same thing without looping. Any ideas?</p>
<pre><code class="lang-auto">import torch

image = torch.rand((3, 200, 300))

# randomize indizes so that all patches are inside image
indy = torch.randint(196, (1000, )) + 1
indx = torch.randint(296, (1000, )) + 1

# extract patches for all images
all_patches = torch.zeros((1000, 3, 4, 4), dtype=image.dtype)
for j in [-1, 0, 1, 2]:
    for k in [-1, 0, 1, 2]:
        all_patches[:, :, j+1, k+1] = image[:, indy+j, indx+k].transpose(0, 1)
</code></pre>
<p>Thanks!</p>

username: Frida
cooked: <p><img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji only-emoji" alt=":grinning:"><br>
Found???</p>
Train on MSE with transfomed neural network output

username: Brunobio
cooked: <p>I would like to train a NN that miminizes the MSE between a <em>transformation</em> of the output of the NN, <em>i.e.</em> :</p>
<pre><code class="lang-auto">x_pred = NN(x)
y = f(x_pred)
loss_func(y_pred,x)
</code></pre>
<p>where f is a trasnformation of x_pred. I tried doing like this:</p>
<pre><code class="lang-auto">    # train the network
    for epoch in range(EPOCH):

        # for each training step
        for step, (batch_x, batch_y) in enumerate(loader):
            b_x = Variable(batch_x)
            b_y = Variable(batch_y)

            prediction = net(b_x)  # input x and predict based on x

            x_plus = integrate(sampling_interval, prediction, b_x)

            loss = loss_func(x_plus, b_y)  # must be (1. nn output, 2. target)

            optimizer.zero_grad()  # clear gradients for next train
            loss.backward()  # backpropagation, compute gradients
            optimizer.step()  # apply gradients
            print(loss.item())
</code></pre>
<p>where the function integrate (<code>f</code>) is</p>
<pre><code class="lang-auto">def integrate(sampling_interval, r, x0):
    M = 4  # RK4 steps per interval
    DT = sampling_interval / M
    X = x0
   
    for j in range(M):
        k1 = ode(X, r)
        k2 = ode(X + DT / 2 * k1, r)
        k3 = ode(X + DT / 2 * k2, r)
        k4 = ode(X + DT * k3, r)
        X = X + DT / 6 * (k1 + 2 * k2 + 2 * k3 + k4)

    return X


def ode(x0, r):
    qF = r[0, 0]
    qA = r[0, 1]
    qP = r[0, 2]
    mu = r[0, 3]

    FRU = x0[0, 0]
    AMC = x0[0, 1]
    PHB = x0[0, 2]
    TBM = x0[0, 3]

    fFRU = qF * TBM  
    fAMC = qA * TBM 
    fPHB = qP - mu * PHB
    fTBM = mu * TBM

    return torch.tensor([fFRU, fAMC, fPHB, fTBM])

</code></pre>
<p>But I get when I do this the MSE jumps from one number to the other and it does not converge. Am I making some mistakes?</p>
<p>Thanks a lot</p>

username: Brunobio
cooked: <p>I found the problem, I was losing the gradient at the end of the ode function, because I was  creating a new tensor instead of simply concatenate the tensor like so:</p>
<pre><code class="lang-auto">    def ode(self, x0, r):
        qF = r[0, 0]
        qA = r[0, 1]
        qP = r[0, 2]
        mu = r[0, 3]

        FRU = x0[0, 0]
        AMC = x0[0, 1]
        PHB = x0[0, 2]
        TBM = x0[0, 3]

        fFRU = qF * TBM 
        fAMC = qA * TBM 
        fPHB = qP - mu * PHB
        fTBM = mu * TBM

        return torch.stack((fFRU, fAMC, fPHB, fTBM), 0)
</code></pre>
Saving and loading tensors as training data

username: R_Tux
cooked: <p>I have created my training data for a CNN and would like to save it for future training.<br>
How would I go about saving my training data and then loading it again?</p>
<p>My training data looks like:</p>
<pre><code class="lang-python">X.shape
Out[346]: torch.Size([30, 1, 362, 362])

y.shape
Out[347]: torch.Size([30, 131044, 3])
</code></pre>

username: ptrblck
cooked: <p><code>torch.save</code> and <code>torch.load</code> should work. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: R_Tux
cooked: <p>That works,</p>
<p>cheers <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Trianing GAN give me something wrong

username: Xu_YI
cooked: <p>I am trying GAN to genarate human face,but it does not works well.here is my output:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/e/ee701bd4061d5edcb53a8205d27ccf5edcc22d32.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/ee701bd4061d5edcb53a8205d27ccf5edcc22d32" title="R({MB%2X$A8IZ05TN~)$GJ"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee701bd4061d5edcb53a8205d27ccf5edcc22d32_2_690x346.jpeg" alt="R({MB%2X$A8IZ05TN~)$GJ" data-base62-sha1="y1jPrAq3tflnjEkUNCG6C4de7E6" width="690" height="346" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee701bd4061d5edcb53a8205d27ccf5edcc22d32_2_690x346.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/e/e/ee701bd4061d5edcb53a8205d27ccf5edcc22d32.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/e/ee701bd4061d5edcb53a8205d27ccf5edcc22d32.jpeg 2x" data-dominant-color="CBCBB8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">R({MB%2X$A8IZ05TN~)$GJ</span><span class="informations">921×463 277 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I random some numbers ,batchsize==32,I input numbers in Generator,but the output is the same</p>
<p>I search on the Internet and find some advice :<br>
1,first train Generator with real data,then use fake data ,then train Discriminator.<br>
2,use batchnorm layer<br>
3,use another good backbone(I am use stargan)<br>
4,change real label from 1 to (0.7-1),fake label from 0 to (0-0.3)<br>
5,give smaller batch_size.</p>
<p>all of them have no use<br>
is that a gradient vanish?I am using nn.BCEWithLogitsLoss(),is that cause of a sigmod in nn.BCEWithLogitsLoss()？do I have a better choice?</p>
<p>Thanks a lot</p>
This code is giving me an error on transfer learning for computer vision tutorial WHY?

username: R_D
cooked: <p>#<span class="mention">@title</span> Default title text</p>
<h1>Data augmentation and normalization for training</h1>
<h1>Just normalization for validation</h1>
<p>data_transforms = {<br>
‘train’: transforms.Compose([<br>
transforms.RandomResizedCrop(224),<br>
transforms.RandomHorizontalFlip(),<br>
transforms.ToTensor(),<br>
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])<br>
]),<br>
‘val’: transforms.Compose([<br>
transforms.Resize(256),<br>
transforms.CenterCrop(224),<br>
transforms.ToTensor(),<br>
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])<br>
]),<br>
}<br>
data_dir = ‘data/hymenoptera_data’<br>
image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x),Preformatted textdata_transforms[x])for x in [‘train’, ‘val’]}<br>
dataloaders = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=4,<br>
shuffle=True, num_workers=4)<br>
for x in [‘train’, ‘val’]}<br>
dataset_sizes = {x: len(image_datasets[x]) for x in [‘train’, ‘val’]}<br>
class_names = image_datasets[‘train’].classes</p>
<p>device = torch.device(“cuda:0” if torch.cuda.is_available() else “cpu”)</p>

username: ptrblck
cooked: <p>Could you post the error message please?</p>
How to use hook to visualise feature maps

username: CSstudent
cooked: <p>I’ve written the following code to visualise the feature maps (four from each layer) from a fully-trained CNN network, by passing images forward through the network. I’ve seen examples where layer activations are saved by registering a hook into the network but how do I do this?</p>
<pre><code class="lang-auto">def feature_map_visualisation(images, image_index):

    images = images.to(device)

    conv1_activation = model_gpu.first_layer[0](images)

    conv1_active_relu = model_gpu.first_layer[1](conv1_activation)

    conv1_active_pooling = model_gpu.first_layer[2](conv1_active_relu)

    conv1_active_drop = model_gpu.first_layer[3](conv1_active_pooling)

    conv2_activation = model_gpu.second_layer[0](conv1_active_drop)

    figure = plt.figure(figsize = (16,16))

    k = 0

for i in range(0, 4):

      k += 1

      figure.add_subplot(1,4,k)

      img = conv1_activation[image_index][i].cpu().detach().clone().numpy()

      img = rgb2gray(img)

      plt.imshow(img, cmap="gray")

    j=0

for i in range(0,4):

      j+=1

      figure.add_subplot(2,4,j)

      img1=conv2_activation[image_index][i].cpu().detach().clone().numpy()

      img1 = rgb2gray(img1)

      plt.imshow(img1, cmap="gray")
</code></pre>

username: ptrblck
cooked: <p><a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">Here</a> is a small example of using hooks to get the activations. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
FasterRCNN Batch Based Usage

username: ahkarami
cooked: <p>Can we use the <code>Faster-RCNN</code> model of torchvision, batch-based on the inference time?<br>
for example:</p>
<pre><code class="lang-python">import torch
from torchvision import models

model = models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
model.eval()

img = torch.randn(2, 3, 224, 224)  # [N, C, H, W]

pred = model([img])  # Pass the image to the model
</code></pre>
<p>the above code has error. Is there any solution to address it and use faster-rcnn batch-based?</p>

username: ahkarami
cooked: <p>Dear <a class="mention" href="/u/ptrblck">@ptrblck</a>,<br>
Would you please kindly answer the above question?</p>

username: ptrblck
cooked: <p>Batching should be done under the hood, if you pass a list of tensors.<br>
<a href="https://github.com/pytorch/vision/blob/b2e95657cd5f389e3973212ba7ddbdcc751a7878/torchvision/models/detection/generalized_rcnn.py#L66">self.transform</a> will e.g. resize all tensors to <code>[3, 800, 800]</code> and create a batch as <code>[batch_size, 3, 800, 800]</code>.</p>
[feature request] adding **kwargs to all existing classes?

username: justanhduc
cooked: <p>Hi. I just want to ask if it is possible to add <code>**kwargs</code> to all existing classes? the current class <code>__init__</code> makes it very inconvenient to perform multiple inheritance.</p>

username: richard
cooked: <p>Could you provide an example of how this is a benefit? I’m not sure how multiple inheritance looks like in python.</p>

username: justanhduc
cooked: <p>thanks for the reply. my need for <code>**kwargs</code> arises from the following kinda situation. i want to add some attributes and methods to, say, the nn.Conv2D class. it will be like</p>
<pre><code class="lang-auto">class Foo:
    def __init__(self, foo1, foo2, **kwargs):
        pass

class FooConv2D(nn.Conv2D, Foo):
    def __init__(self, in_channels, ou_channels, ..., foo1, foo2, **kwargs):
        ....
</code></pre>
<p>as you can see, i cant initialize in one call with <code>super</code> because of the different signatures of the parents. of course i dont have to use <code>super()</code>, but well, isnt it uglier?</p>

username: xvdp
cooked: <p>I second the request. for a different reason.<br>
Im working on automl. Ensembling different model types. I have a master function that tests different models, often without changing the args, but I do pass **kwargs to all of them.<br>
the model functions (e.g. resnet18()) contain **kwargs, but the base classes (eg ResNet() do not. adding **kwargs to the <strong>init</strong>() of the base model classes would allow me not to have to resort to either (cloning the models folder and changing it myself) or this horrendous hack</p>
<pre><code class="lang-python"># I pass model_name,  pretrained, num_classes and **kwargs to all
# e.g.
_model_check = models.__dict__[model_name](pretrained=pretrained, num_classes=num_classes)
_args = models.__dict__[_model_check.__class__.__name__].__init__.__code__.co_varnames
_kwargs = {k:kwargs[k] for k in kwargs if k in _args}
self.model = models.__dict__[model_name](pretrained=pretrained, num_classes=num_classes, **_kwargs).to(device=self.device)
</code></pre>
Pytorch duplicate a neuron in a layer and change size

username: Ge0rges
cooked: <p>I am using pytorch 0.3.0. I’m trying to selectively copy a neuron and it’s weights within the same layer, then replace the original neuron with an another set of weights. Here’s my attempt at that:</p>
<pre><code class="lang-auto">reshaped_data2 = data2.unsqueeze(0)
new_layer_data = torch.cat([new_layer.data, reshaped_data2], dim=0)
new_layer_data[i] = data1
new_layer.data.copy_(new_layer_data)
</code></pre>
<p>First I unsqueezed  <code>data2</code>  to make it a  <code>1*X</code>  tensor instead of  <code>0*X</code> . Then I concatenate my layer’s tensor with the reshaped  <code>data2</code>  along dimension 0. I then replace the original  <code>data2</code>  located at index  <code>i</code>  with  <code>data1</code> . Finally, I copy all of that into my layer.</p>
<p>The error I get is:</p>
<pre><code class="lang-auto">RuntimeError: inconsistent tensor size, expected tensor [10 x 128] and src [11 x 128] to have the same number of elements, but got 1280 and 1408 elements respectively at /Users/soumith/code/builder/wheel/pytorch-src/torch/lib/TH/generic/THTensorCopy.c:86
</code></pre>
<p>If I do a simple assignment instead of copy I get</p>
<pre><code class="lang-auto">RuntimeError: The expanded size of the tensor (11) must match the existing size (10) at non-singleton dimension 1. at /Users/soumith/code/builder/wheel/pytorch-src/torch/lib/TH/generic/THTensor.c:309
</code></pre>
<p>I understand the error, but what is the right way to go about this?</p>
<p>PS: Also asked on SO, get some karma <a href="https://stackoverflow.com/questions/60458029/pytorch-copy-a-neuron-in-a-layer" rel="nofollow noopener">here</a></p>
How to modify values inside input dataset

username: harryZZZ
cooked: <p>Hello I have a dataset object using torch.utils.data as input. I can access to the input dataset object using index and see they are all tensors.</p>
<p>I want to update some of the values in my input after each iteration based on my algorithm, how do I do that?</p>
<p>I tried the way of modify a tensor as documentation said but the values do not change:<br>
example:</p>
<p>in raw data, I have data[‘img’] and data[‘prob’]<br>
and I use datasets and dataloader function with data tranform.</p>
<p>for i in range(0, len(dataset):<br>
dataset[i][0][‘prob’][0] = calculated_value[i].item(0)</p>
<p>the value won’t change.</p>
<p>Please help! Really appreciated!</p>

username: Naman-ntc
cooked: <p>Do you want to change the values after an epoch (after a call of enumarate dataset) or during an epoch as well???</p>

username: harryZZZ
cooked: <p>To be more accurate, after several full run, not in each epoch. So basically I run a net fit several times with cross validation and then update the input data and re-run the whole process.</p>

username: Naman-ntc
cooked: <p>why not create a function inside the <code>Dataset object</code>, something like <code>update</code> which might take arguments required and updates the data[‘prob’]. You will have have freedom to call the function when you want (after how many cross validations you want…)<br>
I believe this should work, have done something similar myself once</p>

username: harryZZZ
cooked: <aside class="quote no-group" data-username="Naman-ntc" data-post="4" data-topic="19836">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/naman-ntc/48/32044_2.png" class="avatar"> Naman-ntc:</div>
<blockquote>
<p>I believe this should work, have done something similar myself once</p>
</blockquote>
</aside>
<p>Thank you for the suggestion I will try to implement that.<br>
Still I wonder why the simple modification directly towards tensors inside the dataset object is not working.</p>

username: diningphil
cooked: <p>Any luck? I am interested in doing the same thing.</p>
Pytorch tensor data is disturbed when I convert data loader tensor to numpy array

username: benyaminelc90
cooked: <p>Hi,<br>
I am using a simple train loop for a regression task. To make sure that regression ground-truth values are the same as what I expect in the training loop, I decided to plot each batch of data.<br>
However, I see that when I convert the data loader’s tensor to numpy array and plot it, it is disturbed. I am using myTensor.data.cpu().numpy() for the conversion.</p>
<p>My code is as below:</p>
<pre><code class="lang-auto">train_ds = TensorDataset(x_train, y_train)
train_dl = DataLoader(train_ds, batch_size = 32, shuffle = True, num_workers = 0, drop_last = True)


for epoch in range(epochs):
model.train()
for i, (x, y) in enumerate(train_dl):
       x = x.cuda()
       y = y.cuda()
       yy = y.data.cpu().numpy()
       pyplot.plot(yy[0:32,0])
       pyplot.show()
</code></pre>
<pre><code>   ...</code></pre>

username: benyaminelc90
cooked: <p>Here are the outputs</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/5/45368ec628b70440222f122acfe563d3f840340f.png" data-download-href="https://discuss.pytorch.org/uploads/default/45368ec628b70440222f122acfe563d3f840340f" title="3"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/5/45368ec628b70440222f122acfe563d3f840340f_2_690x246.png" alt="3" data-base62-sha1="9ShPL2wkA9jWy5ryyNkxN9Tr5jF" width="690" height="246" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/5/45368ec628b70440222f122acfe563d3f840340f_2_690x246.png, https://discuss.pytorch.org/uploads/default/original/3X/4/5/45368ec628b70440222f122acfe563d3f840340f.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/5/45368ec628b70440222f122acfe563d3f840340f.png 2x" data-dominant-color="F7F9FA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">3</span><span class="informations">857×306 35.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: benyaminelc90
cooked: <aside class="quote no-group" data-username="benyaminelc90" data-post="1" data-topic="71558">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/ec9cab/48.png" class="avatar"> benyaminelc90:</div>
<blockquote>
<p>train_ds = TensorDataset(x_train, y_train) train_dl = DataLoader(train_ds, batch_size = 32, shuffle = True, num_workers = 0, drop_last = True) for epoch in range(epochs): model.train() for i, (x, y) in enumerate(train_dl): x = x.cuda() y = y.cuda() yy = y.data.cpu().numpy() pyplot.plot(yy[0:32,0]) pyplot.show()</p>
</blockquote>
</aside>
<p>I think it is because I set shuffle = True in data loader.<br>
If I set it to false, it is fine. However, How can I shuffle training batches after each epoch if I set shuffle = False in data loader then?</p>
Pairwise similarity matrix between a set of vectors

username: nullgeppetto
cooked: <p>Let’s suppose that we have a 3D tensor, where the first dimension represents the <code>batch_size</code>, as follows:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
x = torch.randn(32, 100, 25)
</code></pre>
<p>That is, for each <code>i</code>, <code>x[i]</code> is a set of 100 25-dimensional vectors. I would like to compute the similarity (e.g., the cosine similarity – but in general any such pairwise distance/similarity matrix) of these vectors for each batch item.</p>
<p>That is, for each <code>x[i]</code> I need to compute a <code>[100, 100]</code> matrix which will contain the pairwise similarities of the above vectors. More specifically, the (i,j)-th element of this matrix should contain the similarity (or the distance) between the i-th and the j-th row of (the 100x25) <code>x[t]</code>, for all <code>t=1, ..., batch_size</code>.</p>
<p>If I use <code>torch.nn.CosineSimilarity()</code>, no matter what <code>dim</code> I’m using, the result is either <code>[100, 25]</code> (<code>dim=0)</code>, or <code>[32, 25]</code> (<code>dim=1</code>) , where I need a tensor of size <code>[32, 100, 100]</code>. I would expect <code>torch.nn.CosineSimilarity()</code> to work this way (since, at least to me, it looks more intuitive), but it doesn’t.</p>
<p>Could that be done using something like below?</p>
<pre><code class="lang-auto">torch.matmul(x, x.permute(0, 2, 1))
</code></pre>
<p>I guess that this could give a distance matrix, but what if I need an arbitrary pairwise operation? Should I build this operation using the above?</p>
<p>Thank you.</p>