akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
TypeError: predict() takes 1 positional argument but 2 were given

username: chris_verdence
cooked: <p>I have tried to read the text on similar topics, but can’t seem to figure out whats wrong here.</p>
<p>This is the error:<br>
Traceback (most recent call last):<br>
File “”, line 1, in <br>
TypeError: predict() takes 1 positional argument but 2 were given</p>
<p>Here is the code:</p>
<pre><code class="lang-auto">class NeuralNetwork(nn.Module):
   def __init__(self, ):
      super(NeuralNetwork, self).__init__()
      self.inputSize = 2
      self.outputSize = 1
      self.hiddenSize = 3

      self.W1 = torch.randn(self.inputSize, self.hiddenSize)
      self.W2 = torch.randn(self.hiddenSize, self.outputSize)

   def forward(self, X):
      self.z = torch.matmul(X, self.W1)
      self.z2 = self.sigmoid(self.z)
      self.z3 = torch.matmul(self.z2, self.W2)
      o = self.sigmoid(self.z3)
      return o

   def sigmoid(self, s):
      return 1 / (1 + torch.exp(-s))

   def predict(self, x_predicted):
      print('Predicted data based on trained weights: ')
      print('Input (scaled): \n' + str(x_predicted))
         print('Output: \n' + str(self.forward(x_predicted)))
</code></pre>
<pre><code class="lang-auto">from package_name.class_name import NeuralNetwork
xPredicted = torch.load('data/xPredicted.pt')
NN = NeuralNetwork()
NN.predict(xPredicted)

</code></pre>
<p>xPredicted is correctly loaded and defined. What I don’t understand is why predict() only takes one positional argument when it has both self and x_predicted? And how do I set this up so that it works?</p>

username: ptrblck
cooked: <p>I cannot reproduce the issue using your code snippet and get this output:</p>
<pre><code class="lang-python">model = NeuralNetwork()
x = torch.randn(1, 2)
model.predict(x)
&gt; Predicted data based on trained weights: 
Input (scaled): 
tensor([[1.2890, 0.3778]])
Output: 
tensor([[0.8106]])
</code></pre>

username: chris_verdence
cooked: <p>Hi,</p>
<p>Not sure what was wrong, but now it works. Thank you!</p>
<p>Chris</p>
Mixed precision in evaluation

username: doctore
cooked: <p>Hi, I have large evaluation data set, which is the same size as the training data set and I’m performing the validation phase during training to be able to control the behavior of the training process.</p>
<p>I’ve added automatic mixed precision in the training phase, but is it safe to wrap the validation step in the training process with amp.autocast() to speed up that forward propagation?<br>
In general, is it safe/recommended to use mixed precision in model evaluation during the tuning process and if it is, what is the right way to implement?</p>
<pre><code>for epoch in range(epochs):
     # Training phase
     train_loss, train_score = self.train_model(trainset)

     # Validation phase
     valid_loss, valid_score = self.valid_model(validset)

@torch.no_grad()
def valid_model(self, dataloader):
     self.eval()    

     for batch in tqdm(dataloader):
          # Evaluate with mixed precision
          if self.setting.mixed_precision:

               # Runs the forward pass with autocasting, including loss and score calculation
               with amp.autocast():
                    loss, score = self.validation_step(batch)</code></pre>

username: ptrblck
cooked: <p>Yes, you can also use <code>autocast</code> during the validation step and wouldn’t need to apply the gradient scaling, since no gradient are calculated in this phase.</p>
How do I replace every ReLU in a Torchvision model with a different activation function?

username: Sam_Lerman
cooked: <p>As the title says: How do I replace every ReLU in a Torchvision model with a different activation function?</p>
<p>Say, I wanted to replace each ReLU with an ELU.</p>

username: ptrblck
cooked: <p>It depends which model you would like to use.<br>
While some <code>torchvision.models</code> use an attribute for the non-linearity, such as <code>self.act</code>, which can be easily replaced by a new assignment, others use the functional API, so that you would need to override the <code>forward</code> method.</p>

username: Sam_Lerman
cooked: <p>Thanks, that makes sense! Could you tell me which models use self.act and show me a snippet of how to change it?</p>

username: ptrblck
cooked: <p>The ResNet implementations use <code>self.relu</code> <a href="https://github.com/pytorch/vision/blob/be8192e20d2529fa552bcfc099974da45365ffd6/torchvision/models/resnet.py#L148">here</a> and you could replace it via:</p>
<pre><code class="lang-python">model = models.resnet18()
model.relu = nn.SELU()
</code></pre>
Android build cannot find pytorch_android:addHeadersTobundleReleaseAar

username: henry_huang
cooked: <p>hi all，</p>
<p>I am trying to build pytorch android from source by following this website: <a href="https://pytorch.org/mobile/android/#building-pytorch-android-from-source" class="inline-onebox" rel="noopener nofollow ugc">Android | PyTorch</a></p>
<p>I runned it on mac with zsh.</p>
<p>gradle: v5.6.4</p>
<p>jdk 8</p>
<p>witht newest ndk and sdk</p>
<pre><code class="lang-auto">git clone --recursive https://github.com/pytorch/pytorch
cd pytorch
git submodule sync
git submodule update --init --recursive

cd pytorch
sh ./scripts/build_pytorch_android.sh
</code></pre>
<p>at the end I recived the error:</p>
<blockquote>
<p>Task :pytorch_android:bundleReleaseAar FAILED</p>
<p>FAILURE: Build failed with an exception.</p>
<ul>
<li>Where:</li>
</ul>
<p>Build file ‘/Users/huanghenglin/pytorch/android/pytorch_android/build.gradle’ line: 115</p>
<ul>
<li>What went wrong:</li>
</ul>
<p>Execution failed for task ‘:pytorch_android:bundleReleaseAar’.</p>
<p>Could not find method execute() for arguments <span class="chcklst-box fa fa-square-o fa-fw"></span> on task ‘:pytorch_android:addHeadersTobundleReleaseAar’ of type org.gradle.api.DefaultTask.</p>
</blockquote>
<p>what should I do to finish the compilar?</p>
<p>note:</p>
<p>I did changed the build_android.sh, at line 58 from</p>
<blockquote>
<p>CMAKE_ARGS+=(“-DCMAKE_PREFIX_PATH=$(python -c ‘from distutils.sysconfig import get_python_lib; print(get_python_lib())’)”)<br>
CMAKE_ARGS+=(“-DPYTHON_EXECUTABLE=$(python -c ‘import sys; print(sys.executable)’)”)</p>
</blockquote>
<p>to</p>
<blockquote>
<p>CMAKE_ARGS+=(“-DCMAKE_PREFIX_PATH=$(python3 -c ‘from distutils.sysconfig import get_python_lib; print(get_python_lib())’)”)<br>
CMAKE_ARGS+=(“-DPYTHON_EXECUTABLE=$(python3 -c ‘import sys; print(sys.executable)’)”)</p>
</blockquote>
<p>since the system cannot find my python3, but I did not see it is a problem.</p>

username: henry_huang
cooked: <p>I used the pyenv to change the environment, it seems not a python issue</p>

username: 111380
cooked: <p>I resolved it by using gradle version 4.10.2</p>
Getting hook in the inception network

username: maoth
cooked: <p>Hi,<br>
I am using pytorch in my research about adversary machine learning. I am writing a code using inceptionV3 network. I am trying to get the input for the first layer(list(model.children())[0]), which the input should have 3 channel. The structure of the network is showed below:<br>
(See figure 1)<br>
However when I add hook in this layer, shows below in the deal_inception method, I can only get input information of one channel.<br>
(See figure 2)<br>
The first layer and output of the model is showed below:<br>
(See figure 3)<br>
From the size of the inputs which returned by the hook(get_hook method), it is as the following:<br>
(See figure 4)<br>
From the code showed before, the ‘inputs’ is the list where hook function store the input in, and this list only contain one element since we only inference one image in to the network. The inputs[0].shape should be (3,32,127,127), but it turns out (1,32,127,127) instead, it looks like only one channel input information or hooked.<br>
Could you tell me where the other two is, and how to revise my code to get the whole input about the first layer? Thanks.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/a/ea2c07df83993cc1d136347508eeb17d75267983.png" data-download-href="https://discuss.pytorch.org/uploads/default/ea2c07df83993cc1d136347508eeb17d75267983" title="082617462621_01"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/ea2c07df83993cc1d136347508eeb17d75267983_2_309x500.png" alt="082617462621_01" data-base62-sha1="xpA3WfCHtikLrgj5VLNbI98HDyz" width="309" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/ea2c07df83993cc1d136347508eeb17d75267983_2_309x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/ea2c07df83993cc1d136347508eeb17d75267983_2_463x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/ea2c07df83993cc1d136347508eeb17d75267983_2_618x1000.png 2x" data-dominant-color="F3F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">082617462621_01</span><span class="informations">1951×3149 577 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Could you post a code snippet by wrapping it into three backticks ```, which would make debugging easier, as the posted image is hard to read? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to assign value in 5-dimension tensor

username: wooeagle
cooked: <p>hi, i research to training for object tracker using pytorch.<br>
i have question about how to assign value in 5-dimension tensor.</p>
<p>code:<br>
tcls = torch.LongTensor(nB, nA, nGh, nGw, nC).fill_(0).cuda()   # create 5-dim tensor<br>
…</p>
<p>gt_index = gt_index_map[fg_index]  # fg_index shape is nA x nGh x nGw<br>
cls_index = t_cls[gt_index]  # get class id from ground truth<br>
print(cls_index)<br>
cls_index = cls_index.view(-1, 1)</p>
<p>print(‘------------- First ----------------’)<br>
target = tcls[b][fg_index]<br>
target.scatter_(1, cls_index, 1)<br>
print(target)<br>
print(target.shape)</p>
<p>print(‘--------------Second ------------------’)<br>
tcls[b][fg_index].scatter_(1, cls_index, 1)<br>
print(tcls[b][fg_index])<br>
print(tcls[b][fg_index].shape)<br>
print(tcls[b][fg_index].nonzero())</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/6/06990b1172206d69232a85fe9fefe6ce7b59ab70.png" data-download-href="https://discuss.pytorch.org/uploads/default/06990b1172206d69232a85fe9fefe6ce7b59ab70" title="1"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/6/06990b1172206d69232a85fe9fefe6ce7b59ab70.png" alt="1" data-base62-sha1="WmL1rOdwr6i0PCXZEWQrmNMLxC" width="375" height="500" data-dominant-color="191919"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1</span><span class="informations">617×822 2.64 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>First code was changed greatly! but second code is not changed…<br>
why not change value 0 to 1 in that tensor ??</p>

username: ptrblck
cooked: <p><code>tcls[b][fg_index].scatter_(1, cls_index, 1)</code> applies the <code>scatter_</code> on the slice (which is apparently a copy in this case) so that the manipulation won’t be applied on the original tensor.</p>
<p>Could you post a small code snippet with some desired values so that we could have a look at it?</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which would make debugging easier <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Laplace loss: nan issue involving two output feature values of the CNN

username: jia_lee
cooked: <p>I use a Laplace loss function in keypoint detection, here is the loss:</p>
<pre><code class="lang-auto">def laplace(norm, logb):
    """Element-wise laplace loss"""
    out = logb + torch.mul(norm, torch.exp(-logb))  # + 0.693147
    return out
</code></pre>
<p>The norm is the distance of A feature map with the Groudtruth, and the logb is the inferred value in B feature map.<br>
The forward process is OK and the loss is finite. But the nan issue happens when I did <code>optimizer.zero_grad()</code> and the training totally broke.</p>

username: jia_lee
cooked: <p>Could someone help me?</p>

username: ptrblck
cooked: <p>If the invalid values are created during the backward pass, you could add <code>torch.autograd.set_detect_anomaly(True)</code> at the beginning of your script, which should yield a stack trace pointing to the operation in the backward pass, which caused the NaNs.</p>
Additional inputs (constant matrices) in Loss function

username: So1
cooked: <p>Hi,</p>
<p>I’m implementing a loss function in Pytorch 1.7.0 but I don’t know how I can take an auxiliary matrix D (a constant matrix) as an input in addition to the output and target tensors it gets from the model?</p>
<pre><code class="lang-auto">def my_loss(output, target):
    loss = torch.mean((output - target)**2)
    return loss
model = nn.Linear(2, 2)
x = torch.randn(1, 2)
target = torch.randn(1, 2)
output = model(x)
loss = my_loss(output, target)
loss.backward()
print(model.weight.grad)
</code></pre>
<p>Any help?<br>
Regards,</p>

username: ptrblck
cooked: <p>You could add another argument to <code>my_loss</code> and pass this matrix <code>D</code> to it:</p>
<pre><code class="lang-python">def my_loss(output, target, D):
    ...
</code></pre>
<p>Let me know, if I misunderstood the question. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to apply transformation on a custom dataset that is sliced from cifar10?

username: Kalyan
cooked: <p>I have used the following code to get only 60% of ‘deer’ class and  60% of ‘dog’ class from Cifar10.</p>
<pre><code class="lang-auto">from torchvision.datasets import CIFAR10
from torch.utils.data import Subset

ds = CIFAR10(root='for_custom_dataset/', train=True, download=True)
dog_indices, deer_indices = [], []
dog_idx, deer_idx = ds.class_to_idx['dog'], ds.class_to_idx['deer']

for i in range(len(ds)):
  current_class = ds[i][1]
  if current_class == dog_idx:
    dog_indices.append(i)
  elif current_class == deer_idx:
    deer_indices.append(i)
dog_indices = dog_indices[:int(0.6 * len(dog_indices))]
deer_indices = deer_indices[:int(0.6 * len(deer_indices))]
new_dataset = Subset(ds, dog_indices+deer_indices)

</code></pre>
<p>I want to apply transformations on the 'new_dataset ’ which is of type ‘torch.utils.data.dataset.Subset object’.</p>
<p>I am aware of applying transformations on cifar10 directly like,</p>
<pre><code class="lang-auto">my_transforms = transforms.Compose([ transforms.ToTensor() ])

train_dataset = datasets.CIFAR10(root='dataset/', train=True, 
                                 transform=my_transforms, download=True)
train_loader = DataLoader(dataset=train_dataset, batch_size=batch_size, shuffle=True)
</code></pre>
<p>but, how can I perform the same transformation on ‘new_dataset’ object.</p>

username: ptrblck
cooked: <p>The passed <code>my_transforms</code> will still be applied on the dataset wrapped by the <code>Subset</code>.<br>
<code>Subset</code> is only a thin wrapper, which uses the specified indices and forwards it to the underlying <code>Dataset</code>. The same <code>Dataset.__getitem__</code> (with a subset of indices) will be used to load and transform the samples.</p>
Model Zoo equivalent for C++ loadable models

username: a_d
cooked: <p>Hello all,…<br>
I would like to deploy a solution using the pretrained ssd in c++ using the <code>torch::jit::load</code>. I tried loading the .pt file downloaded via torch.hub but it raised an error saying only serialised files can be loaded…<br>
I was wondering if there is some reliable/official source where I could download a variety of pretrained models which can be loaded by libtorch?<br>
TIA</p>

username: ptrblck
cooked: <p>You would have to script your model first, save it, and load it in C++ via <code>libtorch</code>.<br>
<a href="https://pytorch.org/tutorials/advanced/cpp_export.html">This tutorial</a> explains the steps. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: a_d
cooked: <p>Thanks for replying <a class="mention" href="/u/ptrblck">@ptrblck</a> I went through that immidiately after posting the query and tried to trace the <code>fasterrcnn-resnet50-fpn</code>. and came across error tracing it. Still trying though</p>
How to train two dataset on one model

username: agree
cooked: <p>I’ve two face image dataset, one for face alignment regression, one for age  regression. If I use one model to regression those two task like this:</p>
<pre><code class="lang-auto">class Net(nn.Module):
  def forward(self, x):
        x = self.features(x)
        x1 = self.features1(x)
        x2 = self.features2(x)
        return [x1, x2]  


class LossFN(nn.Module):
   def forward(self, input, target):
    landmark_loss = self.L1Loss1(input[0][0], target[0])
    age_loss = self.L1Loss2(input[1][1], target[1])
    loss = self.alignment_ratio * landmark_loss + age_loss
    return loss

for image, label in dataloader:
  image1 = image[0]
  image2 = image[1]
  output = []
  output.append(model(image1))
  output.append(model(image2))
  
  loss = loss_fn(output, label)
  optimizer.zero_grad()
  loss.backward()
  optimizer.step()
</code></pre>
<p>When I use this process for two task(regression and classification), the loss goes well, but when I use for two regression task, training loss goes well, while the valid loss not. The wired thing is when I switch eval mode to train mode for validating, the loss goes well. Is there any possible reason for this?</p>
How to select x number of images and it's labels from each class from an image dataset like cifar10?

username: Kalyan
cooked: <p>I would like to use only 100 images from each class from cifar10 to train my model. i.e. 100 images for every class in cifar10 which is (100 images)*(10 classes) = 1000 images.</p>

username: ptrblck
cooked: <p>You can access the underlying target tensors directly in the <code>CIFAR10</code> dataset via:</p>
<pre><code class="lang-python">dataset.targets
</code></pre>
<p>Using these targets you could compute the indices for 100 samples for each class.<br>
Once you have these indices, you could use a <code>Subset</code> with these indices to only draw these specified samples.</p>
Issue with multiple gpus

username: Griffintaur1
cooked: <p>I have an image, and I create four rotated versions of it using.</p>
<pre><code class="lang-auto">def rotate4dtensor(x):
    x90 = x.transpose(2, 3)
    x180 = x.flip(2)
    x270 = x.transpose(2, 3).flip(3)
    return [x,x90,x180,x270]
</code></pre>
<p>and now when during training I call the function in this manner</p>
<pre><code class="lang-auto">target = target.cuda()
        input = input.cuda()
        input = torch.autograd.Variable(input)
        target_var = torch.autograd.Variable(target)
        output = model(input)
        loss = criterion(output, target_var)
        
        
        
        #####vectorized version
        losses_rot = []
        #rotation loss for supervised
        sup_rot = rotate4dtensor(input); tgts = []
        sup_rot4 = torch.stack(sup_rot)
        out = model(sup_rot4,is_rotation=True)
</code></pre>
<p>It throws the following error when no of gpus 3,5,6 but works fine with 1,2,and 4 why?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/f/0f4bb73d170bad3750ba847454e62efd23e3d645.png" data-download-href="https://discuss.pytorch.org/uploads/default/0f4bb73d170bad3750ba847454e62efd23e3d645" title="image (2)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/f/0f4bb73d170bad3750ba847454e62efd23e3d645_2_690x200.png" alt="image (2)" data-base62-sha1="2bjnHMJemXfQvFWMc2ofeQLlxgp" width="690" height="200" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/f/0f4bb73d170bad3750ba847454e62efd23e3d645_2_690x200.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/f/0f4bb73d170bad3750ba847454e62efd23e3d645_2_1035x300.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/f/0f4bb73d170bad3750ba847454e62efd23e3d645_2_1380x400.png 2x" data-dominant-color="383943"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image (2)</span><span class="informations">3160×916 797 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>How are you specifying the GPU ids?<br>
In your current code snippet you are using <code>.cuda()</code>, which would push the data to the default device.<br>
If you haven’t masked the GPUs via <code>CUDA_VISIBLE_DEVICES</code>, it should use GPU0.</p>
<p>Also, <code>Variables</code> are deprecated since PyTorch <code>0.4</code>, so you can use tensors directly in newer versions.</p>
Better memory efficiency vs speed for sequence datasets?

username: aarrvv
cooked: <p>I have an ordered dataset (X, Y), where each data sample is a sliding window of X with length n. That is, the data used by the model is from X’ = [ X[0:n], X[1:n+1], … ]</p>
<p>Since both X and n are large I cannot fit X’ into memory. I tested an indexed  dataset that constructs X’ in realtime:</p>
<pre><code class="lang-auto">class CustomDataset(torch.utils.data.Dataset):
    def __init__(self, X, Y, n, **kwargs):
        super(CustomDataset, self).__init__(**kwargs)
        self.X = X
        self.Y = Y
        self.n = n
        self.len = len(X) - n

    def __getitem__(self, idx):        
        return self.X[idx: idx + self.n, :], ...

    def __len__(self):
        return self.len
</code></pre>
<p>While this does work, it is extremely slow. Any ideas how to improve this?</p>
<p>(A CPU only solution would be fine).</p>
Multiple parallel DataLoaders cause exception

username: Noam_Gat
cooked: <p>Hello,</p>
<p>I’m training one model on two different sources of data.<br>
My training loop is essentially</p>
<p>for …<br>
train one batch from source A<br>
train one batch from source B</p>
<p>Note that the code to train from the two sources isnt exactly the same, but the model is.</p>
<p>When I initialize both DataLoaders with num_workers, I get the following execption:</p>
<p>RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the ‘spawn’ start method</p>
<p>Using the spawn method causes other side effects, so I would prefer to stay in this mode.<br>
It there a way to get this to work?</p>

username: Noam_Gat
cooked: <p>The problem was different - one of the data loader uses cuda internally (because the data goes through a cleaning network before being returned). This was the problem. Is there a way to a void ‘spawn’ method in this case?</p>

username: SimonW
cooked: <p>You can’t use CUDA with fork. But you can set the ctx for just one data loader by specifying <code>multiprocessing_context='spawn'</code> when creating the data loader.</p>

username: Noam_Gat
cooked: <p>Thanks! Didn’t know about the per-loader context flag. Much cleaner solution.</p>
Mahalanobis distance between all nodes

username: razvanc92
cooked: <p>Hello, I’ve been trying to implement the Mahalanobis distance between multiple nodes. So far I’ve started with an input of size (batch, time_stamps, num_nodes, embeding_size) and I will like to have an output of size (batch_size, time_stamps, num_nodes, num_nodes). The computation is quite simple for each pair of nodes (x_i, x_j, where the batch and the time_stamp matches) I need to compute: (x_i - x_j)^T * M * (x_j - x_i). I managed to implement by looping, but it was too time consuming, I also implemented it by using torch.repeat and some reshapes but i’m running out of memory.</p>
<p>So far I have this:</p>
<pre><code class="lang-auto">            sub1 = x_perm.repeat([1, 1, num_nodes, 1]).reshape(batch_size, ts, num_nodes, num_nodes,
                                                               num_channels).transpose(
                3, 2) - \
                   x_perm.repeat([1, 1, num_nodes, 1]).reshape(batch_size, ts, num_nodes,
                                                               num_nodes, num_channels)
            sub2 = x_perm.repeat([1, 1, num_nodes, 1]).reshape(batch_size, ts, num_nodes,
                                                               num_nodes, num_channels) - \
                   x_perm.repeat([1, 1, num_nodes, 1]).reshape(batch_size, ts, num_nodes,
                                                               num_nodes, num_channels).transpose(
                       3, 2)

            sub1 = sub1.unsqueeze(4)
            sub2 = sub2.unsqueeze(4)

            score = torch.matmul(sub1, M])
            score = torch.matmul(score, sub2.transpose(5, 4))
            score = score.reshape(batch_size, ts, num_nodes, num_nodes)
</code></pre>
<p>Any suggestions are more than welcome.</p>

username: mortonjt
cooked: <p>I’m wondering if you could take advantage of the <a href="https://github.com/pytorch/pytorch/blob/master/torch/distributions/multivariate_normal.py#L23" rel="nofollow noopener">batch mahalanobius distance</a> that is utilized internally within the multivariate normal.</p>
<p>Without completely understanding the context of your code, I have a sneaky feeling that those repeats should be removed.</p>
Mask shapes for dice loss + cross entropy loss

username: Hwarang_Kim
cooked: <p>Hello, I am currently working on semantic segmentation.<br>
Originally, i used only cross entropy loss, so i made mask shape as [batch_size, height, width].<br>
But as i try to adapt dice loss too, i use this code to make mask shape as [batch_size, class_number(=3), height, width]</p>
<pre><code class="lang-auto">def target_shape_transform(target):

    tr_tar = target.cpu().numpy()
    tr_tar = (np.arange(3) == tr_tar[...,None])
    tr_tar = np.transpose(tr_tar,(0,3,1,2))
    
    return torch.from_numpy(tr_tar).cuda()

def calc_loss(pred, target, metrics, ce_weight=0.2):
    ce = nn.CrossEntropyLoss()
    ce_loss = ce(pred,target.long())
    target = target_shape_transform(target)
    dice = dice_loss(pred, target)
    loss = ce_loss * ce_weight + (1.0 - dice) * (1.0 - ce_weight)
    return loss
</code></pre>
<p>It worked anyway, but i cannot find any other reference how to use ce+dice loss.<br>
Can anybody tell me if this approach is right?</p>

username: suraj.pt
cooked: <p>Hello Hwarang,</p>
<p>It is correct to combine loss and backpropagate it this way. Your network parameters will be updated with respect to <code>ce_loss</code> and <code>dice</code>.</p>
LSTM hidden shape problem

username: Topsoil
cooked: <p>If i have <code>nn.LSTM</code> with hidden size of 400, one layer, and batch_first = True, then h_0 and c_0 inputs should be of shape (batch_size,1,400), or do i not understand something correcty? Since it throws me an error:</p>
<p>Expected hidden[0] size (1, 2, 400), got (2, 1, 400) (My batch_size is 2)</p>

username: vdw
cooked: <p>The shape of the hidden state is not affected by the value of <code>batch_first</code>. The shape is always:</p>
<pre><code>(num_layers * num_directions, batch_size, hidden_dim)</code></pre>
TypeError: batch must contain tensors, numbers, dicts or lists; found <class 'PIL.PngImagePlugin.PngImageFile'>

username: sofmonk
cooked: <p>Despite of doing</p>
<pre><code class="lang-auto">
transform = transforms.Compose([transforms.Resize(256),  transforms.ToTensor()])



voc_data = torchvision.datasets.VOCSegmentation('/home/as/Desktop/mi/datasets/', download = True, transform=transform)
data_loader = torch.utils.data.DataLoader(voc_data, batch_size)
</code></pre>
<p>Getting error"</p>
<pre><code class="lang-auto">Using downloaded and verified file: /home/as/Desktop/mi/datasets/VOCtrainval_11-May-2012.tar
Traceback (most recent call last):
  File "main.py", line 108, in &lt;module&gt;
    train(10)
  File "main.py", line 63, in train
    for batch_idx, image, segment in enumerate(data_loader):
  File "/home/as/vir_env/mine/local/lib/python2.7/site-packages/torch/utils/data/dataloader.py", line 560, in __next__
    batch = self.collate_fn([self.dataset[i] for i in indices])
  File "/home/as/vir_env/mine/local/lib/python2.7/site-packages/torch/utils/data/_utils/collate.py", line 68, in default_collate
    return [default_collate(samples) for samples in transposed]
  File "/home/as/vir_env/mine/local/lib/python2.7/site-packages/torch/utils/data/_utils/collate.py", line 70, in default_collate
    raise TypeError((error_msg_fmt.format(type(batch[0]))))
TypeError: batch must contain tensors, numbers, dicts or lists; found &lt;class 'PIL.PngImagePlugin.PngImageFile'&gt;
</code></pre>

username: Yuqi_Jiang
cooked: <p>Hello. I was facing the same problem. Do you have any idea about this now?</p>

username: WhyJS
cooked: <p>Hello.I am facing the same problem too…Is there any solution about this issue?</p>

username: cwyeth
cooked: <p>I had the same problem, the solution is to also set target_transform in the dataset. I believe it is the masks that are causing the error, and simply converting them to tensors fixed the error for me; haven’t trained a model yet so can’t say for sure that’s the right thing to do.</p>
Understanding memory consumption

username: Trinayan_Baruah
cooked: <p>Hi,</p>
<p>I am trying to a do a calculation of the size occupied by the feature maps in a DNN model. For things like ReLU we can use nn.Functional or define ReLU as a layer in the init function. My question is in the nn.Functional case would we be considering the ReLU as a layer and using the inputs/outputs from it to calculate feature map size. When I define it as a layer in init, it does make sense to do that. But I am not sure about that when using ReLU from nn.Functional considering its not really a layer that i have defined. Can someone please clarify?. Thanks</p>

username: googlebot
cooked: <p>Memory allocations are linked with operations on tensors, layers just wrap groups of these operations together. So, it is kinda hard to measure memory, as layer code may create intermediate tensors, some will be freed early, some kept for backpropagation. Check <a href="https://github.com/Stonesjtu/pytorch_memlab" rel="nofollow noopener">pytorch_memlab</a> if you need a detailed profile.</p>
<p>ReLU allocates new memory for output by default, and it is usually not released, as next layer (linear/conv) keeps it for gradient calculations.</p>
Why are custom autograd functions so slow? Is there any way to speed them up?

username: Sam_Lerman
cooked: <p>For example, the custom ReLU defined in <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html</a><br>
is much slower than <code>nn.ReLU</code>.</p>
<p>Why is that? Can it be remedied?</p>

username: albanD
cooked: <p>Hi,</p>
<p>It depends what you mean by “much slower”. It runs in python while the <code>nn.ReLU</code> version is implemented completely in c++ so it will be a bit faster.<br>
Do you have benchmark that show surprising results?</p>

username: googlebot
cooked: <p>they also interfere with multi-threaded backward, don’t they?</p>

username: albanD
cooked: <p>Well “multi-threaded backward” is hard to get right now anyways (very unlikely that you’re using it unless you’re trying very hard) <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"><br>
But they will acquire the GIL yes.</p>
How to convert a 3D tensor to a tenro of lists

username: mfcs
cooked: <p>Hi everyone!! <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
<p>Could you help me with something?<br>
I am trying to convert a 3D tensor that contains two layers in a tensor of lists or tuples by combining these two layers. Here is an example:</p>
<p>Initial 3D tensor with two layers</p>
<pre><code class="lang-auto">tensor([[[ 67.6958,  67.6958,  67.6958,  67.6958,  67.6958],
         [ 67.7049,  67.7049,  67.7049,  67.7049,  67.7049],
         [ 67.6958,  67.6958,  67.6958,  67.6958,  67.6958],
         [-66.9094, -66.9094, -66.9094, -66.9094, -66.9094],
         [-66.9094, -66.9094, -66.9094, -66.9094, -66.9094]],

        [[ 56.8013,  56.8013,  56.8013,  56.8013,  56.8013],
         [ 56.4468,  56.4468,  56.4468,  56.4468,  56.4468],
         [ 56.8013,  56.8013,  56.8013,  56.8013,  56.8013],
         [ 45.8811,  45.8811,  45.8811,  45.8811,  45.8811],
         [ 45.8811,  45.8811,  45.8811,  45.8811,  45.8811]]],
       dtype=torch.float64)
</code></pre>
<p>List of tuples</p>
<pre><code class="lang-auto">tensor([[67.6958, 56.8013],[67.6958, 56.8013],[-66.9094, 45.8811],...], dtype=torch.float64)
</code></pre>
<p>Note that it is as if I were placing one layer in front of the other and for each position of the element I would take both values and create a list of two items or a tuple with two values.</p>
<p>Best regards,</p>
<p>Matheus Santos.</p>

username: mariosasko
cooked: <p>Try this:</p>
<pre><code class="lang-python">output = torch.stack((input[0].flatten(), input[1].flatten()), dim=1)
</code></pre>

username: mfcs
cooked: <p>Hey! Thanks!!<br>
I will try this <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
How does one do the data transforms or collate_fn in GPU?

username: pinocchio
cooked: <p>I have to do some computations in GPU before passing my data to my models. Lets say for simplicity I compute the mean (1st moment) and std (2nd moment). How do I make sure this is done in GPU in during the dataset/dataloader processing?</p>
<p>(I am assuming that doing such computations on say, numpy, is worse than just doing it directly in Pytorch’s GPU code)</p>
<p>e.g. code I just made up:</p>
<pre><code class="lang-auto">def get_data():
    D_emb1 = 6
    D_emb2 = 7
    D_emb3 = 8
    a = np.random.rand(3, D_emb1, D_emb2, D_emb3)
    b = np.random.rand(5, D_emb1, D_emb2, D_emb3)
    c = np.random.rand(2, D_emb1, D_emb2, D_emb3)
    d = np.random.rand(1, D_emb1, D_emb2, D_emb3)
    return a, b, c, d

def collate_fn_gpu(batch):
    '''
    Collate function that maps data samples to Tensor then calls the ops on GPU.
    '''
    batch_size = len(batch)
    trailing_dims = batch[0].shape[1:]
    #new_batch = torch.zeros(batch_size,*trailing_dims)
    new_batch = torch.zeros(batch_size)
    for i in range(batch_size):
        new_batch[i] = torch.Tensor(batch[i]).to(device).std()
    return new_batch

def main():
    transform = None
    #transform, trainset, trainloader, testset, testloader = get_cifar10(collate_fn)
    ## generate fake data
    a,b,c,d = get_data(name)
    data = [a, b, c, d]
    ## create data set
    dataset = DataSet(data)
    ## create data laoder
    trainloader = torch.utils.data.DataLoader(dataset, batch_size=3, shuffle=True, num_workers=0, collate_fn=collate_fn_gpu)
    ##
    for data in trainloader:
        print(data)

if __name__ == '__main__':
    main()
</code></pre>

username: pinocchio
cooked: <p>btw, how do I make sure these ops are not trainable?</p>
Distributed data parallel behavior

username: Trinayan_Baruah
cooked: <p>Hi,</p>
<p>What is the behavior of using DistributedDataParallel without running the training dataset using the DistributedSampler?. Will it mean that the models are deployed on multiple GPUs , but they end up working on the same data?. I am sort of confused about the behavior. Would be good to have some clarification. Thanks</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/trinayan_baruah">@Trinayan_Baruah</a></p>
<p>Quote a recent discussion: <a href="https://discuss.pytorch.org/t/comparison-data-parallel-distributed-data-parallel/93271" class="inline-onebox">Comparison Data Parallel Distributed data parallel</a></p>
<p>Please also see <a href="https://pytorch.org/docs/stable/notes/ddp.html" rel="noopener nofollow ugc">this brief note</a> and <a href="https://arxiv.org/pdf/2006.15704.pdf" rel="noopener nofollow ugc">this full paper</a></p>
<blockquote>
<p>What is the behavior of using DistributedDataParallel without running the training dataset using the DistributedSampler? Will it mean that the models are deployed on multiple GPUs , but they end up working on the same data.</p>
</blockquote>
<p>Yep, if you don’t use DistributedSampler or manually shard input data for each process, they will be working on the same data. In this case, every DDP instance in each process will end up with the same gradient in every iteration. As a result local gradients and synchronized global gradients will be the same, making DDP useless.</p>

username: Trinayan_Baruah
cooked: <p>Ok . thank you. it makes sense</p>
Creating weighted-combo tensor

username: ilkarman
cooked: <p>At some point in my network I have 3 tensors (x1, x2, x3) that I would like to sum with a learned weighting scheme. Although trying to be explicit that my ‘host_weights’ vector of 3 weights requires a gradient it doesn’t seem to get updated at all:</p>
<pre><code class="lang-auto">    def __init__(self, **kwargs):
        ....                      
        self.register_parameter(name='host_weights', param=nn.Parameter(torch.randn(3, requires_grad=True)))
        

    def forward(self, x1, x2, x3):

        x1 = self.resnet_x1(x1)
        x2 = self.resnet_x2(x2)
        x3 = self.resnet_x3(x3)
        
        weights_scaled = F.softmax(self.host_weights, dim=0)
        print(self.host_weights)
        x = weights_scaled[0]*x1 + weights_scaled[1]*x2 + weights_scaled[2]*x3
        ...
</code></pre>
<p>However:</p>
<blockquote>
<p>Parameter containing:<br>
tensor([-0.8687, -0.4497, -0.9619], device=‘cuda:0’, requires_grad=True)<br>
Parameter containing:<br>
tensor([-0.8687, -0.4497, -0.9619], device=‘cuda:0’, requires_grad=True)<br>
Parameter containing:<br>
tensor([-0.8687, -0.4497, -0.9619], device=‘cuda:0’, requires_grad=True)<br>
Parameter containing:<br>
tensor([-0.8687, -0.4497, -0.9619], device=‘cuda:0’, requires_grad=True)</p>
</blockquote>
<p>I’m not sure why however, could the indexing be creating a break in autograd?</p>

username: mariosasko
cooked: <aside class="quote no-group" data-username="ilkarman" data-post="1" data-topic="94158">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ilkarman/48/1350_2.png" class="avatar"> ilkarman:</div>
<blockquote>
<p>could the indexing be creating a break in autograd?</p>
</blockquote>
</aside>
<p>It should’t break the computation graph. Applying softmax and then indexing the weights is a valid sequence of ops. Additional code would be helpful (the whole forward pass and training loop) to solve the issue.</p>

username: ilkarman
cooked: <p>Ah it was a silly error elsewhere where optimizer. zero_grad() was above optimizer. step() killing my gradient!</p>
Triplet loss become zero after first batch

username: Shekar_Neo
cooked: <p>i am using siamese network with triplet loss using resnet50,<br>
when i train the network after second iteration loss becoming ‘0’.</p>
<p>code:</p>
<pre><code class="lang-auto">res_model = resnet50(pretrained=True)
# Unfreeze model weights
for param in res_model.parameters():
    param.requires_grad = True

class SiameseNetwork(nn.Module):
    def __init__(self):
        super(SiameseNetwork, self).__init__()
        self.res = nn.Sequential(OrderedDict([
            ('res', res_model)
        ]))
            
            
        self.fc_layers = nn.Sequential(OrderedDict([
            ('fc1', nn.Linear(1000, 1000, bias=False)),
            ('fbn', nn.BatchNorm1d(1000)),
            ('frelu', nn.ReLU()),
            ('fc2', nn.Linear(1000, 512)),
        ]))
        
            
    def forward(self, inputs):
        x = self.res(inputs)
        print(x.shape)
        
        x = x.view(x.shape[0], 1000)
        x = self.fc_layers(x)
        print(x.shape)
        
        return x
       

class siaNet(nn.Module):
    def __init__(self):
        super(siaNet, self).__init__()
        self.anchor = nn.Sequential(OrderedDict([
            ('anchor', SiameseNetwork())
        ]))
        
        self.positive = nn.Sequential(OrderedDict([
            ('positive', SiameseNetwork())
        ]))
        
        self.negative = nn.Sequential(OrderedDict([
            ('negative', SiameseNetwork())
        ]))

    def forward(self,input1,input2,input3):
        output1 = self.anchor(input1)
        output2 = self.positive(input2)
        output3 = self.negative(input3)
        
        return output1,output2,output3

model = siaNet()

class TripletLoss(nn.Module):
    def __init__(self, margin=1.0):
        super(TripletLoss, self).__init__()
        self.margin = margin
        
    def calc_euclidean(self, x1, x2):
        return (x1 - x2).pow(2).sum(1)
    
    def forward(self, anchor: torch.Tensor, positive: torch.Tensor, negative: torch.Tensor) -&gt; torch.Tensor:
        distance_positive = self.calc_euclidean(anchor, positive)
        distance_negative = self.calc_euclidean(anchor, negative)
        losses = torch.relu(distance_positive - distance_negative + self.margin)

        return losses.mean()

anc, pos, neg = train_x  
batch_size = 8
for epoch in range(25):
     
    for i in range(0, len(anc), batch_size):
        anchor_img = torch.from_numpy(anc[i:i+2]).to(device).permute(0,3,1,2).float()
        positive_img = torch.from_numpy(pos[i:i+2]).to(device).permute(0,3,1,2).float()
        negative_img = torch.from_numpy(neg[i:i+2]).to(device).permute(0,3,1,2).float()
        
        model.train()
        optimizer.zero_grad()
        anchor_out,positive_out,negative_out  = model(anchor_img,positive_img,negative_img)
        loss = criterion(anchor_out, positive_out, negative_out)
        loss.backward()
        optimizer.step()
        
        print("Epoch number {}\n Current loss {}\n".format(epoch,loss.item()))
    
</code></pre>
Tracing Torchvision Models

username: a_d
cooked: <p>Hello All,<br>
I am trying to trace the <code>fasterrcnn-resnet50-fpn</code> and save it in a .pt file later to be read and used by libtorch. I went through the thread- <a href="https://discuss.pytorch.org/t/fasterrcnn-resnet50-jit-trace/62337/15">jit trace of fasterRCNN</a> and saw that the PR to add scriptability of fasterRCNN was merged<br>
I am trying trace it using the following code (pytorch 1.6.0)</p>
<pre><code class="lang-auto">import torch
import torchvision
from PIL import Image

model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True).eval()
example = tuple(list(torch.rand((3, 640, 480))))
traced_Script = torch.jit.trace(model, example_inputs=example)
traced_Script.save("/Models/fasterrcnn.pt")
</code></pre>
<p>and run into the following error -</p>
<pre><code class="lang-auto">TypeError: forward() takes from 2 to 3 positional arguments but 4 were given
</code></pre>
<p>The torchvision documentation says we are supposed to provide the input as list so I went ahead with that,<br>
Could anyone please tell what could be going wrong<br>
TIA</p>
How to manipulate image in a batch(after batch has been created)?

username: Vaibhav_Baswal
cooked: <p>I am trying to perform a custom alignment code on batch of images from the dataloader.</p>
<p>The custom alignment code that I have written takes one image as a base reference and align other images based on that. My alignment code runs on cv2 python package, hence I need image format not Tensors, as returned by dataloader during training to align before sending them into the network.<br>
Till now, I have been using a Static Image file to align images in a batch in the <strong>getitem</strong> implementation. I need to align the batch based on the very first image in the batch. It will help me to generalize better, I think.</p>
<p>Let me know If anyone can help me. I would be so greatful as I am facing this problem since very long and trying to figure out on my own.</p>

username: JuanFMontesinos
cooked: <p>You can write your own collate function. However it won’t get the advantages of multiprocessing.<br>
Thus, you can just preprocess it before feeding it into the network by reconverting tensors in numpy.</p>
Torch.mm - pytorch 0.3.1 - unable to perform (t.sp.FloatTensor, t.FloatTensor),

username: TheRealPieter
cooked: <p>I currently have two variables:<br>
v1 = Variable[torch.sparse.FloatTensor]<br>
v2 = Variable[torch.FloatTensor]</p>
<p>I am attempting to perform matrix multiplication on it through:<br>
torch.mm(v1, v2)<br>
but receiving the following error:</p>
<p>RuntimeError: Expected object of type Variable(torch.sparse.FloatTensor) but found type<br>
Variable[torch.FloatTensor] for argument <span class="hashtag">#1</span> ‘mat2’.</p>
<p>should this not be possible? I am currently using pytorch 0.3.1</p>

username: albanD
cooked: <p>Hi,</p>
<p>No I don’t think this is possible with such an old version of pytorch <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>

username: TheRealPieter
cooked: <p>I tried in pytorch 0.4.1 and it seems to work there. Just weird as I borrowed this code from a github repository based on a paper.</p>