akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
RuntimeError: The size of tensor a (32) must match the size of tensor b (23) at non-singleton dimension 1

Chamroukhi: <pre><code class="lang-auto">RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-53-62648c78c606&gt; in &lt;module&gt;
      6 val_loss , val_accuracy = [], []
      7 # Train and evaluate
----&gt; 8 model, hist = train_model(model, data_loaders, criterion, optimizer_ft, num_epochs=num_epochs)

&lt;ipython-input-43-7052c1a4d9e9&gt; in train_model(model, dataloaders, criterion, optimizer, num_epochs)
     51                 # statistics
     52                 running_loss += loss.item() * inputs.size(0)
---&gt; 53                 running_corrects += torch.sum(preds == labels.data)
     54 
     55             epoch_loss = running_loss / len(dataloaders[phase].dataset)

~/anaconda3/lib/python3.7/site-packages/torch/tensor.py in wrapped(*args, **kwargs)
     25             return handle_torch_function(wrapped, args, *args, **kwargs)
     26         try:
---&gt; 27             return f(*args, **kwargs)
     28         except TypeError:
     29             return NotImplemented

RuntimeError: The size of tensor a (32) must match the size of tensor b (23) at non-singleton dimension 1

</code></pre>

jmandivarapu1: <p>try printing the shapes for both of the tensor you are passing.</p>

ReemY: <p>How was it solved? i have the same issue.</p>
Does BCE and CrossEntropyLoss in pytorch already include activation function like Sigmoid and Softmax?

Peterwisu: <p>Hi, I am curious with classification loss in pyotrch, I have seen tutorial online some of them apply activation function like softmax and sigmoid for CrossEntropy and BCE loss repectively to convert an outpuy to probabilites but some do not. So does both loss function in pytorch already inlclude those activation function inside it ?? because when I check the pytorch Gtihub repository , the comment on Crossentropy function written like this    &gt;&gt;&gt;<br>
&gt;&gt;&gt; # Example of target with class probabilities<br>
&gt;&gt;&gt; input = torch.randn(3, 5, requires_grad=True)<br>
&gt;&gt;&gt; target = torch.randn(3, 5).softmax(dim=1)<br>
&gt;&gt;&gt; output = loss(input, target)<br>
&gt;&gt;&gt; output.backward()</p>
<p>Thank you</p>

srishti-git1110: <p>Hi,</p>
<p>For torch.nn.CrossEntropyLoss, from the docs:</p>
<blockquote>
<p>Note that this case is equivalent to the combination of <a href="https://pytorch.org/docs/stable/generated/torch.nn.LogSoftmax.html#torch.nn.LogSoftmax" rel="noopener nofollow ugc"><code>LogSoftmax</code></a> and <a href="https://pytorch.org/docs/stable/generated/torch.nn.NLLLoss.html#torch.nn.NLLLoss" rel="noopener nofollow ugc"><code>NLLLoss</code></a>.</p>
</blockquote>
<p>which means softmax isn’t explicitly needed and is inherently applied by PyTorch.</p>
<p>For <code>torch.nn.BCELoss</code>, a sigmoid layer is explicitly required.</p>
<p>Consider using <code>torch.nn.BCEWithLogitsLoss</code> that combines <code>torch.nn.BCELoss</code> and a sigmoid layer and is numerically more stable than using a plain Sigmoid followed by a BCELoss.</p>
Implementing negative sampling in PyTorch

Evan_Weissburg: <p>I have a dataset with the following columns: <code>book</code>, <code>char1</code>, <code>char2</code>, <code>span</code>.</p>
<p><code>book</code>, <code>char1</code>, and <code>char2</code> are integers, whereas <code>span</code> is a matrix Tensor of integers.</p>
<p>I would like to implement negative sampling so that, for each batch that I retrieve from my DataLoader that wraps the dataset, I also get a batch of negative samples.</p>
<hr>
<p>For each individual data row retrieved (there may be multiple rows retrieved per batch, of course), I would like to have <code>N</code> negative samples retrieved as well, so that a negative sample is a single row from <em>any</em> of the <code>span</code> matrices in my dataset.</p>
<p>Naively, this is how I would retrieve a single negative sample (just to illustrate):</p>
<pre><code>def getNegativeSamples(dataset, N):   
    ret = [] 
    for i in range(N):
         # Choose which row the data will be pulled from
         dataset_row = dataset[random(0, len(dataset))]
         span = dataset_row[3]
         # Choose which row of the span matrix we will use
         span_entry = span[random(0, len(span))]
         ret.append(span_entry)

    # ret has N samples in it
    return ret
</code></pre>
<p>How can I implement this cleanly in PyTorch?</p>

ruchitpatel: <p>I’m also looking for similar problem. Has anyone found a solution for this yet?</p>

aaronrmm: <p>I found this searching for something similar. I have since found this <a href="https://kevinmusgrave.github.io/pytorch-metric-learning/samplers/#hierarchicalsampler" rel="noopener nofollow ugc">implementation</a>, which I think is very similar to your requirements (where the span matrix would almost fit exactly into the “labels” argument of the constructor), or at least a good example.</p>
Loading from .npy files causes way too much ram usage

osman_bayram: <p>Hi i am using a <a href="https://www.kaggle.com/datasets/osmanf/camelyon-preprocessed" rel="noopener nofollow ugc">this dataset</a></p>
<p>with this dataset class and dataloader</p>
<pre><code class="lang-python">
class CamelyonDataset(torch.utils.data.Dataset):
    def __init__(self, subset='train', dataset_path=dataset_path, transform=None):
        self.subset = subset
        self.transform = transform

        x_filename = subset + "_img_data.npy"
        y_filename = subset + "_mask_data.npy"
        
        # load them from the downloaded folder
        self._x = np.load(os.path.join(dataset_path, x_filename), allow_pickle=True)
        self._y = np.load(os.path.join(dataset_path, y_filename), allow_pickle=True)

        # convert to torch tensors
        self._x = torch.from_numpy(self._x).float()
        self._y = torch.from_numpy(self._y).float()
        
        # change channel order to match pytorch
        self._x = self._x.permute(0, 3, 1, 2)

        # add channel dimension to y
        self._y = self._y.unsqueeze(1)

    def __len__(self):
        return len(self._x)

    def __getitem__(self, idx):
        x = self._x[idx]
        y = self._y[idx]

        if self.transform:
            x = self.transform(x)

        return x, y
        
train_loader = torch.utils.data.DataLoader(CamelyonDataset(subset='train'), batch_size=16, shuffle=True)
val_loader = torch.utils.data.DataLoader(CamelyonDataset(subset='val'), batch_size=16, shuffle=True)
</code></pre>
<p>And here is the training loop:</p>
<pre><code class="lang-python">model = UNet(n_channels=3, n_classes=1)

optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
criterion = nn.BCEWithLogitsLoss()

def train(model, train_loader, val_loader, optimizer, criterion, epochs=10):
    for epoch in range(epochs):
        model.train()
        for i, (x, y) in enumerate(train_loader):
            optimizer.zero_grad()
            y_pred = model(x)
            loss = criterion(y_pred, y)
            loss.backward()
            optimizer.step()
            print(f"Epoch {epoch} | Batch {i} | Loss {loss.item()}")

        model.eval()
        with torch.no_grad():
            for i, (x, y) in enumerate(val_loader):
                y_pred = model(x)
                loss = criterion(y_pred, y)
                print(f"Epoch {epoch} | Batch {i} | Loss {loss.item()}")

train(model, train_loader, val_loader, optimizer, criterion, epochs=1)

</code></pre>
<p>Even though the loaded <code>.npy</code> files is less than 2 gb, the training loop uses way too much RAM and I couldn’t make <a href="https://www.kaggle.com/code/osmanf/cpu-unet-training/notebook" rel="noopener nofollow ugc">this kaggle notebook</a> work in GPU (kaggle CPU sessions have 32 gb of RAM).</p>
<p>Is there a way for me to partially load or better manage the memory of this dataloader/dataset?</p>

jhans: <p>Have you tried decreasing the batchsize in the train- and val-Loader?</p>

osman_bayram: <p>Decreasing batch size to 8 worked, thank you.</p>
<p>The GPU instance has just enough RAM to train.</p>
Conv2d produces grid on image

georzabo: <p>Hello everyone,</p>
<p>I am training an SRGAN and after each output I see a visual grid on each image. The problem seems to be originating from the Conv2d and is highly related to the kernel size. the higher the kernel size, the thicker the grid is. This grid only appears when working with images that have more than 1 channels, if you use 1 channel grayscale images for example, the grid does not appear. It is as if internally the image gets tiled and retiled after each convolutional layer.</p>
<p>Here is a demonstration of the issue (35 kernel to make the grid easier to see):</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/3/b3259a7d422977ef7027ea27e2795c80bbf09fd1.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/b3259a7d422977ef7027ea27e2795c80bbf09fd1" title="Screenshot_5"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b3259a7d422977ef7027ea27e2795c80bbf09fd1_2_690x254.jpeg" alt="Screenshot_5" data-base62-sha1="pyO205z507tErII93RqQO7NuOKB" width="690" height="254" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b3259a7d422977ef7027ea27e2795c80bbf09fd1_2_690x254.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b3259a7d422977ef7027ea27e2795c80bbf09fd1_2_1035x381.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b3259a7d422977ef7027ea27e2795c80bbf09fd1_2_1380x508.jpeg 2x" data-dominant-color="535454"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot_5</span><span class="informations">1920×708 87.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The same SRGAN implementation in tensorflow produces none of the above issues. any thoughts?</p>
<p>Thanks in advance.</p>
Using small images with pretrained models?

deepcode: <p>is it possible that I resize the pretrained models to work with smaller images (40x40) instead of 256x256?  I want to try decreasing training time by first doing my experiments on smaller images (40x40). Is it even possible to take a pretrained network and change it to process 40x40 and then fine tune on top of that?  Are there any other issues with doing this?  And if its possible, how would would go about doing this? I read all the docs on pretrained models and didnt see a lot of information.</p>

smth: <p>all the pre-trained models require 224x224 images as input atleast.<br>
If you have smaller images, you might have to upsample to 224x224.</p>

andife: <p>Hello, I’ve same question. Is this a standard way? Or is there a specific model rep with for small images? Does anyone know publications dealing with the possible upscaling topic?</p>
Loading many small variable-length numpy array files

Messiah: <p>Hi,</p>
<p>I am loading many small .npy array files that each contain a variable-length sequence of elements of the shape <code>TxN</code>, where <code>T</code> is a particular sequence length and <code>N</code> is the feature size (the same for all arrays).</p>
<p>I am currently loading each .npy array file, converting it to a tensor, and then appending it to a python list. This is all happening in the <code>__init__</code> of a <code>Dataset</code> class.</p>
<p>Is there a more efficient way to be loading the data in my scenario? If my sequence lengths were constant, I could pre-allocate a tensor of the appropriate size and then do in-memory slicing on them directly from the loaded data. I have also heard about loading the data ‘lazily’ in the <code>__getitem__</code> in cases where the data may be too large to fit in memory, but would this make a difference in terms of the time it takes to load the data? Also, is my current approach amenable to multi-processing using num_workers?</p>

JuanFMontesinos: <p>Hi,<br>
You are facing a trade-off between RAM use vs speed.<br>
If you have enough ram to do so, your process is the fastest, as the arrays are already loaded and allocated there. As you mention, you could load the files dynamically in the getitem function and optimizer a bit everything but you aren’t gonna find a real performance improvement by spending time on this.</p>
<p>The only interesting thing you can do is pad the sequences and create a tensor allocated in the GPU (in case you ahve space), as that would save you some time.</p>
Error with my custom concat class with TorchScript

fermat97: <p>Hi I try to use <code>torch.jit.script</code> on my custom <code>Concat</code> class:</p>
<pre><code class="lang-auto">class Concat(nn.Module):
    def __init__(self, dimension=1):
        super(Concat, self).__init__()
        self.dim = dimension

    def forward(self, x):
        return torch.cat(x, dim=self.dim)
</code></pre>
<p>But I got the following error:</p>
<blockquote>
<p>RuntimeError:<br>
Arguments for call are not valid.<br>
The following variants are available:<br>
aten::cat(Tensor<span class="chcklst-box fa fa-square-o fa-fw"></span> tensors, int dim=0) → (Tensor):<br>
Expected a value of type ‘List[Tensor]’ for argument ‘tensors’ but instead found type ‘Tensor (inferred)’.<br>
Inferred the value for argument ‘tensors’ to be of type ‘Tensor’ because it was not annotated with an explicit type.</p>
<p>aten::cat.names(Tensor<span class="chcklst-box fa fa-square-o fa-fw"></span> tensors, str dim) → (Tensor):<br>
Expected a value of type ‘List[Tensor]’ for argument ‘tensors’ but instead found type ‘Tensor (inferred)’.<br>
Inferred the value for argument ‘tensors’ to be of type ‘Tensor’ because it was not annotated with an explicit type.</p>
<p>aten::cat.names_out(Tensor<span class="chcklst-box fa fa-square-o fa-fw"></span> tensors, str dim, *, Tensor(a!) out) → (Tensor(a!)):<br>
Expected a value of type ‘List[Tensor]’ for argument ‘tensors’ but instead found type ‘Tensor (inferred)’.<br>
Inferred the value for argument ‘tensors’ to be of type ‘Tensor’ because it was not annotated with an explicit type.</p>
<p>aten::cat.out(Tensor<span class="chcklst-box fa fa-square-o fa-fw"></span> tensors, int dim=0, *, Tensor(a!) out) → (Tensor(a!)):<br>
Expected a value of type ‘List[Tensor]’ for argument ‘tensors’ but instead found type ‘Tensor (inferred)’.<br>
Inferred the value for argument ‘tensors’ to be of type ‘Tensor’ because it was not annotated with an explicit type.</p>
</blockquote>
<p>I am using PyTorch version <code>1.7.1+cu92</code>, and Python <code>3.7.6</code>. Is it related to this <a href="https://github.com/pytorch/pytorch/issues/34294" rel="noopener nofollow ugc">bug</a>?</p>

ansley: <p>TorchScript is a statically-typed language, and, in most cases, we’re forced to infer untyped variables to be instances of <code>torch.Tensor</code>. (One of our projects this half is improving our type inference.) An easy fix is to simply annotate the relevant variables with their correct type.</p>
<pre><code class="lang-python">def forward(self, x: List[torch.Tensor]):
    return torch.cat(x, dim=self.dim)
</code></pre>

fermat97: <aside class="quote no-group" data-username="ansley" data-post="2" data-topic="112440">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/ba9def/48.png" class="avatar"> ansley:</div>
<blockquote>
<p><code>List[torch.Tensor]</code></p>
</blockquote>
</aside>
<p>Thank you it is fixed by your proposal.</p>

Tai_DOAN: <p>Thank you, it solved my problem too.</p>
Pytorch for python 3.10

lucB: <p>It seems that pythorch is currently not available for python 3.10. Could you please indicate a release date on the installation page ? I’ve upgraded my Linux system which erased python3.9. All my code is now just not working.</p>

ptrblck: <p>You could track the Python 3.10 support <a href="https://github.com/pytorch/pytorch/issues/66424">here</a> and in the meantime create a new virtual environment with Python&lt;=3.9.</p>

Steve54: <p>Thanks for sharing this info this is useful keep it up<a href="https://www.crediblebh.me/" rel="noopener nofollow ugc">.</a></p>

dinhanhx: <p>So I have read the issue, from PyTorch 1.12, I can use with Python 3.10 right?</p>
How to implement LSTM in pytorch with 3d input and 1d output

roger0904: <p>I’m trying to do sequence binary classification with LSTM in pytorch. The input data dimension is (3014, 48, 184) and the output shape is (3014,). The purpose is to do medical prediction, which means there are 3014 patients, each patient has 48 hours data, each hour contains 184 features.</p>
<pre><code class="lang-auto">device = torch.device("cuda")
lr = 0.001
n_epochs = 10
input_dim = 184    
hidden_dim = 184
layer_dim = 2
output_dim = 1
batch_size = 64
model = RNN(input_dim, hidden_dim, layer_dim, output_dim, batch_size)
model.to(device)
#print(model)
criterion = nn.BCEWithLogitsLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=lr, weight_decay=1e-5)

print('Start model training')

for epoch in range(1, n_epochs + 1):

    for i, (x_batch, y_batch) in enumerate(trainloader):
        x_batch = x_batch.to(device)
        y_batch = y_batch.to(device)
        optimizer.zero_grad()
        out = model(x_batch)
        loss = criterion(out.squeeze(1), y_batch)
        loss.backward()
        optimizer.step()
        print("Epoch {:2d} | lr {:.5f} | loss {:.5f} ".format(epoch, lr, loss))
</code></pre>
<pre><code class="lang-auto">def __init__(self, input_dim, hidden_dim, layer_dim, output_dim, batch_size):
    super().__init__()
    self.hidden_dim = hidden_dim
    self.layer_dim = layer_dim
    self.lstm = nn.LSTM(input_dim, hidden_dim, layer_dim,  batch_first=True)
    self.fc = nn.Linear(hidden_dim, output_dim)
    self.batch_size = batch_size
    self.hidden = None

def forward(self, x):
    #initializing the hidden states
    h0 = torch.zeros(self.layer_dim, x.size(0), self.hidden_dim).to(device)
    c0 = torch.zeros(self.layer_dim, x.size(0), self.hidden_dim).to(device)
    output, (hn, cn) = self.lstm(x, (h0, c0))
    output=self.fc(output[:,-1])
    return output```

The loss of every epochs is not decreasing and the loss function seems not working, I'm wondering that maybe I implement the model with wrong way. Hope to solve this problem ASAP. Thank you.

I'm expecting the loss will decrese, and the loss function will work properly. I was using keras to build the model before, and it works, I don't know how to bulid LSTM model in pytorch</code></pre>
Tensor target dimension torch.Size([50, 2]) is not valid. torch.Size([50, 2]) - Captum

a_d: <p>Hello all, I want to measure the feature importance in a multivariate time series. The model definition is as follows -</p>
<pre><code class="lang-auto">class Network(nn.Module):
    def __init__(self, input_size = 4, hidden_dimensions = 16, num_layers = 3, num_outputs = 2, seq_length = 3):
        super(Network, self).__init__()

        self.num_layers = num_layers
        self.hidden_size = hidden_dimensions
        self.seq_length = seq_length

        self.GRU = nn.GRU(input_size = input_size, hidden_size = hidden_dimensions, num_layers = num_layers,
                            bias = True, batch_first = True, bidirectional = True)
                            
        self.predictor = nn.Linear(in_features = 2 * hidden_dimensions * seq_length, out_features = num_outputs)


    def forward(self, input):
        batch_size = input.shape[0]

        h0 = torch.randn((2 * self.num_layers, batch_size,self.hidden_size))

        out, h_n = self.GRU(input, h0)

        out = out.reshape(batch_size, 2 * self.hidden_size * self.seq_length)

        out = self.predictor(out)

        return out
</code></pre>
<p>The time series has 3-time steps with 4 feature channels and predicts 2 values that have to regressed.<br>
Therefore, the output of the model is <code>batch, 2</code></p>
<p>I use Integrated Gradients to find the feature performance as follows -</p>
<pre><code class="lang-auto">integratedGradients = IntegratedGradients(model)
attributions_ig, delta = integratedGradients.attribute(aisData, target=Coordinate, return_convergence_delta=True)
</code></pre>
<p>I however run into the issue mentioned in the heading.<br>
The full trace is as follows -</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "main.py", line 257, in &lt;module&gt;
    attributions_ig, delta = integratedGradients.attribute(aisData, target=Coordinate, return_convergence_delta=True)
  File "/home/atharva/.local/lib/python3.8/site-packages/captum/log/__init__.py", line 35, in wrapper
    return func(*args, **kwargs)
  File "/home/atharva/.local/lib/python3.8/site-packages/captum/attr/_core/integrated_gradients.py", line 286, in attribute
    attributions = self._attribute(
  File "/home/atharva/.local/lib/python3.8/site-packages/captum/attr/_core/integrated_gradients.py", line 351, in _attribute
    grads = self.gradient_func(
  File "/home/atharva/.local/lib/python3.8/site-packages/captum/_utils/gradient.py", line 112, in compute_gradients
    outputs = _run_forward(forward_fn, inputs, target_ind, additional_forward_args)
  File "/home/atharva/.local/lib/python3.8/site-packages/captum/_utils/common.py", line 455, in _run_forward
    return _select_targets(output, target)
  File "/home/atharva/.local/lib/python3.8/site-packages/captum/_utils/common.py", line 474, in _select_targets
    raise AssertionError(
AssertionError: Tensor target dimension torch.Size([50, 2]) is not valid. torch.Size([50, 2])
</code></pre>
<p>How can I use Captum for this scenario ?</p>
<p>TIA</p>

YuCheng-Qi: <p><a class="mention" href="/u/a_d">@a_d</a> Is your problem solved?</p>
Max_unpooling3d_forward_kernel failed with error code 0

Riccardo_De_Feo: <p>After receiving a memory error I ran my training loop using CUDA_LAUNCH_BLOCKING=1 and now receive this rather esoteric error: <code>max_unpooling3d_forward_kernel failed with error code 0</code>. What does error code 0 stand for? Also, this happens after 3 successful forward and backward passes through the network.</p>
<p>As I changed a number of things about my training loop from the last working version, without touching the actual network, having an hint where to look for a bug would really help.</p>

albanD: <p>This is surprising indeed.<br>
Are you running out of memory? Does reducing the batch size help?</p>

Riccardo_De_Feo: <p>When I had low memory issues in the past I would get a much clearer “out of memory” error.</p>

albanD: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> any idea of what could be causing this?</p>

ptrblck: <p>What do you mean by “memory error”?<br>
Was it an out of memory error or some memory access violation?<br>
Could you post the complete stack trace and, if possible, a code snippet to reproduce this issue, so that we can debug it?</p>

vijay_srinivas_tida: <p>Hi,</p>
<p>How does the backpropagation of max unpooling layer will work?</p>

ptrblck: <p>The <code>nn.MaxUnpool2d</code> layer will direct the gradients to the inputs directly as seen here:</p>
<pre><code class="lang-python">pool = nn.MaxPool2d(kernel_size=2, return_indices=True)

x = torch.randn(1, 1, 4, 4, requires_grad=True)
print(x)
# tensor([[[[-0.0680, -0.7748, -0.1858,  0.4980],
#           [-1.9051, -1.8637, -0.3653,  1.4009],
#           [-0.5979,  0.5564,  1.4532,  0.9443],
#           [ 0.2864, -0.3723, -0.9621,  1.0871]]]], requires_grad=True)

act, idx = pool(x)
print(act)
# tensor([[[[-0.0680,  1.4009],
#           [ 0.5564,  1.4532]]]], grad_fn=&lt;MaxPool2DWithIndicesBackward0&gt;)
print(idx)
# tensor([[[[ 0,  7],
#           [ 9, 10]]]])

act.retain_grad()
unpool = nn.MaxUnpool2d(2)

out = unpool(act, idx)
print(out)
# tensor([[[[-0.0680,  0.0000,  0.0000,  0.0000],
#           [ 0.0000,  0.0000,  0.0000,  1.4009],
#           [ 0.0000,  0.5564,  1.4532,  0.0000],
#           [ 0.0000,  0.0000,  0.0000,  0.0000]]]],
#        grad_fn=&lt;MaxUnpool2DBackward0&gt;)

out.mean().backward()
print(act.grad)
# tensor([[[[0.0625, 0.0625],
#           [0.0625, 0.0625]]]])
print(x.grad)
# tensor([[[[0.0625, 0.0000, 0.0000, 0.0000],
#           [0.0000, 0.0000, 0.0000, 0.0625],
#           [0.0000, 0.0625, 0.0625, 0.0000],
#           [0.0000, 0.0000, 0.0000, 0.0000]]]])
</code></pre>

vijay_srinivas_tida: <p>Thank you very much.</p>
Torch multiprocessing Process and Queue

tsp_t: <p>I’m tring to use multiprocessing.Process, I’m looking into torch.multiprocessing to use torch. But torch.multiprocessing.Queue and torch.multiprocessing.Process don’t seem to be compatible with each other.</p>
<p>So I tried several methods and found some combinations that are compatible with each other.</p>
<p>Is there any documentation on how to use it correctly for this part?</p>
<pre><code class="lang-auto">import torch.multiprocessing as ctx

queue = ctx.get_context("spawn").Queue()
pipe = ctx.Pipe()
process = ctx.Process(target = ...)
process.start()
</code></pre>
Loss.backward() error while using with custom autograd function

HamidGadirov: <p>Hi, I have a custom <code>autograd.Function</code> for correlation layer (that implements cost volume construction on cuda) in addition to the neural net that contains <code>forward</code> with its loss components. When I call <code>loss.backward()</code> there is an error:</p>
<p><code>RuntimeError: function CorrelationFunctionBackward returned an incorrect number of gradients (expected 8, got 2)</code></p>
<p>which is, as I figured out, related to the <code>autograd.Function</code>. This is how it looks like:</p>
<pre><code class="lang-auto">
class CorrelationFunction(Function):

    @staticmethod
    def forward(ctx, input1, input2,
            pad_size=3, kernel_size=3, max_displacement=20, stride1=1, stride2=2, corr_multiply=1):
        # print("in CorrelationFunction forward")
        ctx.pad_size = pad_size
        ctx.kernel_size = kernel_size
        ctx.max_displacement = max_displacement
        ctx.stride1 = stride1
        ctx.stride2 = stride2
        ctx.corr_multiply = corr_multiply
        
        ctx.save_for_backward(input1, input2)

        with torch.cuda.device_of(input1):
            rbot1 = input1.new()
            rbot2 = input2.new()
            output = input1.new()

            correlation_cuda.forward(input1, input2, rbot1, rbot2, output, 
                ctx.pad_size, ctx.kernel_size, ctx.max_displacement, ctx.stride1, ctx.stride2, ctx.corr_multiply)

        return output

    @staticmethod
    def backward(ctx, grad_output):
        input1, input2 = ctx.saved_tensors

        with torch.cuda.device_of(input1):
            rbot1 = input1.new()
            rbot2 = input2.new()

            grad_input1 = input1.new()
            grad_input2 = input2.new()

            correlation_cuda.backward(input1, input2, rbot1, rbot2, grad_output, grad_input1, grad_input2,
                ctx.pad_size, ctx.kernel_size, ctx.max_displacement, ctx.stride1, ctx.stride2, ctx.corr_multiply)

        return grad_input1, grad_input2
</code></pre>
<p>What do I need to change in the <code>backward</code>? Before I made it work for the forward pass after reimplementing <code>CorrelationFunction</code> for pytorch 1.13. Thanks in advance!</p>

ptrblck: <p>Yes, you need to return the gradient in your <code>backward</code> for all inputs in your <code>forward</code>. If some arguments do not have valid grads, return <code>None</code> for these.</p>

HamidGadirov: <p>Thanks, the issue was solved after I added <code>None</code> in <code>backwards</code> return for all inputs that were network parameters.</p>
What is the difference between src_mask and src_key_padding_mask

Yoo: <p>Here is a simple example:</p>
<pre><code class="lang-auto">&gt; q = torch.randn(5, 1, 10) # source sequence length 5, batch size 1, embedding size 10
&gt; 
&gt; def src_mask(sz):
&gt;       mask = (torch.triu(torch.ones(sz, sz)) == 1).transpose(0, 1) # lower triangular 
&gt;       mask = mask.float().masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0))
&gt;       return mask
&gt; attn_w1 = nn.MultiheadAttention(10, 1, attn_mask=src_mask(5))[1]
&gt; 
&gt; src_key_padding_mask = torch.tensor([[0, 0, 0, 1, 1]]).bool()
&gt; attn_w2 = attn(q, q, q, key_padding_mask=src_key_padding_mask)[1]
</code></pre>
<p>These two mask arguments are confusing.</p>
<p>I understand attention mask requres size [batch, seq_len, seq_len] and the other [batch, seq_len] for padding mask,<br>
but can we use them interchangeably, as for padding mask, we can also design a mask to fit the attention map size [seq_len, seq_len]?</p>
<p>Also, is src_key_padding_mask only working on key matrix? In self-attention case, should we mask the other two, query and value, also?</p>
Function composition

keyboardAnt: <p>Where can I find more information about how Autograd works for function composition?</p>
<p>Given</p>
<pre><code class="lang-auto">$$g_u \circ f_v \circ \cdots \circ f_v = g_u \circ (f_v)^n$$
</code></pre>
<p>with the learned parameters $u$ and $v$. The derivative is</p>
<pre><code class="lang-auto">$$\frac{d g_u}{du} \circ \left( \frac{f_v}{dv} \right)^n ~.$$
</code></pre>
<p>I’m interested in whether the vector of weights $v$ is updated more than once or if anything else takes more time than derivating the similar expression $g_u \circ f_v$ that doesn’t introduce composition.</p>

soulitzer: <p>If you use a tensor more than once those operations involving the tensor will also be recorded to the graph, so all contributions will be properly accumulated. Not sure if that answers your question.</p>

keyboardAnt: <p>Hello <a class="mention" href="/u/soulitzer">@soulitzer</a>,</p>
<p>We appreciate your reply.</p>
<p>I’m not yet familiar with the internals of Autograd. Please correct me when I’m wrong. You say that PyTorch maintains the computational graph whenever we implement such a calculation. My question is: In the above case of $n$ compositions, does the graph contains $n$ copies of $f$?</p>
<p>Our motivation is to understand whether backpropagating through the expression $f_w^n$ (i.e., applying $n$ times the function $f_w$) is currently more expensive than through $f_w$ (i.e., a single forward pass)—although both should update the learned weights $w$ only once.<br>
If backpropagating the compositions takes more time (as we suspect), it means a bottleneck for our research, and therefore we want to develop a solution.</p>
<p>Thanks!</p>

soulitzer: <p>Yes if you just compose f n times in forward, then it would appear n times in the bw graph, but not sure you are able to avoid that no matter what you do?</p>

keyboardAnt: <p>Why? Can’t we implement backpropagation ourselves? (Namely, alter PyTorch to work with our computational graph—which is efficient for compositions.)<br>
Thanks!</p>

soulitzer: <p>Oh right, I guess what I mean is that no general solution exists. Though if you can solve the backward analytically for your particular graph that could be more efficient, e.g. you could use a custom autograd Function.</p>
Loaded model gives much worse accuracy than trianed model

Bled_Clement: <p>Hi all,</p>
<p>I am trying to test an image denoising model with the same validation set that was used in training but my results are much worse than in training. During training the validation dataset gave me <strong>37 dB</strong> and after loading the model, <strong>I get only 25 dB.</strong></p>
<p>I have used this testing script before and it usually works as expected. I do the usual stuff of setting the model to eval() and do: with no grad:</p>
<p>The only notable difference with this model is that I have implemented a custom batchnorm module from a  paper I read. The module is as follows:</p>
<pre><code class="lang-auto">class BFBatchNorm2d(nn.BatchNorm2d):
    def __init__(self, num_features, eps=1e-5, momentum=0.1, use_bias = False, affine=True):
        super(BFBatchNorm2d, self).__init__(num_features, eps, momentum)
        self.use_bias = use_bias
        # self.training = True
        # self.affine=False
    def forward(self, x):
        self._check_input_dim(x)
        y = x.transpose(0,1)
        return_shape = y.shape
        y = y.contiguous().view(x.size(1), -1)
        if self.use_bias:
            mu = y.mean(dim=1)
        sigma2 = y.var(dim=1)

        if self.training is not True:
            if self.use_bias:
                y = y - self.running_mean.view(-1, 1)
            y = y / ( self.running_var.view(-1, 1)**0.5 + self.eps)
        else:
            if self.track_running_stats is True:
                with torch.no_grad():
                    if self.use_bias:
                        self.running_mean = (1-self.momentum)*self.running_mean + self.momentum * mu
                    self.running_var = (1-self.momentum)*self.running_var + self.momentum * sigma2
            if self.use_bias:
                y = y - mu.view(-1,1)
            y = y / (sigma2.view(-1,1)**.5 + self.eps)

        if self.affine:
            y = self.weight.view(-1, 1) * y;
            if self.use_bias:
                y += self.bias.view(-1, 1)

        return y.view(return_shape).transpose(0,1)
</code></pre>
<p>I have checked and confirmed that the .ckpt file contains the weights, biases, runnning mean and running variance of every layer that employs this BN layer. It seems to me that the weights are being correctly loaded. I have also made sure data paths have not been mixed up and dataloaders are identical.</p>
<p>Another detail is that this model was trained in Lightning but I have tried loading the without any lightning modules, and defined the model as a normal python module, and my output PSNR is the same.</p>
<p>Any ideas would be appreciated because I have exhausted all possibilities.</p>
Torchrun and python -m

kimihailv: <p>How to launch multi-node training via torchrun if my script contains relative imports?<br>
Earlier, my script spawned workers by itself, so I could bypass torch.distributed/torchrun and launch training like this:</p>
<blockquote>
<p>python -m parent.train</p>
</blockquote>
<p>But now I want to use torchrun.</p>

wanchaol: <p>are you facing failures of torchrun when using relative imports? could you post a simple repro program and also attach the error paste</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/kimihailv">@kimihailv</a>,</p>
<aside class="quote no-group" data-username="kimihailv" data-post="1" data-topic="165360">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kimihailv/48/45534_2.png" class="avatar"> kimihailv:</div>
<blockquote>
<p>How to launch multi-node training via torchrun if my script contains relative imports?</p>
</blockquote>
</aside>
<p>Have you tried using absolute paths instead of relative paths? That’s general practice when running on HPCs.</p>
Dist.init_process_group hangs with more GPUs

Alymostafa: <p>I am trying to initialize the Data Parallelism communications on four GPUs as follows:</p>
<pre><code class="lang-auto">rank = 0
os.environ['MASTER_ADDR'] = '127.0.0.1'
os.environ['MASTER_PORT'] = str(34567 + local_rank)
dist.init_process_group("gloo", rank=0, world_size=4)
</code></pre>
<p>Using world_size = 4 hangs forever but changing it to 1 works. I tried to use NCCL instead of gloo but didn’t make a difference.</p>

wanchaol: <p>Hi, you need to spawn 4 processes, and call this on each process</p>
Evaluation doesn't make sense when using 8 gpus with 1 batch-size and Data Parallelism

robertsenps: <p>Hi everyone,</p>
<p>I’m going to evaluate a model that behave like this:</p>
<ul>
<li>Takes images as an input</li>
<li>Uses last frame data, so it should behave sequentially frame by frame. You can imagine it is like a video.</li>
<li>Using data parallelism (nn.parallel.dataparallelism). I have 8 gpus.</li>
<li>I use shuffle = False for the DataLoader</li>
</ul>
<p>So, I want to evaluate that model using batch_size = 1, because I will only have 1 gpu after deployment. I tried to evaluate by using “CUDA_VISIBLE_DEVICES=0” and got:</p>
<pre><code class="lang-auto">First Frame:
tensor([ 8.4655,  8.5176,  8.4897,  ..., 49.1040, 49.0972, 49.1562],
       device='cuda:0')

Second Frame:
tensor([ 8.2219,  7.8709,  8.1458,  ..., 50.7412, 50.6144, 50.0625],
       device='cuda:0')
</code></pre>
<p>but If I tried to evaluate the model with 8 gpus, it produces different output, especially in the second frame.</p>
<pre><code class="lang-auto">First frame:
tensor([ 8.4655,  8.5176,  8.4897,  ..., 49.1040, 49.0972, 49.1562],
       device='cuda:0')

Second frame:
tensor([ 8.6638,  8.3508,  8.6230,  ..., 50.7001, 50.6111, 50.0625],
       device='cuda:0')
</code></pre>
<p>eventhough that when I checked the nvidia-smi, the only gpu that is utilizing is only gpu 0.<br>
Furthermore, I tried to delete “model = nn.parallel.DataParallel(model)” this line, and both of the trials produce the same result.</p>
<p>Does anyone here know what happened here? And is there any better ways to train or evaluating the data sequentially?</p>

wanchaol: <p><a class="mention" href="/u/robertsenps">@robertsenps</a> I think for your use case it’s not a data parallelism pattern because you want to evaluate sequentially, so better not use data parallel wrapper</p>
Can not get all data from __getitem__

Nang_Nguyen: <p>Hi, I normalize the 100 data, randomlly spilit to train set and test set, but I seem like I only get 1 data from <strong>getitem</strong> function. Appreciate if anyone can give a hint. Here are my codes:</p>
<pre><code>def __getitem__(self, idx):
    if not isinstance(idx, int):
        idx = idx.item()
    else:
        for idx in (self.data):
            x = self.normalized_data[idx,:-1].astype('float32')
            y = self.normalized_data[idx, -1].astype('float32')
            sample = {"input": x, "label": y}
    return sample
</code></pre>
<p>class Data_Loaders():<br>
def <strong>init</strong>(self, batch_size):<br>
self.nav_dataset = Nav_Dataset()<br>
split = torch.utils.data.random_split(self.nav_dataset, [80, 20])<br>
self.train_loader = split[0]<br>
self.test_loader = split[1]</p>
<p>I got full list of train_loader and test_loader, but it does not show up here:</p>
<p>batch_size = 16<br>
data_loaders = Data_Loaders(batch_size)</p>
<p>for idx, sample in enumerate(data_loaders.train_loader):<br>
train_input, train_label = sample[“input”], sample[“label”]</p>
<p>Result:<br>
print(list(sample[“input”]))</p>
<p>[1.0, 0.8850779, 0.69589293, 0.89152443, 1.0, 0.5]</p>

nivek: <p>Based on your code snippet, it doesn’t seem like you are passing your <code>Dataset</code> into <code>torch.utils.data.DataLoader</code>? You should import <code>DataLoader</code> and pass <code>split[0]</code> into it (along with any additional parameters).</p>
Any efficient way to compute batched minimum spanning tree?

Zing: <p>Hi!</p>
<p>In my research I need to compute the minimum spanning trees (MST) from a batch of adjacency matrix of size B x N x N, where the N x N dimension refers to a matrix whose elements are distances, and B is the batch dimenion.  And I want to find a way to output a matrix of size B x N x N, where N x N is the MST computed from the given adjacency matrices.</p>
<p>The problem is to deal with the batch. For non-batched version I plan on calling scipy.sparse.csgraph.minimum_spanning_tree, but it doe not seem to support a batch dimension, and computing the mst sequentially would be too slow.</p>
<p>Thanks!</p>
<p>%% I have the feeling that it would be really nice if torch.sparse could support some sort of MST algorithm for gpu?</p>

albanD: <p>Hi,</p>
<p>I think the problem is that algorithms on graphs are quite tricky to implement on gpu.<br>
Moreover, they are not used a lot in the context of neural networks and I’m not aware of any implementation.</p>

Zing: <p>Hmmmmm,  I see, maybe I will have to implement this myself, do you have any suggestions about how I should implement it?</p>
<p>Maybe I will just implement a non-batched version, since my graphs are pretty small (about 20x20), the speed might still be standable,</p>

Zing: <p>The thing is I really want to avoid copying things back and forth between CPU and GPU, which I think defeats the purpose of doing batch learning on GPus</p>

SHRAMAN_PRAMANICK: <p>Hi Zing, Its very late to reply to this thread, but I wanted to know if you implemented MST with batched input? If so, it will be very interesting to learn your approach.</p>

leoduan: <p>I recently ran into the same problem, and here’s my solution combining Prim’s algorithm &amp; array operations. Perhaps this could be useful for someone.</p>
<p>The following function assumes the input log-weight matrix, logQ, having dimensions S x nV x nV, with S the batch size, nV the number of nodes,</p>
<pre><code>def computeMaxSpanningTrees(logQ):

    logQ -= logQ.max()
    
    Q = logQ.exp()

    nV = Q.shape[1]

    grid = torch.arange(0, (nV)*(nV)).reshape(nV,nV).triu(1)

    triuIdx = grid[grid&gt;0]

    seqS= torch.arange(0,S)

    def extractTriuAndFlatten(A):
        return A.reshape(S,nV*nV)[:, triuIdx]

    inTree = torch.zeros((S, nV, 1), dtype=torch.bool)
    inTree[:,nV-1] = True

    triuBool = (grid&gt;0)
    Q_flat = Q.reshape([S, nV*nV])[:,triuIdx]

    A_trees= torch.zeros(S,nV,nV)

    for step in range(nV-1):

        validEdgeIdx = (inTree != torch.transpose(inTree,1,2)) &amp; triuBool
        drawnEdgeIdx = triuIdx[ (Q_flat * extractTriuAndFlatten(validEdgeIdx)).argmax(1)]

        drawn_j = drawnEdgeIdx % nV
        drawn_i = (drawnEdgeIdx / nV).long()

        inTree[seqS,drawn_i]=True
        inTree[seqS,drawn_j]=True
        A_trees[seqS,drawn_i,drawn_j]= True
        A_trees[seqS,drawn_j,drawn_i]= True
    
    return A_trees
</code></pre>
Slightly quicker forward pass when looping through batch in forwad function

nassim: <p>Hello</p>
<p>I 'm having this weird thing in PyTorch when i pass my data in the forward function, i will have a slightly faster forward pass if i loop  through each element in the batch and pass it individually. My batch is of shape [8,18,500,500]  where 8 is the batch_size and it is passed on T4 GPU. is there any reason for this behavior?</p>
<p>Thanks</p>

ptrblck: <p>Could you post a minimal, executable code snippet showing this behavior please?</p>
Is it okay to call two loss functions in .backward()?

skarah00: <p>I have a simple model that i’m trying to obtain evaluation metrics (segmentation task, originally minimizing binary cross entropy loss). However, given the unbalanced nature of my data, i decided to implement a Focal Loss function. Given it is a loss, do i have to also backprop through these weights? I call loss.backward on my BCE, but i suppose i’m looking for some intuition as to why one would have two loss functions in their problem, if i should just focus on one and remove the other, or if there is a way to properly incorporate both. My code implementation is below, first the weighted focal loss function, then my training function.</p>
<pre><code class="lang-auto">def weighted_focal_loss(outputs, targets, alpha, gamma):
    alpha = torch.tensor([alpha, 1-alpha])
    gamma = gamma
    BCE_loss = F.binary_cross_entropy_with_logits(outputs, targets, reduction='none').squeeze(dim=1).flatten()
    targets = targets.type(torch.long)
    at = alpha.gather(0, targets.data.view(-1))
    pt = torch.exp(-BCE_loss).flatten()
    F_loss = at*(1-pt)**gamma * BCE_loss
    return F_loss.mean()

def train_val(model, loss, opt, scheduler, epochs):
    train_ious = []
    train_acc = []    
    train_loss = []
    train_fl = []
        
    for epoch in range(epochs):
        model.train()
        running_loss = 0
        running_acc = 0
        running_iou = 0
        running_fl =0
        for i, batch in tqdm(enumerate(train_dl), desc='training'):
            x = batch['img'].float()
            y = batch['fpt'].float().unsqueeze(dim=1)
            output = model(x)
            
            running_acc += pixel_accuracy(output, y)
            running_iou += mIoU(output,y)
            
            
            running_fl += weighted_focal_loss(output,y, 0.7, 3)
            
      
            loss_epoch = loss(output, y)
            running_loss += loss_epoch.item()
            
            opt.zero_grad()
            loss_epoch.backward(retain_graph=False)
            opt.step()
</code></pre>

mvalente: <p>You can add the losses and then do backpropagation.</p>
<pre><code class="lang-auto">loss1 = MyLoss1(out,y)
loss2 = MyLoss2(out,y)

loss = loss1 + loss2
loss.backward()
</code></pre>
<p>You can even add multipliers to each loss during the addition to give it more or less weight.</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/skarah00">@skarah00</a>,</p>
<p>When you call <code>.backward()</code> for 2 separate loss functions, Autograd will accumulate both loss gradients together. So your gradients will be <code>d loss1 / d param</code> + <code>d loss 2 / d param</code> for all parameters.</p>
Loss function on output slices

alemelis: <p>hello</p>
<p>I’m doing regression and my network output is a 14 values vector</p>
<p>by using MSE loss function the training converge but I’d like to use different criteria for the different values</p>
<p>as a starter, I tried something like this</p>
<pre><code class="lang-auto">lass MyLoss(nn.Module):
    def __init__(self):
        super().__init__()
        self.mse = nn.MSELoss()

    def forward(self, yhat, y):
        loss = 2*self.mse(yhat[:2], y[:2]) + 5*self.mse(yhat[2:5], y[2:5]) + self.mse(yhat[5:], y[5:])
        return loss
</code></pre>
<p>however this results in (I set batch size to 2 for testing purposes)</p>
<p><code>UserWarning: Using a target size (torch.Size([0, 14])) that is different to the input size (torch.Size([2, 14])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.</code></p>
<p>I reckon I’m not slicing the tensor correctly, or is this completely the wrong way to go about this?</p>
<p>The alternative I have in mind is to have multiple output layers with a single loss function each; is that a better idea?</p>

Eta_C: <pre><code class="lang-python">class MyLoss(nn.Module):
    def __init__(self):
        super().__init__()
        self.mse = nn.MSELoss()

    def forward(self, yhat, y):
        loss = 2*self.mse(yhat[:, :2], y[:, :2]) + 5*self.mse(yhat[:, 2:5], y[:, 2:5]) + self.mse(yhat[:, 5:], y[:, 5:])
        return loss
</code></pre>

keqiao_li: <p>I wonder if slicing the input of the loss function is ok to do that? This will not affect the autograd in PyTorch? In fact, I’m having the similar problem that I need to slice the input and feed it to a customized loss function.</p>
<p>Y = Model(input)      # Y is a batch_size by n by n matrix<br>
loss = loss_1(Y[:, :10, :10]) + loss_2(Y[:, 10:, 10:])</p>
<p>Thanks</p>
How important is saving the state dict of the optimizer?

Ioan: <p>I am training a language model, however due to time constraints I need to stop and restart training again.</p>
<p>I am aware of the right way to do it, with checkpointing.</p>
<p>How big is the impact of not saving the optimizer’s state? At the moment the model seems to go back to scratch when it learns. Is this due to the absence of the optimizer’s state loading or it might be a bug in the actual loading of the model?</p>
<pre><code class="lang-auto">initial_dict = copy.deepcopy(model.state_dict())
    if args.model_path is not None:
        print("--------USING PRETRAINED MODEL-----------", file=output)
        # strict = False because the model might have more or less heads than the pretrained one
        # normally we are only interested in the LM part with the cloze head
        model.load_state_dict(torch.load(args.model_path), strict=False)

    for k, v in model.state_dict().items():
        if k in initial_dict and not torch.equal(initial_dict[k], v):
            print(f"Updated {k} using pretrained model", file=output)
        else:
            print(f"Leaving {k} untouched", file=output)```</code></pre>

mvalente: <p>Optimizers and schedulers keep running statistics that can influence the resume. But it’s hard to say without having a full picture.</p>