akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to load models on multiple gpus and forward() it?

juhyung: <p>I load my 2 model on gpu1 and gpu2. current_device is set on gpu1</p>
<p>then I can forward model on gpu1 but cannot model on gpu2 with this error</p>
<pre><code class="lang-auto">RuntimeError: all tensors must be on devices[0]
</code></pre>
<p>After I change the current device to gpu1 with this line, the same error occurs</p>
<pre><code class="lang-auto">torch.cuda.set_device(1)
</code></pre>

mmisiur: <p>did you follow PyTorch <a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html#sphx-glr-beginner-blitz-data-parallel-tutorial-py" rel="nofollow noopener">Data Parallelism tutorial</a>?</p>

juhyung: <p><a class="mention" href="/u/mmisiur">@mmisiur</a> yes, but I think my use case is a little bit different.<br>
I want to load some models to multiple gpus respectively and run each model on its gpu.<br>
Thanks for reminding it.</p>

mmisiur: <p>Ok, right. So it’s hard to say what is wrong without your code. But if I understand what you want to do (load one model on one gpu, second model on second gpu, and pass some input through them) I think the proper way to do this, and one that works for me is:</p>
<pre><code class="lang-auto"># imports
import torch

# define models
m0 = torch.nn.Linear(10,5)
m1 = torch.nn.Linear(10,5)

# define devices
d0 = torch.device("cuda:0")
d1 = torch.device("cuda:1")

# define tensors
t0 = torch.rand(10)
t1 = torch.rand(10)

# move to devices
t0 = t0.to(d0)
t1 = t1.to(d1)
m0 = m0.to(d0)
m1 = m1.to(d1)

# forward pass
out0 = m0(t0)
out1 = m1(t1)
</code></pre>

juhyung: <p>Oh I just noticed that I didn’t move tensors…<br>
Thanks!!</p>
transforms.Normalize Runtime error

vipiet: <p>Hey Guys,</p>
<p>I want to normalize my Tensor.</p>
<p>When calling Normalize((0.5,0.5,0.5), (0.5,0.5,0.5)), I’m getting following error:</p>
<blockquote>
<p>tensor.sub_(mean[:, None, None]).div_(std[:, None, None])<br>
RuntimeError: expected device cpu but got device cuda:0</p>
</blockquote>
<p>How do I send normalize to a cuda device? Is this even possible?</p>
<p>Thanks for helping.</p>

ptrblck: <p>Which torchvision version are you using?<br>
<code>mean</code> and <code>std</code> will be created as tensors on the same device as the input as seen in <a href="https://github.com/pytorch/vision/blob/b6f28ec1a8c5fdb8d01cc61946e8f87dddcfa830/torchvision/transforms/functional.py#L210-L211">these lines of code</a> and the code works fine:</p>
<pre><code class="lang-python">x = torch.randn(3, 224, 224).cuda()
norm = transforms.Normalize((0.5,0.5,0.5), (0.5,0.5,0.5))
out = norm(x)
print(out.device)
&gt; cuda:0
</code></pre>

vipiet: <p>Yeah, found the problem yesterday had torchvision version 0.2, updated it and now it works. I forgot to post the solution, sorry.</p>
<p>Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Transform CenterCrop

hkitano: <p>For the method torchvision.transforms.CenterCrop, does it always crop from the center? For example, if we began with a 256x256 image and resized it to 224x224, would that mean we crop 16 pixels from top, bottom, left, and right?</p>

tom: <p>Yes. <a href="https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.CenterCrop" rel="nofollow noopener">CenterCrop</a> works this way.</p>
<p>Best regards</p>
<p>Thomas</p>
Dimensions in torch.topk()

CCL: <p>I have a (n, n) mask and corresponding (3, n, n) image. I’m trying to use the <code>torch.topk()</code> function to get the top k sigmoid pixel values and erase them from the corresponding image. However, I’m struggling quite a bit trying to understand how <code>topk()</code> deals with dimensions… To properly compute this, do I need to unsqueeze the matrix? Thanks!</p>

tom: <p>topk takes the top k over a single dimension. So if you want to take the top k over the two spatial dimensions, you need to .view(…) your tensor to combine them to one and then “unravel” the indices.</p>
<p>Best regards</p>
<p>Thomas</p>
Validation loss/error curve isn't as smooth as training loss curve

WeiQin_Chuah: <p>Hi all,</p>
<p>I am training a network that perform depth estimation. My training loss/epoch curve is decreasing smoothly but my validation loss and error / epoch curves although are decreasing but is like a roller coaster ride. I am just wondering if this phenomenon normal? I can’t really seem to find a good answer elsewhere to decided to ask here.</p>
<p>Thank you.</p>
<p>More info:<br>
Learning rate started with 0.001 and is decayed by 0.0003 after every 10 epochs.<br>
Total epochs: 30<br>
Batch Size: 16</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/c/bc3feaf6379e7462f76dcc619facb411cb9c9f8d.png" alt="image" data-base62-sha1="qRkHSyvWvQvQhh1gDoRmH6qek0d" width="572" height="416"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/6/a61e19ea1855c47c50d565798b798bd2d9f44b2c.png" alt="image" data-base62-sha1="nHxKBs4jTWJKvRuIQmB1IQgdDRy" width="606" height="411"></p>

ptrblck: <p>How large is your validation set?<br>
If you are calculating the validation loss on fewer samples, it might explain the noise in the loss curve.</p>

WeiQin_Chuah: <p>My validation set has a size of 1460 and my training set is 35454. So my validation set is about 4% of my training set. Is this size too small?</p>
<p>But intuitively validation is just to check if our model is converging right, so I can ignore the noise as long as it shows that my model is doing its job. Correct me on this if I am wrong.</p>
<p>Thank you</p>

ptrblck: <p>This might explain the noise, but you could double check it by calculating the mean and std using the loss for all samples in the validation set once.</p>
<aside class="quote no-group" data-username="WeiQin_Chuah" data-post="3" data-topic="72076">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/weiqin_chuah/48/13348_2.png" class="avatar"> WeiQin_Chuah:</div>
<blockquote>
<p>But intuitively validation is just to check if our model is converging right, so I can ignore the noise as long as it shows that my model is doing its job. Correct me on this if I am wrong.</p>
</blockquote>
</aside>
<p>Yes, that’s correct. The validation loss should give you a signal about your model performance on unseen data.</p>

WeiQin_Chuah: <p>What information can I get from calculating the mean and std?</p>

ptrblck: <p>The std would show the noise in each of your estimates.</p>

WeiQin_Chuah: <p>From the graphs, is it possible to tell if my model is overfitting?</p>

ptrblck: <p>Your validation loss seems to be way lower than your training loss.<br>
Also, it’s a bit hard to tell based on the noise, but the validation loss might still go down after 30 epochs.</p>
Indexing a heatmap from variabel's x,y location

aiyulu: <p>I want to do something about indexing with gradient, which I have review most of topic like <a href="https://discuss.pytorch.org/t/indexing-a-variable-with-a-variable/2111/4">Indexing a variable with a variable</a>:<br>
But my problem is a little different from them.For instance,</p>
<pre><code class="lang-auto">import torch
from torch.autograd import Variable
x = Variable(torch.randn(68, 2)) # x is an output from a network, which have 68 points location
heatmaps = Variable(torch.zeros(68, 64, 64), requires_grad=True)
for i in range(x.shape[0]):
    heatmaps[i, x[i, 1].int(), x[i, 0].int()] = 1
target_heatmaps = Variable(torch.randn(68, 64, 64), requires_grad=True) # which is some meaningful label
loss = torch.sum(target_heatmaps - heatmaps)
</code></pre>
<p>How can I get the gradients of x, it seams difficult for one reason the gradient of x[,].int() is zero, and another is heatmaps[i, x[i, 1].int(), x[i, 0].int()] = 1 has assert like leaf variable has been moved into the graph interior.</p>

albanD: <blockquote>
<p>heatmaps[i, x[i, 1].int(), x[i, 0].int()] = 1 has assert like leaf variable has been moved into the graph interior.</p>
</blockquote>
<p>This is just because you modify heatmaps inplace while you created it your self.<br>
You can use <code>new_heatmaps = heatmaps.clone()</code> and use <code>new_heatmaps</code> in the for-loop and your loss computation to avoid that.</p>
<blockquote>
<p>it seams difficult for one reason the gradient of x[,].int() is zero</p>
</blockquote>
<p>Gradients for integer values are not really possible as you don’t even have a continuous function. You will need to have a softer version of this.<br>
Like make your network output a mask of size 68, 64, 64 that goes through a softmax and use that as the target heatmap?</p>

aiyulu: <p>Thanks for your reply, I have realized my implementation is not work, even though I have get a gradient of x according topic like <a href="https://discuss.pytorch.org/t/indexing-a-variable-with-a-variable/2111/4">Indexing a variable with a variable</a>.<br>
Thanks for your patience again. <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:"></p>
Weights didn't update

Teng-Martin-Ma: <p>I am new to pytorch, not familar with the gradient graph. I just created 2-layers NN, however the weights did not update during training process. Please help me out.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

torch.set_default_dtype(torch.float64)

class Nets(nn.Module):
    def __init__(self):
        super().__init__()
        self.hidden1 = nn.Linear(26, 15)
        self.hidden2 = nn.Linear(15, 15)
        self.out = nn.Linear(15, 1)
        
    def forward(self, training_sets):
        trained_results = torch.tensor([], requires_grad=True)
        for ts in training_sets:
            x = self.hidden1(ts)
            x = torch.tanh(x)

            # flow to 2nd hidden layer
            x = self.hidden2(x)
            x = torch.tanh(x)

            # flow to output layer
            x = self.out(x)
            e = torch.sum(x)
            trained_results = torch.cat((trained_results, torch.tensor([e])))
        return trained_results
</code></pre>
<pre><code class="lang-auto">import torch.nn as nn
import numpy as np
from math import sqrt

net = Nets()
# Mean Square Error
loss_func = nn.MSELoss(reduction='mean')
learning_rate = 0.01
optimizer = torch.optim.Adam(net.parameters(), lr=learning_rate)

epochs = 1001
error = []
for i in range(epochs):
    loss = loss_func(net(inputs), target_results)
    if i % 100 == 0:
        print(i, loss, net(inputs))
        error.append(loss)       
    optimizer.zero_grad()
    loss.backward(retain_graph=True)
    optimizer.step()
</code></pre>
<p>results outputs:<br>
0 tensor(5629.6088, grad_fn=) tensor([-17.9612, -17.9524, -17.9473], grad_fn=)<br>
100 tensor(5629.6088, grad_fn=) tensor([-17.9612, -17.9524, -17.9473], grad_fn=)<br>
200 tensor(5629.6088, grad_fn=) tensor([-17.9612, -17.9524, -17.9473], grad_fn=)<br>
300 tensor(5629.6088, grad_fn=) tensor([-17.9612, -17.9524, -17.9473], grad_fn=)<br>
400 tensor(5629.6088, grad_fn=) tensor([-17.9612, -17.9524, -17.9473], grad_fn=)<br>
500 tensor(5629.6088, grad_fn=) tensor([-17.9612, -17.9524, -17.9473], grad_fn=)<br>
600 tensor(5629.6088, grad_fn=) tensor([-17.9612, -17.9524, -17.9473], grad_fn=)<br>
700 tensor(5629.6088, grad_fn=) tensor([-17.9612, -17.9524, -17.9473], grad_fn=)<br>
800 tensor(5629.6088, grad_fn=) tensor([-17.9612, -17.9524, -17.9473], grad_fn=)<br>
900 tensor(5629.6088, grad_fn=) tensor([-17.9612, -17.9524, -17.9473], grad_fn=)<br>
1000 tensor(5629.6088, grad_fn=) tensor([-17.9612, -17.9524, -17.9473], grad_fn=)</p>

ptrblck: <p>Recreating a tensor will break the computation graph in <code>torch.tensor([e])</code>.<br>
Try to either concatenate this tensor directly, or alternatively append <code>e</code> in a list and call <code>torch.stack</code> on it afterwards.</p>
Len() for Iterabledataset

ykukkim: <p>Hey all,</p>
<p>I am using IterableDataset as my data’s length are not uniform and it varies a lot. Now I am trying to implement a validation step in my training loop which requires the length of the dataset, but as it iterable, len() cannot be used.</p>
<blockquote>
<pre><code>def val(self, epoch):
</code></pre>
</blockquote>
<pre><code>    test_loss = 0
    correct = 0

    with torch.no_grad():
        self.model.eval()
        for data, target in self.val_loader:
            data, target = data.to(self.device), target.to(self.device)
            predictions = self.model(data.float())
            pdb.set_trace()
            correct += (prediction == target).sum().item()
            total += target.size(0)
            val_loss += self.criterion(predictions.float(), target.float()).item()

        test_loss /= len(self.test_loader.dataset)
        accuracy = 100. * correct / len(self.test_loader.dataset)

        print('\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(
            test_loss, correct, len(self.test_loader.dataset), accuracy))

        with val_summary_writer.as_default():
            summary.scalar('val_loss', val_loss.item(), step=self.globaliter)
            summary.scalar('accuracy', accuracy, step=self.globaliter)
</code></pre>
<p>This is my validation function.</p>
<p>Is there a way to get len() for iterable dataset?</p>
<p>Thanks!</p>

ptrblck: <p>I think your approach of storing the number of samples in <code>total</code> is valid for your use case.<br>
Instead of dividing by <code>len(self.test_loader.dataset)</code>, just divide by <code>total</code>.</p>
Why is it called WeightedRandomSampler if shuffle is False

alx: <p>Hi all,</p>
<p>For a project I’m working on I need to load my images randomly as well as weighted. In other words I need batch 1 must be composed of various classes.</p>
<p>Is there a way to do this?</p>

ptrblck: <p>The <code>WeightedRandomSampler</code> will use its weights to randomly sample from your dataset, so shuffling won’t do anything.<br>
Also, have a look at <a href="https://discuss.pytorch.org/t/weightedrandomsampler-sampler-not-working-properly/69063/6">this topic</a> for more discussion.</p>
How to change the dilation of a convolutional layer in training phase

dbfc13f1f0d08fda0026: <p>I would like to change the dilation during training mode. In other words, I’d like to share weights between two conv layers with different dilation. Take following sudo code for example:</p>
<pre><code>init:
    conv1 = 3x3 kernel with dilation (1,1)
    conv1_dilation = conv1 with dilation (2,2)
forward(X1):
    R1 = conv1(X1)
    R2 = conv1_dilat(X1)
    return R1,R2
</code></pre>
<p>where conv1 and conv1_dilat have the same kernel size and value but different dilation.<br>
Could anyone tell me how to implement this?</p>

nutszebra: <p>This is the code:</p>
<pre><code class="lang-auto">import torch
from torch.nn import Conv2d
import torch.nn.functional as F

conv = Conv2d(3,10,3,1,1)
x = torch.Tensor(torch.rand(1,3,10,10))
R1 = conv(x)
R2 = F.conv2d(x, weight=conv.weight, bias=conv.bias, stride=conv.stride, padding=conv.padding, dilation=(2, 2), groups=conv.groups)
</code></pre>

Quang_Tran_Ngoc: <p>Hi all!</p>
<p>I have one question. Is there any function to replace dilation rate in conv2d ?</p>

ptrblck: <p>You could change it directly by assigning a new value to it:</p>
<pre><code class="lang-python">conv = nn.Conv2d(1, 6, 3, 1, 1, 1)
x = torch.randn(1, 1, 24, 24)
out = conv(x)
print(out.shape)

conv.dilation = (2, 2)
out = conv(x)
print(out.shape)
</code></pre>
Plotting accuracy scores and losses in ResNet

gss: <p>I am trying to plot the graph for validation/training accuracy and validation/training loss for a ResNet model.</p>
<p>Here is how I am training it:</p>
<pre><code class="lang-auto">LR = 0.01
N_EPOCHS = 50
BATCH_SIZE = 128

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")


resnet50 = models.resnet50(pretrained=True)

# Change the last layer
num_ftrs = resnet50.fc.in_features
resnet50.fc = nn.Linear(num_ftrs, 9)

model = resnet50.to(device)
optimizer = torch.optim.SGD(model.parameters(), lr=LR)
criterion = nn.CrossEntropyLoss()

print("Starting training loop...")
for epoch in range(N_EPOCHS):
    model.train()
    loss_train = 0
    for iter, traindata in enumerate(train_loader):
        train_inputs, train_labels = traindata
        train_inputs, train_labels = train_inputs.to(device), train_labels.to(device)
        optimizer.zero_grad()
        logits = model(train_inputs)
        loss = criterion(logits, train_labels)
        loss.backward()
        optimizer.step()
        loss_train += loss.item()
        #torch.save(model.state_dict(), "project.pt")
        print('Batch {} and Loss {:.5f}'.format(iter,loss_train/BATCH_SIZE))
    #model.load_state_dict(torch.load("project.pt"))
    model.eval()

    with torch.no_grad():
        for iter, valdata in enumerate(valid_loader, 0):
            val_inputs, val_labels = valdata
            val_inputs1, val_labels1 = val_inputs.to(device), val_labels.to(device)
            y_test_pred = model(val_inputs1)
            tar_=val_labels.cpu().numpy()
            loss = criterion(y_test_pred, val_labels1)
            print('Validation Loss {:.5f}'.format(loss.item()))

    print("Epoch {} | Train Loss {:.5f}".format( epoch, loss_train/BATCH_SIZE))
</code></pre>
<p>Any idea on how I could plot both graphs (accuracies and losses)? Thanks</p>

ptrblck: <p>You could create a list for both statistics and append them for each iteration or epoch.<br>
Also, make sure to detach the tensors before storing them in the list (<code>tensor.detach()</code> or <code>tensor,item()</code>). Otherwise you could store the whole computation graph, which will increase the memory usage.</p>
How model is loaded on gpu?

juhyung: <p>I’m testing loading same model in gpu like this.</p>
<pre><code class="lang-auto">import model
import numpy as np

tt = np.zeros((512, 512, 3), dtype=np.uint8)
ms = {}
while True:
	for i in range(300):
		if not ms.get(i):
			ms.update({i:model.Model()})
			for k,v in ms.items():
				v.infer(tt)
			print(i, 'loaded')
			print(len(ms), 'length')
</code></pre>
<p>When I load just 1 model, it occupies 1G of GPU mem.<br>
But when i load 160 model like above it occupies just 16G of gpu mem. why usage of gpu mem is not increasing linearly?</p>

ptrblck: <p>The first CUDA call will initialize the CUDA context, which will use some memory on your device (depending on the CUDA version, used GPU etc.).<br>
You could check the memory usage in PyTorch using <code>print(torch.cuda.memory_allocated())</code>.</p>
Is there anyone knows the meaning of this regular expression?

JachinMa: <p>I found this code in PyTorch tutorial,but I can’t search out the exact meaning of it.So I came here to get some help.</p>
<p>s = re.sub(r"([.!?])",r"\1",s)<br>
s = re.sub(r"[^a-zA-Z.!?]+",r" ",s)</p>
<p>Very thx to who would like to answer my question.</p>

ptrblck: <p>The first expression should replace the punctuations with a whitespace and the first matching group.<br>
Note that you’ve forgotten to paste the whitespace in the replacement expression:</p>
<pre><code class="lang-python">re.sub(r"([.!?])", r" \1", s)
</code></pre>
<p>Example:</p>
<pre><code class="lang-python">s = 'This is a sentence.'
print(re.sub(r"([.!?])", r" \1", s))
&gt; 'This is a sentence .'
</code></pre>
<p>The second regular expression matches everything except letters and punctuation and replaces it with a single whitespace. This is often used to get rid of multiple whitespace characters in the text.<br>
Example:</p>
<pre><code class="lang-python">s = 'This  is  a  sentence  with         whitespaces.'
print(re.sub(r"[^a-zA-Z.!?]+", r" ", s))
&gt; 'This is a sentence with whitespaces.'
</code></pre>

JachinMa: <p>thanks a lot<br>
sorry for replying so late<br>
cause I almost forget this question…<br>
I’m very grateful for your kind help</p>
Convert Pytorch Model to Keras

Sana: <p>I am trying to convert the following Pytorch Model to Keras.</p>
<p>class Net(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Net, self).<strong>init</strong>()<br>
self.fc1 = nn.Linear(28 * 28, 128)<br>
self.fc2 = nn.Linear(128, 64)<br>
self.fc3 = nn.Linear(64, 10)</p>
<pre><code>def forward(self, x):
    x = x.view(-1, 28 * 28)
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = self.fc3(x)
    return x
</code></pre>
<p>input_shape is  (28,28,1). Since Linear layer in Pytorch is equivalent to Dense in Keras, I tried the following:</p>
<p>model = Sequential()<br>
model.add(Dense(128, input_shape=(28,28,1)))<br>
model.add(Activation(‘relu’))<br>
model.add(Dropout(0.2))<br>
model.add(Flatten())<br>
model.add(Dense(64, input_shape=(28,28,128)))<br>
model.add(Activation(‘relu’))<br>
model.add(Dropout(0.2))<br>
model.add(Dense(10,input_shape=(28,28,64)))</p>
<p>Is this conversion correct?</p>
<p>Many thanks for your help!</p>
Difference between set_grad_enabled(False) and no_grad()

tlaurent: <p>Just a quick question: the following are exactly equivalent, correct?</p>
<p>with torch.no_grad():<br>
block of code</p>
<p>with torch.set_grad_enabled(False):<br>
block of code</p>

klory: <p>It seems there is no difference: <a href="https://stackoverflow.com/a/53447634/6888630" rel="nofollow noopener">https://stackoverflow.com/a/53447634/6888630</a></p>

albanD: <p>There is no difference.<br>
Just that one takes a bool as input and not the other.<br>
Also <code> torch.set_grad_enabled(False)</code> can be used as just a function to set the grad mode forever.</p>

tlaurent: <p>Great, thanks a lot. Very convenient to use the same piece of code for training and evaluating.</p>
Building my own environment for DQN to learn from, how to input state variables to input layer

E_OArwa: <p>Hello. I am new to Pytorch tensor manipulations. I have created an energy management environment for my DQN, which is a simple network like this with the forward method:</p>
<pre><code>`class DQN(nn.Module):
   def __init__(self, n_actions):
    self.n_actions = len([i for i in range (10, 100, 5)])
    super().__init__()
        
    self.fc1 = nn.Linear(in_features=5, out_features=24)   
    self.fc2 = nn.Linear(in_features=24, out_features=32)
    self.out = nn.Linear(in_features=32, out_features= n_actions)

def forward(self, t):
    t = t.unsqueeze_(5)
    t = t.flatten(start_dim=1)
    t = F.relu(self.fc1(t))
    t = F.relu(self.fc2(t))
    t = self.out(t)
    return t`
</code></pre>
<p>Now, each state is a list/vector of 5 numbers (floats), the actions are integer numbers in the range(10, 100, 5). How do I feed these states without getting mismatch errors?</p>
Column-wise operation and gradient

Hippolyte_Dubois: <p>Hi,<br>
I’m trying to manipulate a matrix by multiplying each column by a matrix.<br>
My code looks like this:</p>
<pre><code>for i in range(s.shape[0]):
	s[i,:] = torch.matmul(s[i,:],M)
return s
</code></pre>
<p><code>M</code> and <code>s</code> are <code>m*m</code> Tensors.</p>
<p>When I try to backward, I get this error:</p>
<blockquote>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1, 81]], which is output 0 of UnsqueezeBackward0, is at version 81; expected version 80 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</p>
</blockquote>
<p>and</p>
<blockquote>
<p>…\torch\csrc\autograd\python_anomaly_mode.cpp:57: UserWarning: Traceback of forward call that caused the error:</p>
</blockquote>
<p>point to this block of code.</p>
<p>What am i doing wrong, and how can I modify this ?</p>

tom: <p>Two possible ways:</p>
<ol>
<li>Use batch matrix multiplication (or einsum or whatever).</li>
<li>Instead of overwriting <code>s[i]</code>, collect the results in a list and do <code>torch.stack</code> on it after the for loop.</li>
</ol>
<p>Best regards</p>
<p>Thomas</p>

Hippolyte_Dubois: <p>What would the einsum look like ? I’m struggling a bit understanding the syntax</p>

tom: <p>What are your input shapes?</p>
<p>Best regards</p>
<p>Thomas</p>
Two separate gradient computations vs one total

sigma_x: <p>There are two loss functions: <code>L</code> and <code>M</code>, the total loss is <code>T=L+M</code>. Is there any difference between computing</p>
<pre><code class="lang-auto">L.backward()
M.backward()
optimizer.step()
</code></pre>
<p>and</p>
<pre><code class="lang-auto">T=L+M
T.backward()
optimizer.step()
</code></pre>

albanD: <p>The main difference will be that the second one will be faster to get the same result <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
torch.autograd.Variable seems to be deprecated but still included in the lib. Are most deprecated things going to be removed at some point?

kirk86: <p>I was wondering if things that show up in pytorch as deprecated or duplicates are going to be removed at some point in order to have a cleaner and more consistent interface?</p>

albanD: <p>Hi,</p>
<p>Yes they are. But because Variable what so widely used by the users, we are leaving it for now.<br>
It is only a no-op that gives you a Tensor right now.</p>
<p>But for deprecation:</p>
<ul>
<li>We just removed old style autograd.Function code: they will be actual errors in 1.5</li>
<li>We are removing .data internally to prepare for adding warning and deleting it (or making it a no-op like Variable is right now)</li>
</ul>
How to compute gradient of sample of Categorical distribution?

Tessa_van_der_Heiden: <p>Hi!</p>
<p>For the Normal distribution, one can use rsample() allowing computation of the gradient through a sample. This does not exist for a Categorical distribution.</p>
<p>How can we do the same?</p>
<p>Example:</p>
<pre><code class="lang-auto">mu, sigma = neural_net(input)
sample = Normal(mu, sigma).rsample()
</code></pre>
<p>I would like to do something like:</p>
<pre><code class="lang-auto">probs = neural_net(input)
sample = Categorical(probs).rsample()
</code></pre>

tom: <p>The problem is that the samples from the categorical distribution are discrete, so there is no gradient to compute. This is in contrast to the Gaussian where you can write <code>X = Z * sigma + mu</code> with <code>Z ~ N(0,1)</code> to get a <code>N(mu, sigma)</code>-distributed variable  (the <em>reparametrization trick</em> in some circles).<br>
A common way around this is to <em>not</em> sample, but compute the loss for all categories, weighted with their respective probabilities. This then allows to compute gradients w.r.t. the probabilities.</p>
<p>Best regards</p>
<p>Thomas</p>

Tessa_van_der_Heiden: <p>Hi Thomas!</p>
<p>That was clear, thanks! The solution is to use the Gumbel-Softmax and to make sure the gradients remain for the category probabilities, I did the following (dot product mm):</p>
<pre><code class="lang-auto">z, probs = g_s_net(input, temp)
sample = torch.mm(z, torch.FloatTensor(some_categorical_list))
loss = probs * z
</code></pre>
<p>Here z is the categorical one hot tensor and multiplying it with the probs will give the probs for the corresponding class.</p>
Understanding and applying 3dconvs

alxwo: <p>Summary:</p>
<pre><code class="lang-auto">t1 = torch.zeros(8, 256, 64, 64)
t2 = torch.zeros(8, 256, 64, 64)
catted = torch.cat((t1, t2), dim=1)  # shape = (8, 512, 64, 64)
print(catted.shape)
conv2d = nn.Conv2d(512, 256, kernel_size=3, padding=1)  
result_conv2d = conv2d(catted)  # shape = (8, 256, 64, 64)
print(result_conv2d.shape)

stacked = torch.stack((t1, t2), dim=2)  # shape = (8, 256, 2, 64, 64)
print(stacked.shape)
conv3d = nn.Conv3d(256, 256, kernel_size=(2,3,3), padding=(0,1,1))  
result_conv3d = conv3d(stacked)  # shape = (8, 256, 64, 64)
print(result_conv3d.squeeze().shape)
</code></pre>
<p>Would in this case conv3d be doing the same as conv2d?</p>
<p>Background:<br>
I’m currently experimenting and playing around with fusion of data in conv nets, and was thinking that it might be sensible to include 3d convolutions at some places. However after thinking some more about it, I began to question how exactly the 3d convolution would even work…<br>
Lets say I have 2 different, but somewhat related images as input to 2 separate feature extraction branches, 1 image per feature extraction branch.<br>
Now after some convolutions, I end up with 2 tensors of sizes (N, C, H, W).<br>
Standard procedure would now be, to concatenate both tensors along the channel dimension C, so we get 1 tensor of size (N, 2C, H, W) and then apply a 2d convolution.<br>
For this example lets assume the 2d convolution has kernel_size=3 and padding=1 such that the input and output shape stays the same (except for the channels)<br>
However maybe it might be sensible to make use of a 3d convolution, as in theory, a 3d convolution should extract some kind of information about how the 2 inputs correlate in the depth dimension.<br>
So regarding the implementation:<br>
I can simply torch.stack my 2 tensors with dim=2 to get 1 tensor of shape (N, C, D, H, W), where D = 2 that I could then feed to a conv3d.<br>
But now, if I apply the 3d convolution, I still end up with a (N, C, D, H, W) tensor however I would like to get a (N, C, H, W) tensor in the end.<br>
My first idea was, to do a conv3d with kernel_size=(D, 3, 3) and padding=(0, 1, 1) such that the depth dimension has size 1 and we can then squeeze the resulting tensor.<br>
However isn’t a conv3d with kernel_size(D, 3,3) exactly the same as a conv2d with kernel_size(3,3)? Because if our kernels D dimension is equal to the tensors D dimension and we have no D padding, we cannot move the filter along the D dimension. Therefore the filter only moves along the H and W dimensions and it has exactly the same amount of parameters as the conv2d filter. Ergo it should be exactly the same or not?</p>
Adding a Linear layer to an LSTM makes the validation loss shoot up

shamoons: <p>My model was:</p>
<pre><code class="lang-auto">    def forward(self, x, hidden=None):
        lstm_out, hidden = self.lstm(x, hidden)

        lstm_out = (lstm_out[:, :, :self.hidden_size] +
                    lstm_out[:, :, self.hidden_size:])

        out = torch.nn.SELU()(lstm_out)
        return out, hidden
</code></pre>
<p>It now is:</p>
<pre><code class="lang-auto">    def forward(self, x, hidden=None):
        lstm_out, hidden = self.lstm(x, hidden)
        batch_size = lstm_out.size(0)

        flattened_out = lstm_out.view(-1, self.hidden_size * 2)

        lstm_out = (lstm_out[:, :, :self.hidden_size] +
                    lstm_out[:, :, self.hidden_size:])
        out = self.linear(flattened_out)

        out = torch.nn.functional.relu(out)

        view_out = out.view(batch_size, self.seq_length, -1)

        return view_out, hidden
</code></pre>
<p>I used to get validation loss (with MSELoss) under <code>1000</code> after 2-3 epochs. Now with the Linear layer, it is skyrocketing up to <code>15000</code> even after 10 epochs. Why would this be?</p>
Dataloader from Subset: how to directly get the i-th batch values

kuzand: <p>I am creating a dataloader from a Subset as follows:</p>
<pre><code class="lang-auto">dataset = torch.utils.data.TensorDataset(torch.rand(100))
train_subset, test_subset = torch.utils.data.random_split(dataset, [80, 20])
train_loader = torch.utils.data.DataLoader(train_subset, batch_size = 10)
test_loader = torch.utils.data.DataLoader(test_subset, batch_size = 10)
</code></pre>
<p>How can we get the i-th batch directly, without a for-loop? I am doing the following:</p>
<p><code>train_loader.dataset.dataset[train_loader.dataset.indices][0][list(train_loader.batch_sampler)[i]]</code></p>
<p>which looks ugly and I was wondering if there is a better way.</p>

Oli: <p>You can get a sample at index i with train_subset[i]. Since your dataloaders arent shuffled, you can calculate which indices you want and slice the dataset</p>
<pre><code class="lang-auto">train_subset[i:i+ batch_size]
</code></pre>

kuzand: <p>What if instead of the TensorDataset (which I used to simplify the example) I have ImageDataset and moreover I use a sampler (WeightedRandomSampler) for dataloader?</p>

Oli: <p>I don’t know how to solve that in a few lines of code. If you can get a hold of the indicies for the i-th batch you can easily slice it out of a custom dataset (ImageDataset in your case I guess).</p>
<p>I’ve done a sampler before where I manually shuffled the indicies which allowed me to do similar things do what you describe. Perhaps you could wrap the WeightedRandomSampler class and do something similar for yourself?</p>
BatchNorm hangs with save and load state_dict while training with multi-processes

bapi: <p>Hello,</p>
<p>In my implementation, I train the model by multiple processes and save its state dictionary by a concurrent process to evaluate/test it after the training is complete. During evaluation, I load the saved state dictionaries. I need to do this in order to compute some more info from my distributed/concurrent training algorithm. It also helps me in un-engaging the processors from test-computation-load while training.</p>
<p>This implementation works perfectly for all the models that I worked with when it is done on a CPU. On a GPU, when the number of training processes is one, again it is fine for each of those models. Furthermore, it also works fine if there is no BatchNorm in the model and I train it using multiple processes over a GPU.</p>
<p>However, with BatchNorm and multiple processes training on a GPU,  when I do a forward pass later during evaluation it hangs at Conv2d (it might be at some other place further inside, however, I could pin in my debugger up to Conv2d).</p>
<p>What does exactly happen with the BatchNorm+multiprocessing combination?</p>