akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Inverse of tensor.fold

tcapelle: <p>Hello,<br>
I want to cut a big image into patches, so I am doing the following:</p>
<pre data-code-wrap="python"><code class="lang-python">tensor_img.shape
&gt;&gt; torch.Size([3, 1536, 2048])

tensor_img = tensor_img.unfold(1, 128, 128).unfold(2, 128, 128)
tensor_img.shape
&gt;&gt; torch.Size([3, 12, 16, 128, 128])
</code></pre>
<p>How do I recover the original image?<br>
A simple <code>tensor_img.view(3, 1536, 2048)</code> does not work, as it gives me a “wrong strided” image.</p>
<blockquote>
<p>Note: I am trying to make inference on to large for memory images, so wanted to chop them down, and feed them as tiles, then reconstruct the output.</p>
</blockquote>

tcapelle: <p>Thanks, i was hoping for a more elegant solution with some crazy nice operator.</p>
Linalg for tridiagonal/Banded matrix

Jonas: <p>Hi guys</p>
<p>I was wondering if there is a way to efficiently to do matrix factorizations/solve linear systems of equations on sparse matrices that have a special banded structure, such as tridiagonality.</p>
<p>In scipy there are a bunch of functions like <a href="https://docs.scipy.org/doc/scipy/reference/linalg.html" rel="nofollow noopener">solve_triangular, solve_banded, eig_banded and cholesky_banded</a> but I couldn’t find anything for pytorch and am unforntunately not expert enough to implement that myself.</p>
<p>Any pointer would be highly appreciated</p>
<p>Cheers<br>
Jonas</p>

fjchange: <p>I think it can be realized by using torch.triu()</p>
<p>Supposed that we want a tridiagonal matrix:</p>
<pre><code class="lang-auto">A= torch.ones([5,5]),
A=torch.triu(A,-1)
Target=A*A.t()
</code></pre>
Validation accuracy varies much

111242: <p>Sometimes, there are as much as 10% validation accuracy difference between different seeds. Here’s how I am setting the seeds.</p>
<p>torch.manual_seed(1)<br>
np.random.seed(1)</p>
<p>I am also keeping sklearn’s random state identical.</p>

omarfoq: <p>Hello,</p>
<p>What is the size of your validation set and what’s the number of classes you have?</p>
RuntimeError: Expected object of scalar type Long but got scalar type Float for argument #3 'mat2' in call to _th_addmm_out

learningpytorch: <p>Hi everyone, I am getting this error and I am not sure what is happening. I am trying to predict air passenger data with a LTSM. The notebook is available on this link:</p>
<p><a href="https://drive.google.com/file/d/1lk9uKhjbMlLHBvvqmaIuBFvbVfoKAia7/view?usp=sharing" rel="noopener nofollow ugc">https://drive.google.com/file/d/1lk9uKhjbMlLHBvvqmaIuBFvbVfoKAia7/view?usp=sharing</a></p>
<p>Thanks in advance to anyone willing to help.</p>

Dwight_Foster: <p>Have you tried just converting the seq variable to a float in your training loop like this</p>
<pre><code class="lang-auto">    for seq, y_train in train_data:
        
        seq = seq.float()
        optimizer.zero_grad()
        model.hidden = (torch.zeros(1,1,model.hidden_size),
                        torch.zeros(1,1,model.hidden_size))
</code></pre>
<p>if that doesn’t work can you print out the type of seq and the hidden states.</p>

omarfoq: <p>Hello,</p>
<p>This is just because LSTM expects Float in input, and your data is Long. Use this code</p>
<pre><code class="lang-auto">for seq, y_train in train_data:
    seq = seq.type(torch.float32)
    optimizer.zero_grad()
    y_pred = model(seq)
    loss = criterion(y_pred, y_train)
    loss.backward()
    optimizer.step()
</code></pre>

learningpytorch: <p>Thank you very much <a class="mention" href="/u/omarfoq">@omarfoq</a>  and <a class="mention" href="/u/dwight_foster">@Dwight_Foster</a> but I have tried both options getting a confusing message. You can see the updated notebook here:<br>
<a href="https://drive.google.com/file/d/1VKpZCkSeEJoP_iyXAi0xBZRlPqE8B1fE/view?usp=sharing" rel="noopener nofollow ugc">https://drive.google.com/file/d/1VKpZCkSeEJoP_iyXAi0xBZRlPqE8B1fE/view?usp=sharing</a></p>
<p>How can I proceed?</p>

Dwight_Foster: <p>So now it seems like your loss is the problem. Try this:</p>
<pre><code class="lang-auto">        
        loss = criterion(y_pred, y_train).float()
        loss.backward()
        optimizer.step()
        
</code></pre>

learningpytorch: <p>It throws the same error. It is very strange.</p>
Resize Tensor With PIL and preserve Gradients

Islam_Abdelrahman: <p>Is there any way to Convert tensor that has Gradients into PIL image and then resize Image and convert it to tensor again without lossing the Gradients ?</p>

ptrblck: <p>I don’t think this exact workflow would be possible as the transformation into a <code>PILImage</code> would break the computation graph and remove the gradients. <code>PIL.Images</code> are using numpy arrays internally, which do not contain gradients.<br>
You could of course reshape the tensor directly, if this would fit your use case, but this would also break the gradients I think.<br>
What is your exact use case?</p>

Islam_Abdelrahman: <p>First Thanks for the fast response.<br>
My case is:<br>
I’m training Variational Auto Encoder That Takes images with Size 64,128,256, etc</p>
<p>But instead of using pixel wise loss I’m using perceptual loss, that’s mean I take the original image and the reconstructed image and feed them respectively to a pre-trained model (ssd) and calculate the loss of it’s hidden layers values.</p>
<p>The problem is ssd take only 300x300 image size and I train the VAE with 128, so inorder to compute the loss via SSD I should resize the reconstructed image (which has the gradients).</p>

ptrblck: <p>Would it work to use the original images, feed them to the VAE, resize the output, and feed to the SSD?<br>
It would be similar to this small example, which would still preserve the gradients in the smaller image, since the interpolation is differentiable:</p>
<pre><code class="lang-python">x = torch.randn(3, 24, 24, requires_grad=True)
x_res = F.interpolate(x, size=30)

loss = x_res.mean()
loss.backward()
print(x.grad)
</code></pre>
About MSE loss function

spring: <p>It’s a very simple problem, but it’s a bit confusing.<br>
Below is a typical MSE loss function that I am aware of.</p>
<p>1/n(∑  (y_i - y’_i)^2)<br>
*∑ → i=1 to n</p>
<p>What does n mean here is the size of a mini-batch?</p>
<p>One more question is, shouldn’t the general MSE formula be marked as follows in order to express the loss in the neural network with more than 2 outputs?</p>
<p>1/(m*k)(∑  ∑  (y_i,j - y’_i,j)^2)<br>
*First ∑ → i=1 to m, second ∑ → j=1 to k,</p>
<p>m = mini-batch size<br>
k = The number of output neurons</p>
What is the difference between DataParallel and DistributedDataParallel?

Dawny33: <p>I am going through this imagenet example: <a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py" rel="nofollow noopener">https://github.com/pytorch/examples/blob/master/imagenet/main.py</a></p>
<p>And, in line 88, the module <code>DistributedDataParallel</code> is used. When I searched for the same in the docs, I haven’t found anything. Possible to redirect me to it if any such doc exist for the module.</p>
<p>Else, would like to know what is the difference between the DataParallel and DistributedDataParallel modules.</p>

fmassa: <p><code>DataParallel</code> is for performing <a href="http://pytorch.org/docs/master/nn.html?highlight=dataparallel#torch.nn.DataParallel">training on multiple GPUs, single machine</a>.<br>
<code>DistributedDataParallel</code> is <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/parallel/distributed.py#L24-L78">useful when you want to use multiple machines</a>.</p>

csarofeen: <p>Sorry for resurrecting this old thread. The answer above made some confusion with some folks I’ve talked to.</p>
<p>Distributed Data Parallel can very much be advantageous perf wise for single node multi-gpu runs. When run in a 1 gpu / process configuration Distributed Data Parallel can be beneficial as CPU based overheads are now spread across multiple processes.</p>
<p>Perf gains will especially be prominent in networks that have many  small layers/operations. I primarily recommend to folks that they use single gpu / process Distributed Data Parallel over Data Parallel even for single node cases if they want to scale past 2 GPUs.</p>

Toru: <p>Can you please elaborate on “When run in a 1 gpu / process configuration Distributed Data Parallel can be beneficial as CPU based overheads are now spread across multiple processes”? Thanks!</p>

Hjjiang: <p>Totally agree with you！<br>
“I primarily recommend to folks that they use single gpu / process Distributed Data Parallel over Data Parallel even for single node cases if they want to scale past 2 GPUs.”</p>
Sliced tensor.mean numerical issues

patrickbryant: <p>I’ve found what seems to be a strange bug in PyTorch 1.5.0 that is absent from 1.7.1.</p>
<p>In 1.5.0 the mean of a large tensor containing large values is incorrect when including more than one column in the slice:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; t=torch.randn(1000000,2,1,dtype=torch.float).abs()+1000
&gt;&gt;&gt; t[:,0:1].mean(dim=0), t[:,0:2].mean(dim=0)[0]
(tensor([[1000.2383]]), tensor([1010.0195]))
</code></pre>
<p>In 1.7.1 the same code works as expected:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; t=torch.randn(1000000,2,1,dtype=torch.float).abs()+1000
&gt;&gt;&gt; t[:,0:1].mean(dim=0), t[:,0:2].mean(dim=0)[0]
(tensor([[1000.7977]]), tensor([1000.7977]))
</code></pre>
<p>I’d like to be able to work around this in 1.5.0 if possible. Any ideas what is going on?</p>

ptrblck: <p>The errors are most likely created due to the limited floating point precision and the applied algorithm, which might have changed between <code>1.5.0</code> and <code>1.7.1</code>.</p>
Dimension out of range (expected to be in range of [-1, 0], but got 1) in pytorch

shgh: <p>I get the following error</p>
<pre><code class="lang-auto">IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)
</code></pre>
<p>from this line: <code> x = torch.cat([a, b], 1)</code></p>
<p>in the snippet code below:</p>
<pre><code class="lang-auto">class Critic(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(Critic, self).__init__()
        self.linear1 = nn.Linear(input_size, hidden_size)
        self.linear2 = nn.Linear(hidden_size, hidden_size)
        self.linear3 = nn.Linear(hidden_size, output_size)

    def forward(self, a, b):

        x = torch.cat([a, b], 1)
        x = F.relu(self.linear1(x))
        x = F.relu(self.linear2(x))
        x = self.linear3(x)

        return x

</code></pre>
<p>Any help on this appreciated.</p>

ptrblck: <p>The error is raised, if <code>a</code> and <code>b</code> have less that two dimensions, while <code>torch.cat</code> tries to concatenate these tensors in <code>dim1</code>. Here is an example:</p>
<pre><code class="lang-python"># both tensors have a single dimension
a, b = torch.randn(10), torch.randn(10)
x = torch.cat([a, b], 1)
&gt; IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

# works
a, b = torch.randn(10, 1), torch.randn(10, 1)
x = torch.cat([a, b], 1)
</code></pre>
<p>Make sure that both tenors have at least 2 dimensions before passing them into the <code>forward</code> method.</p>
Sanity check info needed before submiting patch

Harish_S: <p>Hi all,<br>
I am new to pytorch contribution,please suggest me how to perform sanity checks(i.e blanks,spaces,indentation) before submiting new patch</p>
<p>Regards,<br>
S.Harish</p>

ptrblck: <p>The <a href="https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md">Contributing</a> docs have some information on how to lint your code and other tips for PyTorch development. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
DataLoader TypeError: img should be PIL Image. Got <class 'numpy.ndarray'>

oasjd7: <p>I have <code>.npy</code> dataset file and It works fine when <code>transform=None</code>.<br>
But when I apply <code>transform=transform_train</code>,<br>
<code>TypeError: img should be PIL Image. Got &lt;class 'numpy.ndarray'&gt;</code> occurs.<br>
How can I fix it?</p>
<pre><code class="lang-auto">import numpy as np
import os


class CIFAR10:
    def __init__(self, root='/database', split="l_train", transform=None):
        self.dataset = np.load(os.path.join(root, "cifar10", split + ".npy"), allow_pickle=True).item()
        self.transform = transform

    def __getitem__(self, idx):
        image = self.dataset["images"][idx]
        label = self.dataset["labels"][idx]
        # print(type(image)) # &lt;class 'numpy.ndarray'&gt;
        if self.transform is not None:
            image = self.transform(image)
        return image, label

    def __len__(self):
        return len(self.dataset["images"])


transform_train = transforms.Compose([
    transforms.RandomHorizontalFlip(),
    transforms.RandomCrop(size=32),
    transforms.ToTensor(),
    transforms.Normalize(mean, std)
])
trainset_x = CIFAR10(split='l_train', transform=transform_train)

</code></pre>

tlim: <p>Pytorch transform accept PIL Image, Tensor image or batch of tensor images as input.</p>
<p>Reference: <a href="https://pytorch.org/vision/stable/transforms.html" class="inline-onebox" rel="noopener nofollow ugc">torchvision.transforms — Torchvision master documentation</a></p>
<p>Maybe you can add this to your code, test whether it works:</p>
<pre><code class="lang-auto">from PIL import Image

image = Image.fromarray(self.dataset["images"][idx]
</code></pre>
RuntimeError: mat1 and mat2 shapes cannot be multiplied

shgh: <p>I get the error below</p>
<pre><code class="lang-auto">RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x256 and 3x256)
</code></pre>
<p>for this line of my code<code> x = F.relu(self.linear1(c))</code><br>
from snippet below:</p>
<pre><code class="lang-auto">class Test(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(Test, self).__init__()
        self.linear1 = nn.Linear(input_size, hidden_size)
        self.linear2 = nn.Linear(hidden_size, hidden_size)
        self.linear3 = nn.Linear(hidden_size, output_size)

    def forward(self, c):
        x = F.relu(self.linear1(c))
        x = F.relu(self.linear2(x))
        x = torch.tanh(self.linear3(x))

        return x
</code></pre>
<p>Can anyone give me a clue how to solve this?</p>

aza: <p>I believe this is due to the dimensions of the tensors in the linear calculation – c.view(c.size(0),-1) looks to be instantiating a (1x256 tensor) tensor. If you want to keep the input_size, hidden_size as 3 and 256 respectively, you will need to modify the view function so that the tensor is (256x1)</p>
By using relu function my model is giving the outputs according to the hidden layer size. if i have 6 or 7 hidden layers its giving 6 or 7 outputs but my output size is 1. Why relu() is not giving the one output?

bc060400164: <p>class LSTM(nn.Module):<br>
def <strong>init</strong>(self, input_size=1, hidden_layer_size=6, output_size=1):<br>
super().<strong>init</strong>()</p>
<pre><code>self.hidden_layer_size = hidden_layer_size
self.lstm = nn.LSTM(input_size, hidden_layer_size, output_size)
self.relu = nn.ReLU()     #activation unit # linear # try with relu ###

self.hidden_cell = (torch.zeros(1, 1, self.hidden_layer_size),   
                    torch.zeros(1, 1, self.hidden_layer_size))
</code></pre>
<p>def forward(self, input_seq):<br>
lstm_out, self.hidden_cell = self.lstm(input_seq.view(len(input_seq), 1, -1), self.hidden_cell)<br>
predictions = self.relu(lstm_out.view(len(input_seq), -1))<br>
return predictions[-1]</p>
<p>model = LSTM()<br>
loss_function = nn.MSELoss()<br>
optimizer = torch.optim.Adam(model.parameters(), lr = 0.0001)</p>
<p>print(model)</p>
<p>epochs = 300</p>
<p>for i in range(epochs):<br>
for seq, labels in train_inout_seq:<br>
optimizer.zero_grad()<br>
model.hidden_cell = (torch.zeros(1, 1, model.hidden_layer_size),<br>
torch.zeros(1, 1, model.hidden_layer_size))<br>
y_pred = model(seq)</p>
<pre><code>single_loss = loss_function(y_pred, labels)
single_loss.backward()
optimizer.step()
if i%25 == 1:
    print(f'epoch: {i:3} loss: {single_loss.item():10.8f}')
</code></pre>
<p>model.eval()</p>
<p>for i in range(fut_period):<br>
seq = torch.FloatTensor(train_inputs[-train_window:])<br>
<span class="hashtag">#print</span> (seq)<br>
with torch.no_grad():<br>
model.hidden = (torch.zeros(1, 1, model.hidden_layer_size),<br>
torch.zeros(1, 1, model.hidden_layer_size))<br>
print(model(seq))<br>
train_inputs.append(model(seq).item())</p>
<p>By using relu function my model is giving the outputs according to the hidden layer size. if i have 6 or 7 hidden layers its giving 6 or 7 outputs but my output size is 1. Why relu() is not giving the one output?</p>
Select Last Index Along Row

Jack_Rolph: <p>Suppose I have a tensor of some arbitrary, unknown length. Take for instance:</p>
<pre><code class="lang-auto">a = torch.randn(A, B, D)
</code></pre>
<p>or</p>
<pre><code class="lang-auto">b = torch.randn(A, B, C, D)
</code></pre>
<p>where <code>A</code> to <code>D</code> are arbitrary.</p>
<p>How do I index the tensor such that I get the last element of <code>X</code>,  which is always the second to last index, producing two tensors of the remaining elements?</p>
<pre><code class="lang-auto">a_x = foo(a)
b_x = foo(b)

torch.allclose(a[:,:,X-1,:], a_x)
&gt; True

torch.allclose(b[:,:,:,X-1,:], b_x)
&gt; True

a_x.shape
&gt; torch.Size([A,B,D])

b_x.shape
&gt; torch.Size([A,B,C,D])

</code></pre>

ptrblck: <p>This should work to index the second to last dimension:</p>
<pre><code class="lang-python">x[..., -2, :]
</code></pre>

Sahil: <p>What <code>...</code> will do? Any documentation regarding this? I have not found this kind of syntax…</p>
<p>Thanks</p>

ptrblck: <p>This is an <a href="https://docs.python.org/dev/library/constants.html#Ellipsis">Ellipsis</a> and can be used for indexing as shown <a href="https://numpy.org/doc/stable/reference/arrays.indexing.html">here</a>.</p>
CNN-RNN - how to manage gradients?

K-Paks: <p>Hello</p>
<p>I’m trying to create a system that will perform action recognition. I’m trying to use a combination of CNN and RNN. For CNN I am using <a href="https://pytorch.org/hub/pytorch_vision_shufflenet_v2/" rel="noopener nofollow ugc">ShuffleNet v2</a>. I replaced the <code>fc</code> of the model with my custom <code>nn.Sequential</code>:</p>
<pre><code class="lang-auto">    rcnn_layer = nn.Sequential(
        nn.Linear(1024, 128),
        View((1, 128)),
        InputModifier(10),
        nn.RNN(input_size=128, hidden_size=64),
        GetLastHidden(),
        nn.Linear(64, 3),
        nn.Softmax(dim=-1)
    )
</code></pre>
<p>The custom classes are:<br>
<code>View</code> - reshapes the output of the first linear layer so that it fits the RNN<br>
<code>InputModifier</code> - retains previous outputs of the model<br>
<code>GetLastHidden</code> - returns last layer of the RNN</p>
<p>In the future I’d like to use the model on a phone (somehow), so I’m trying to make it as efficient as I can. Therefore, in order to make it run faster, in my <code>InputModifier</code> class I’m retaining the previous outputs of the CNN. Here’s the code:</p>
<pre><code class="lang-auto">class InputModifier(nn.Module):
    prev = []

    def __init__(self, max_seq_len):
        assert max_seq_len != 0, '`max_seq_len` cannot be 0.'
        super(InputModifier, self).__init__()
        self.max_seq_len = max_seq_len

    def forward(self, x):
        self.prev.append(x)
        if self.max_seq_len &gt; 0:
            self.prev = self.prev[-self.max_seq_len:]

        inp = torch.cat(self.prev)
        return inp
</code></pre>
<p>The issue is: I don’t know how to get it to work. When I run <code>loss.backward()</code>, I get <code>RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed.</code> error.</p>
<p>If i set <code>loss.backward(retain_graph=True)</code>, I get <code>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1024, 128]], which is output 0 of TBackward, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True). </code>.</p>
<p>PS: The training code:</p>
<pre><code class="lang-auto">model = model.to(device)
optim = torch.optim.Adam(lr=0.01, params=model.parameters())
criterion = nn.CrossEntropyLoss()
model.train()
for i, (img, c) in enumerate(dataloader):

    out = model(img)
    loss = criterion(out, c.view(1))

    optim.zero_grad()
    loss.backward()
    optim.step()
</code></pre>
<p>All advices are welcome. Thanks.</p>
Product between 2D matrix and 3D tensor

nullgeppetto: <p>I’m tying to calculate the matrix-tensor product between a <code>bxd</code> matrix (a batch of <code>d</code>-dimensional vectors) and a <code>bxdxn</code> tensor (a batch of <code>dxn</code> matrices), so as the product is a <code>bxn</code> matrix, so as each row of the result is the product (a vector) between the corresponding vector of the matrix and the matrix of the tensor.</p>
<p>For instance:</p>
<pre><code class="lang-auto">import torch
X = torch.randn(32, 512)
Y = torch.randn(32, 512, 128)
# Z: needs to be of torch.Size([32, 128])
</code></pre>
<p>Thanks!</p>

omarfoq: <p>Hi,</p>
<p>I think you want to do this</p>
<pre><code class="lang-auto">import torch 

X = torch.randn(32, 1, 512)
Y = torch.randn(32, 512, 128)
Z = torch.matmul(X, Y)
</code></pre>
What is this network type called?

apytorch: <p>I can’t seem to find a clear answer to what this type of network/training is called, to see what else has been done in the field. It looks like RNN, but the recursive loop is the size of the network itself.</p>
<p>The idea is to perform iterative classification, instead of just one pass. Going off the assumption that having a prior probability for what you expect at a given input, can be used to help classify (i.e. attention) it. The benefit might be a “shorter” network.</p>
<p>I realize memory usage is one pitfall to doing this. But I’m just trying to see what similar theoretical and practical work has been done.</p>
<pre><code class="lang-auto"># assuming equal prior probability (probably not a good assumption)
initial_predictions = torch.ones([n_classes, 1])/n_classes

# first pass, assuming final network operation is a softmax
model_output = model(static_inputs, initial_predictions)

# could be a fixed number of passes, or could use some measure of the output
# to determine when the predictions have leveled off.
for i in range(4):
    # for 2nd+ pass, use the previous model_output
    model_output = model(static_inputs, model_output)

loss = criterion(model_output, targets, weights)
loss.backward()

</code></pre>

googlebot: <p>this seems to resemble <a href="https://arxiv.org/abs/1909.01377" rel="noopener nofollow ugc">deep equilibrium models</a></p>

apytorch: <p>Interesting paper. I can see the similarity. Although, I think their method looks more complex.</p>
Size Mismatch for Functional Linear Layer

ddicostanzo: <p>Hello,</p>
<p>I apologize that this is probably a simple question that has been answered before, but I could not find the answer. I’m attempting to use a CNN to extract features and then input that into a FC network that outputs 2 variables. I’m attempting to use the functional linear layer as a way to dynamically handle the flattened features. The self.cnn is a Sequential container which last layer is the nn.Flatten(). When I print the size of x after the CNN I see it is 15x152064, so I’m unclear why the F.linear layer is failing to run with the error below. Any help would be appreciated.</p>
<p>RuntimeError: size mismatch, get 15, 15x152064,2</p>
<pre><code class="lang-python">x = self.cnn(x)
batch_size, channels = x.size()
x = F.linear(x, torch.Tensor([256,channels]))
y_hat = self.FC(x)
</code></pre>

omarfoq: <p>Hi,</p>
<p>That’s because, <code>torch.Tensor([256, channels])</code> is just a the vector <code>[256, 152064]</code> which a tensor of shape <code>(2,)</code>. I think you wanted to do something like <code>x = F.linear(x, torch.randn(256, channels))</code> or something equivalent. But I don’t really see what you are trying to do, in fact the last layer you just added will have a predefined weight, and won’t be trainable, so what is the reason you are trying to add it?</p>

ddicostanzo: <p>Thanks for that, it does help. What I am trying to do is dynamically calculate the output size of the CNN and feed that into the FC network. I know I can print the size of the CNN and set the input size of the first linear layer manually, but I wasn’t sure if there is a more elegant way of dealing with possible changes in architecture of the CNN that would cause the output to vary in size.</p>

omarfoq: <p>Hi <a class="mention" href="/u/ddicostanzo">@ddicostanzo</a>,</p>
<p>I think this not possible as you want to do it in torch. However the way you are solving the problem, i.e., using CNN feature extractor followed by a Flatten layer and fully connected head is not the best choice. Modern neural networks (in computer vision) rely on a global pooling layer instead. I think you should use a similar approach.</p>

ddicostanzo: <p>Thanks <a class="mention" href="/u/omarfoq">@omarfoq</a>. I will investigate using the global pooling. Appreciate the help.</p>
Re-use of intermediate graph nodes

jacobbuckman: <p>I want to structure my neural network code in such a way as to put “postprocessing” steps into the model itself, and utilize flags for the forward pass. So for a classification NN <code>big_nn</code> I could call:</p>
<pre><code class="lang-auto">a = big_nn(x, mode='logits') # returns [-2, 2.4, 0.3, -1.1]
b = big_nn(x, mode='prediction') # returns 1
</code></pre>
<p>This leads to a very elegant coding style, but unfortunately in PyTorch it makes my training loop very inefficient! The reason is that it includes code that looks like:</p>
<pre><code class="lang-auto">loss = softmax_cross_entropy(big_nn(x, mode='logits'), y).mean()
error = (big_nn(x, mode='prediction') == y).double().mean()
</code></pre>
<p>Internally, this runs the forward pass for <code>big_nn</code> twice &amp; doubles the number of nodes in the graph, which is very expensive, and something I definitely want to avoid. This could of course be re-made efficient by doing</p>
<pre><code class="lang-auto">logits = big_nn(x, mode='logits')
loss = softmax_cross_entropy(logits, y).mean()
error = (argmax(logits, -1) == y).double().mean()
</code></pre>
<p>…but from a software design perspective, I don’t like having the logic for converting logits to decisions to live in the training code. It belongs inside the neural network class.</p>
<p>Is there a way to write efficient PyTorch code in this style, by forcing it to re-use intermediate computations?</p>

ptrblck: <p>One potential approach would be to calculate both paths internally and either return both (and discard the unwanted output) or to use the <code>mode</code> argument to define which output should be returned.</p>

jacobbuckman: <p>That’s what I’m doing now, but it still doesn’t help me much when the two calls are in different parts of my code. Call A needs output 1, and call B needs output 2; I don’t want to have to pass around stuff in order to do both calls. I just want to have the intermediate computations cached by default, until e.g. <code>.backward()</code> is called.</p>
<p>If there’s no way to do this currently, might it be worth adding as a feature? I think it would bring great efficiency gains to many workflows, and enable much more elegant code. Tensorflow 1.x was able to do this natively, and I used that ability all the time (which is how I developed this coding style).</p>
Assert self.profiler error while calling torch.profiler.profile's key_averages()

Po-Jen: <p>Hi all, I am trying the new profiler released in 1.8.1. My torch version is 1.8.1+cu102.</p>
<p>My code (Basically I just followed <a href="https://pytorch.org/docs/stable/profiler.html" class="inline-onebox" rel="noopener nofollow ugc">torch.profiler — PyTorch 1.8.1 documentation</a>, and uses <a href="https://pytorch.org/tutorials/beginner/pytorch_with_examples.html#pytorch-optim" class="inline-onebox" rel="noopener nofollow ugc">Learning PyTorch with Examples — PyTorch Tutorials 1.8.1+cu102 documentation</a> as the test code for profiling):</p>
<pre><code class="lang-python">import torch
import torch.profiler

import math

# Create Tensors to hold input and outputs.
x = torch.linspace(-math.pi, math.pi, 2000)
y = torch.sin(x)

# Prepare the input tensor (x, x^2, x^3).
p = torch.tensor([1, 2, 3])
xx = x.unsqueeze(-1).pow(p)

# Use the nn package to define our model and loss function.
model = torch.nn.Sequential(torch.nn.Linear(3, 1), torch.nn.Flatten(0, 1))
loss_fn = torch.nn.MSELoss(reduction="sum")

learning_rate = 1e-3
optimizer = torch.optim.RMSprop(model.parameters(), lr=learning_rate)

with torch.profiler.profile(
    schedule=torch.profiler.schedule(wait=1, warmup=1, active=18),
) as p:
    for t in range(20):
        y_pred = model(xx)

        loss = loss_fn(y_pred, y)
        if t % 10 == 9:
            print(t, loss.item())

        optimizer.zero_grad()
        loss.backward()

        optimizer.step()

linear_layer = model[0]
print(
    f"Result: y = {linear_layer.bias.item()} + {linear_layer.weight[:, 0].item()} x + {linear_layer.weight[:, 1].item()} x^2 + {linear_layer.weight[:, 2].item()} x^3"
)

print(type(p))
print("=====================")
print(p.key_averages().table())
</code></pre>
<p>And I got the following output:</p>
<pre><code class="lang-auto">9 3212.458251953125
19 2424.00048828125
Result: y = 0.16131113469600677 + 0.11958461999893188 x + -0.23199747502803802 x^2 + 0.03287645801901817 x^3
&lt;class 'torch.profiler.profiler.profile'&gt;
=====================
Traceback (most recent call last):
  File ".../torch_profiler_test.py", line 45, in &lt;module&gt;
    print(p.key_averages().table())
  File ".../torch/profiler/profiler.py", line 325, in key_averages
    assert self.profiler
AssertionError
</code></pre>
<p>Does anyone know what went wrong? Thank you.</p>

Po-Jen: <p>Nevermind, I’ve found that I missed p.step() in the loop. After modifying to</p>
<pre><code class="lang-python">for t in range(20):
        y_pred = model(xx)

        loss = loss_fn(y_pred, y)
        if t % 10 == 9:
            print(t, loss.item())

        optimizer.zero_grad()
        loss.backward()

        optimizer.step()
        p.step() # was missing in the code above
</code></pre>
<p>I can get the table output.</p>
Simple, complete GPU NN example

Florin_Andrei: <p>I’m testing GPU acceleration with AI frameworks. I am not familiar with PyTorch. I’m looking for a “hello world” example that trains and tests a neural network, and uses the GPU when available. To see what I have in mind, check out this MNIST example with Tensorflow (the left-hand panel):</p>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://www.gstatic.com/devrel-devsite/prod/vb236d130287e88277e2577fd354175933f40167baa8c8ed416d2ac3ecfb27ab2/tensorflow/images/favicon.png" class="site-icon" width="16" height="16">
      <a href="https://www.tensorflow.org/overview" target="_blank" rel="noopener nofollow ugc">TensorFlow</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:130/73;"><img src="https://www.tensorflow.org/images/tf_logo_social.png" class="thumbnail" width="130" height="73"></div>

<h3><a href="https://www.tensorflow.org/overview" target="_blank" rel="noopener nofollow ugc">TensorFlow Core | Machine Learning for Beginners and Experts</a></h3>

<p>Start your machine learning project with the open source ML library supported by a community of researchers, practitioners, students and developers.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>It’s small, it’s complete and in one piece, it does something interesting, and will use a GPU when available.</p>
<p>Strangely, I was not able to find something similar with PyTorch. I’ve seen examples where changes were suggested to make it run on the GPU (but assumed familiarity with PyTorch), I’ve seen fragments of code (e.g. just the training part), I’ve seen very large pieces of code - but never something short, complete, using a neural network, and GPU-enabled.</p>
<p>Could someone please point me at the right example? Thank you.</p>

omarfoq: <p>This <a href="https://pytorch.org/tutorials/beginner/basics/quickstart_tutorial.html" rel="noopener nofollow ugc">tuto</a> is maybe what you are looking for</p>

Florin_Andrei: <p>That was perfect, thank you!</p>
Efficient way to zero out certain diagonals

ADONAI_TZEVAOT: <p>I have a huge matrix A. I want to zero diagonals 2,-2 and 4. Please may I know an efficient way to do this using pytorch?</p>

KFrank: <p>Hi Adonai!</p>
<aside class="quote no-group" data-username="ADONAI_TZEVAOT" data-post="1" data-topic="116570" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/adonai_tzevaot/48/31089_2.png" class="avatar"> ADONAI_TZEVAOT:</div>
<blockquote>
<p>I want to zero diagonals 2,-2 and 4.</p>
</blockquote>
</aside>
<p>You may use <a href="https://pytorch.org/docs/1.8.1/generated/torch.diagonal.html" rel="noopener nofollow ugc">torch.diagonal()</a> to get a modifiable view into your matrix of<br>
the desired diagonals, and then use <a href="https://pytorch.org/docs/1.8.1/tensors.html#torch.Tensor.zero_" rel="noopener nofollow ugc">zero_()</a> to zero them out:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; torch.__version__
'1.7.1'
&gt;&gt;&gt; A = torch.arange (36).view ((6, 6))
&gt;&gt;&gt; A
tensor([[ 0,  1,  2,  3,  4,  5],
        [ 6,  7,  8,  9, 10, 11],
        [12, 13, 14, 15, 16, 17],
        [18, 19, 20, 21, 22, 23],
        [24, 25, 26, 27, 28, 29],
        [30, 31, 32, 33, 34, 35]])
&gt;&gt;&gt; A.diagonal (2).zero_()
tensor([0, 0, 0, 0])
&gt;&gt;&gt; A.diagonal (-2).zero_()
tensor([0, 0, 0, 0])
&gt;&gt;&gt; A.diagonal (4).zero_()
tensor([0, 0])
&gt;&gt;&gt; A
tensor([[ 0,  1,  0,  3,  0,  5],
        [ 6,  7,  8,  0, 10,  0],
        [ 0, 13, 14, 15,  0, 17],
        [18,  0, 20, 21, 22,  0],
        [24, 25,  0, 27, 28, 29],
        [30, 31, 32,  0, 34, 35]])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
Torch.ger() for batch usage

jarvico: <p>Hi all,</p>
<p>My torch version is 1.6.0.</p>
<p>And I can use torch.ger() method to calculate the outer product of two tensors.</p>
<p>For example, I can calculate the outer product of two vectors of shape 1x2 which gives me an output with shape 2x2.</p>
<p>But for example, if I store the tensors of shape 1 x 2 for which I want to calculate outer products with themselves in a batch tensor of shape N x 2  for example, I can’t do the batch calculation. At the end what I want is a solution tensor of shape N x 2 x 2 for example. Below are some example outputs</p>
<p>Would you please comment on how I can calculate this?</p>
<p>And then what I need further is to calculate the mean of the diagonal terms of the previous solution output which was of shape N x 2 x 2. My expected output for this part should be N x 1.<br>
I think I also need in this part as well.</p>
<p>Thank you in advance<br>
-Jarvi</p>
<pre><code class="lang-auto">import torch

print(torch.__version__)


a = torch.tensor([1.,2.])
b = torch.tensor([3.,4.])


print("a is ",a)
print("shape of a is ",a.shape)
print("b is ",b)
print("shape of b is ",b.shape)

outer_a = torch.ger(a,a)
print("outer product of a and a is ",outer_a)
print("shape of outer_a is ",outer_a.shape)

print("mean of the diagonal terms for outer_b is ",torch.diagonal(outer_a, 0).mean().item())

outer_b = torch.ger(b,b)
print("outer product of b and b is ",outer_b)
print("shape of outer_b is ",outer_b.shape)

print("mean of the diagonal terms for outer_b is ", torch.diagonal(outer_b, 0).mean().item())

t = torch.tensor([[1.,2.],[3.,4.]])

print("t is ",t)
print("shape of t is ",t.shape)

#outer_t = torch.ger(t,t) # THIS IS NOT WORKING

solution = torch.tensor([[[ 1.,  2.], [ 2.,  4.]], [[ 9., 12.], [12., 16.]]])

print("solution is ",solution)
print("shape of solution is ",solution.shape)

print("mean of the diagonal terms for solution is   ",torch.tensor([[2.5],[12.5]]))

OUTPUTS : 
1.6.0
a is  tensor([1., 2.])
shape of a is  torch.Size([2])
b is  tensor([3., 4.])
shape of b is  torch.Size([2])
outer product of a and a is  tensor([[1., 2.],
        [2., 4.]])
shape of outer_a is  torch.Size([2, 2])
mean of the diagonal terms for outer_b is  2.5
outer product of b and b is  tensor([[ 9., 12.],
        [12., 16.]])
shape of outer_b is  torch.Size([2, 2])
mean of the diagonal terms for outer_b is  12.5
t is  tensor([[1., 2.],
        [3., 4.]])
shape of t is  torch.Size([2, 2])
solution is  tensor([[[ 1.,  2.],
         [ 2.,  4.]],

        [[ 9., 12.],
         [12., 16.]]])
shape of solution is  torch.Size([2, 2, 2])
mean of the diagonal terms for solution is    tensor([[ 2.5000],
        [12.5000]])
#shape is 2x1
</code></pre>

jarvico: <p>Update after searching on the forum, I think these solve the problems:</p>
<ol>
<li>
</li>
</ol>
<pre><code class="lang-auto">outer_product = a[:, :, None] @ a[:, None, :]
</code></pre>
<ol start="2">
<li>
</li>
</ol>
<pre><code class="lang-auto">solution.diagonal(dim1=-2, dim2=-1)[:]
</code></pre>
ATen native sparse headers not in PIP package

sorenrasmussenai: <p>I can’t find the header files for native sparse ops in the PIP package… They don’t seem to be included.<br>
Specifically <code>ATen/native/sparse/SparseTensorMath.h</code> and <code>ATen/native/sparse/cuda/SparseCudaBlas.cuh</code>.</p>
<p>Is there any particular reason for this omission?</p>
Torch Serve Cors Configuraiton

sixthgalaxy: <p>Have found the below CORS config in the torch serve documentation.<br>
Ref: <a href="https://pytorch.org/serve/configuration.html" class="inline-onebox" rel="noopener nofollow ugc">Advanced configuration — PyTorch/Serve master documentation</a></p>
<pre><code class="lang-auto"># cors_allowed_origin is required to enable CORS, use '*' or your domain name
cors_allowed_origin=https://yourdomain.com
# required if you want to use preflight request
cors_allowed_methods=GET, POST, PUT, OPTIONS
# required if the request has an Access-Control-Request-Headers header
cors_allowed_headers=X-Custom-Header
</code></pre>
<p>Can you please confirm if I am using the right syntax? Trying to generatte a prediction using fetchAPI  and my request header contains a custom “X-Custom-Header”.<br>
(‘X-Custom-Header’, ‘ProcessThis’);</p>
<p>To allow any domains the cors_allowed_origion should be:<br>
<strong>cors_allowed_origin=</strong>*<br>
cors_allowed_headers=X-Custom-Header</p>
Dist.gather tensors of different sizes

Sergii_Makarevych: <p>Hi All.</p>
<p>The task I have is to do <code>dist.gather</code> on <code>tensors</code> of variable size. This happens during prediction stage: often multiple tensors size differ from others by 1. The idea was to pass tensor sizes to destination rank, use these sizes to prepare <code>gather_list</code> and now do <code>dist.gather</code> having proper tensor sizes. However this does not work with error: <code>ValueError: ProcessGroupGloo::gather: invalid tensor size at index 1 (expected (2), got (3))</code>. Changing tensors position in gather_list does not help.   Reproducible code:</p>
<pre><code class="lang-auto">import argparse
import torch
import torch.distributed as dist

parser = argparse.ArgumentParser(description="dist.gather test")
parser.add_argument('--local_rank', type=int, default=0)
opts = parser.parse_args()

torch.cuda.set_device(opts.local_rank)
dist.init_process_group(backend="gloo", group_name='test')
group = torch.distributed.group.WORLD

if torch.distributed.get_rank(group) == 0:
    tensor = torch.Tensor([1, 2])
else:
    tensor = torch.Tensor([4, 5, 6])

gather_list_sizes = []
if torch.distributed.get_rank(group) == 0:
    gather_list_sizes = [torch.zeros(1), torch.zeros(1)]

dist.gather(
    tensor=torch.Tensor([tensor.size()[0]]),
    gather_list=gather_list_sizes,
    dst=0,
    group=group
)
if torch.distributed.get_rank(group) == 0:
    print(gather_list_sizes)

gather_list_tensors = []
if torch.distributed.get_rank(group) == 0:
    gather_list_tensors = [torch.zeros(int(size.numpy()[0])) for size in gather_list_sizes]
    print(gather_list_tensors)

dist.gather(
    tensor=tensor,
    gather_list=gather_list_tensors,
    dst=0,
    group=group,
    async_op=True
)
if torch.distributed.get_rank(group) == 0:
    print(gather_list_tensors)
</code></pre>

Guolei_Sun: <p>Hi, do you solve the problem? I also need to gather tensors of different sizes, but during training stage. The function I want to use is dist.all_gather. I know how to gather tensors of the same size, but I cann’t find any tutorials how to gather different-size tensors.</p>
<p>Thanks</p>

Sergii_Makarevych: <p>atm, I have doubts that was me who wrote that. Unfortunately I do not remember the outcome of this.</p>
How to make dataloader return tensor?

ADONAI_TZEVAOT: <p>Here, my_dataloader is returning lists instead of torch tensor. please help.</p>
<pre><code class="lang-auto">tensor_x = torch.tensor(Mat)      #Mat is a 2d numpy array
my_dataset = TensorDataset(tensor_x)

my_dataloader = DataLoader(my_dataset,batch_size=8, shuffle = True)
</code></pre>

deepfailure: <p>Hello, have you tried:</p>
<pre><code class="lang-auto">for i, (data) in enumerate(my_dataloader):
         print(data.type())
</code></pre>
<p>What is the output of this?</p>

AlphaBetaGamma96: <p>If your data loader is returning a list of tensors, and you want a single tensor you could use <code>torch.stack</code> to stack all elements within the list to a single Tensor.</p>
<pre><code class="lang-auto">list = [ torch.randn(5,5) for _ in range(10) ] 
tensor = torch.stack(list)
tensor.shape() #returns torch.Size([10, 5, 5])
</code></pre>

ADONAI_TZEVAOT: <p>Tried what you said. It gave this : <code>AttributeError: 'list' object has no attribute 'type'</code><br>
What should I do now? Please help</p>

ADONAI_TZEVAOT: <p>I want my dataloader to return a batch of tensors</p>

AlphaBetaGamma96: <p>Does your list object contain 2 tensors? One for the input and one for the target/label?</p>

ADONAI_TZEVAOT: <p>Just inputs , no targets as it is unsupervised learning</p>

AlphaBetaGamma96: <p>But the object returned by your <code>Dataloader</code> seems to be a <code>list</code>, so perhaps it’s just a <code>list</code> of a single Tensor? Or perhaps it’s return a batch of lists?</p>
<aside class="quote no-group quote-modified" data-username="ADONAI_TZEVAOT" data-post="4" data-topic="115946">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/adonai_tzevaot/48/31089_2.png" class="avatar"> ADONAI_TZEVAOT:</div>
<blockquote>
<p>AttributeError: ‘list’ object has no attribute ‘type’</p>
</blockquote>
</aside>

omarfoq: <p>Hi,</p>
<p>You should change the <code>collate_fn</code>, something like this</p>
<pre><code class="lang-auto">import torch
from torch.utils.data import TensorDataset, DataLoader


def collate_fn(batch):
    batch = torch.cat([sample[0].unsqueeze(0) for sample in batch], dim=0)
    return batch

tensor_x = torch.tensor(Mat)      #Mat is a 2d numpy array
dataset = TensorDataset(tensor_x)
loader = DataLoader(dataset, batch_size = 8, shuffle=True, collate_fn=collate_fn)
</code></pre>
(Anyone?) question: Can I update my model structure (not the trainable parameters) during the training process?

LongWU: <p>Hi there,<br>
I want to know if I can change the model structure during the training. I know there are trainable parameter that get updated every backward propagation, and that’s not what I am asking.</p>
<p>So for example, I have a initial model with a nn.conv2d with input size of (1, 3,11,22) and one nn.linear(11,22), however during the training I find that some feature is not important or pure noise, so I need to change my nn.linear to (9,22).</p>
<p>Basically, I want the model update the structure itself so the model after the training might total different than the initial one.  It will getting more efficient by updating itself during the training by focus more on important feature rather than noise.</p>
<p>But the auto machine learning is another story which I think it’s total different strategy. Here I just want to some minor adjust to the model without autoML.<br>
Thanks for any advise.</p>

ptrblck: <p>Yes, you can create new <code>nn.Parameters</code> after each iteration manually.<br>
However, you would need to pass these new parameters to the optimizer again, so that they’ll be updated. In case the optimizer has internal states, you could also try to reuse (some) of them or let the optimizer just create new ones for the new parameter.</p>
<p>Also, you would have to come up with a strategy to change the parameters. I.e. you could slice the <code>weight</code> matrix or reduce it in any other way (e.g. <code>mean</code>) to decrease the features and you could e.g. concatenate a randomly initialized tensor to the <code>weight</code> matrix to increase the size of the <code>weight</code> matrix.<br>
Note that changing the <code>out_features</code> of one layer would also need changes in the subsequent layer in its <code>in_features</code> (and thus also internal parameters).</p>
<p>Given that, it might be easier to use the functional API and work directly with all parameters.</p>

LongWU: <p>Thanks for the answer. I have think of that and it sounds very error prone.<br>
In order to train the existing parameter after the adjustment you have to keep the right parameter and prune the unnecessary one. I haven’t try yet, but considering the large training epoch and so the number of adjustment need to make, it’s very likely to keep the wrong parameter in the next train.<br>
I personally think adjust model in real time is very agile and should be a demanding feature, so I wonder if there are any best practice or framework already address this?</p>
Using 'albumentations' In PyTorch,Yolo Bboxes will generate tags larger than 1

Sang: <p>Bboxes will generate tags larger than 1 when using ‘albumentations’ for data augmentation. Why? What should be done?</p>
<p>The error is as follows:<br>
ValueError: Expected x_max for bbox (0.284, 0.15315315315315314, 1.002, 0.993993993993994, 6.0) to be in the range [0.0, 1.0], got 1.002.</p>
RuntimeError: each element in list of batch should be of equal size

zyc: <pre><code class="lang-python">transform = transforms.Compose([
    transforms.Resize((480, 640)),
    transforms.ToTensor(),
])

train_dataset = datasets.CocoDetection(
    root=args.train_dataset, annFile=args.train_annotation, transform=transform)

train_loader = DataLoader(train_dataset, batch_size=64)

for (img, anno) in train_loader:
    out = model(img)
</code></pre>
<pre><code class="lang-python">python3.7/site-packages/torch/utils/data/_utils/collate.py", line 82, in default_collate
    raise RuntimeError('each element in list of batch should be of equal size')
RuntimeError: each element in list of batch should be of equal size
</code></pre>

zyc: <p>Issue resolved by downgrad back to PyTorch 1.5.0<br>
So it looks like a PyTorch 1.6 issue</p>

haruishi: <p>I too got this error while using pytorch 1.6, so I did the same as you and downgraded it and it worked fine</p>

zyc: <p>Could you please provide more information at <a href="https://github.com/pytorch/pytorch/issues/42654" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/42654</a> ?</p>

Fat_L_GhriBi: <p>how to downgrad back to PyTorch 1.5.0 ??</p>

haolsun: <p>Comment the checking operation in  <strong>site-packages/torch/utils/data/_utils/collate.py</strong> (line 80 81)</p>
<pre><code class="lang-auto">#if not all(len(elem) == elem_size for elem in it):
        #    raise RuntimeError('each element in list of batch should be of equal size')
</code></pre>
<p>That works for me with torch1.8 on Windows10.</p>