akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
😊 C++ Extension Out of Memory

xingjian-zhang: <p>Hi, I am new to the Pytorch C++ extension. I wanted to translate my python code into C++ but got a “Runtime Error: CUDA Out of Memory” while training. Here are my python codes and C++ codes. I wonder if there are any obvious mistakes in my C++ codes leading to an OOM.</p>
<p>The below coding block (in pure python) can run without errors.</p>
<pre><code class="lang-python">class TopKLoss(object):
    def __init__(self, c=5., T=10000, clamp_min=-10):
        self.c = c
        self.T = T
        self.clamp_min = clamp_min

    def __call__(self, outputs, labels):
        loss = []
        for w, A in zip(outputs, labels):
            A = torch.from_numpy(np.array(A)).to(w.device)
            w = w.view(1, -1)
            A = A.view(1, -1)
            loss.append(
                nll_topk_one_sample(
                    w, A, self.c, self.T, self.clamp_min))
        return torch.stack(loss).mean()
</code></pre>
<p>Then I translate it into corresponding C++ implementations and use pybind to create the extension.</p>
<pre><code class="lang-auto">class TopKLoss(object):
    def __init__(self, c=5., T=10000, clamp_min=-10):
        self.c = c
        self.T = T
        self.clamp_min = clamp_min

    def __call__(self, outputs, labels):
        # use c++ extension here
        loss = TopKLoss_call_cimpl(outputs, labels, self.c, self.T, self.clamp_min)
        return loss
</code></pre>
<pre><code class="lang-cpp">torch::Tensor TopKLoss_call_cimpl(torch::Tensor outputs,
                                  std::vector&lt;std::vector&lt;int&gt;&gt; labels,
                                  double c, double T, double clamp_min) {
    std::vector&lt;torch::Tensor&gt; loss;
    int64_t size = (int64_t)labels.size();
    loss.reserve(size);
    at::parallel_for(0, size, 0, [&amp;](int64_t begin, int64_t end) {
        for (auto i = begin; i &lt; end; i++) {
            auto w = outputs.index({i}).view({1, -1});
            auto A = torch::tensor(labels[i], w.device()).view({1, -1});
            loss.push_back(nll_topk_one_sample_cimpl(w, A, c, T, clamp_min));
        }
    });
    return torch::stack(loss).mean();
}
</code></pre>
<p>Then I got</p>
<pre><code class="lang-nohighlight">Traceback (most recent call last):
  File "cmain.py", line 283, in &lt;module&gt;
    loss = criterion(outputs, labels)  # loss
  File "/home/jimmyzxj/Research/ListwiseLTR/closs.py", line 79, in __call__
    loss = TopKLoss_call_cimpl(outputs, labels, self.c, self.T, self.clamp_min)
RuntimeError: CUDA out of memory. Tried to allocate 504.00 MiB (GPU 0; 15.78 GiB total capacity; 13.10 GiB already allocated; 443.50 MiB free; 14.23 GiB reserved in total by PyTorch)
</code></pre>
<p>Please let me know if you can spot any reasons that might lead to an OOM. Great thanks!</p>

xingjian-zhang: <p>Great thanks to any can help me!</p>
Unable to use GPU

Chris_H: <p>I’m trying to set up PyTorch 1.7.0 on a Windows 10 machine with 2 GPUs (2080ti) and CUDA 10.2.</p>
<p>It installs correctly, and at first everything looks ok - torch.cuda.is_available() returns True, device_count() returns 2, get_device_name() returns ‘GeForce RTX 2080 Ti’, and get_device_properties().total_memory shows 11GB for each card.</p>
<p>However, if I create a tensor of size 1 and try to put it on either GPU I get ‘RuntimeError: CUDA error: out of memory’. If I try to put a tensor on GPU 0 after trying to put one on GPU 1, the error changes to ‘RuntimeError: CUDA error: all CUDA-capable devices are busy or unavailable’</p>
<p>I’ve noticed that the first time I call any CUDA function (including is_available) the display goes blank for about 10 to 15 seconds. Windows event log shows that the Nvidia driver crashed and was restarted during this time. is_available returns True when this happens, and subsequent calls do not cause this driver crash.</p>
<p>I have tried different versions of PyTorch and Cuda with similar results, and I have updated Windows and Nvidia drivers. I have another machine with identical hardware also running Windows 10 that can run PyTorch with no problems.</p>
<p>Any ideas?</p>

Chris_H: <p><img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=9" title=":thinking:" class="emoji only-emoji" alt=":thinking:"> <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=9" title=":thinking:" class="emoji only-emoji" alt=":thinking:"></p>
Multiple assignment best practices

rojo: <p>I have two tensors I want to update “at the same time”, and those updates depend on each other, inside a for loop. In “general Python”, I can update them on one line:</p>
<pre><code class="lang-auto">for i in range(10):
    x, y = x + y/2, y + x/2
    x = some_nn(x)
    y = some_nn(y)
</code></pre>
<p>Is this safe to do in PyTorch in the <code>forward</code> method? Specifically is it ok for autograd, and for eventual multiple GPU training?</p>
<p>Alternatively, I think, I could do this as</p>
<pre><code class="lang-auto">for i in range(10):
    x_old = x.clone()  # possibly with .detach(), too?
    y_old = y.clone()
    x = x + y_old/2
    y = y + x_old/2
    x = some_nn(x)
    y = some_nn(y)
</code></pre>
<p>(Of course, I could put <code>some_function</code> a line earlier in both cases, but I don’t think that’s important here.)</p>
<p>Is there a reason to choose one method over the other, or is there a better method I haven’t thought of?</p>
One sample from each class in every batch

Btorch: <p>Hi, I’m training a classification model with large number of classes (40000), and I want that in each batch , there will be <strong>no more than one sample</strong> from each class.<br>
I’ve tried to create a custom sampler , but it had to much of “edge cases”, I tried to make things after I already got the batch “in my hands”, this suffered from the same issue.<br>
Do you guys have any idea? on how to simply do that?<br>
Of course that my batch &lt;&lt; number of classes.</p>
<p>Thanks for you help!</p>
Gradient of cholesky and cholesky_inverse

yzxhd: <p>I have question about the gradient of <code>torch.cholesky</code> and <code> torch.cholesky_inverse</code>. I use the following example to solve a linear equation <code>Ax=b</code>, where <code>A</code> is symmetric. Then <code>x</code> is used to calculate loss. However, when <code>loss.backward()</code>, I find that <code>A.grad</code> is None. I am wondering what the correct way is to compute the gradient of <code>loss</code> w.r.t <code>A</code> and <code>b</code>.</p>
<p>Is it because the backward gradient is not implemented? Moreover, how to check whether a function in Pytorch has backward gradient?</p>
<pre><code class="lang-auto">import torch
A = torch.randn(3, 3, requires_grad=True)
A = torch.mm(A, A.t()) + 1e-05 * torch.eye(3) # make symmetric positive definite
b = torch.randn(3, 1, requires_grad=True)

u = torch.cholesky(A)
Ainv = torch.cholesky_inverse(u)

x = torch.matmul(Ainv, b)
loss = x.mean()
loss.backward()
print(A.grad)
</code></pre>

ptrblck: <p>In your code snippet you are overwriting the leaf tensor <code>A</code> with a new tensor also called <code>A</code> and thus you should also get a warning when you try to access its <code>.grad</code> attribute:</p>
<pre><code class="lang-python">UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the gradient for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations.
  print(A.grad)
</code></pre>
<p>You can avoid this by assigning the matmul result to a new tensor:</p>
<pre><code class="lang-python">A = torch.randn(3, 3, requires_grad=True)
A1 = torch.mm(A, A.t()) + 1e-05 * torch.eye(3) # make symmetric positive definite
b = torch.randn(3, 1, requires_grad=True)

u = torch.cholesky(A1)
Ainv = torch.cholesky_inverse(u)

x = torch.matmul(Ainv, b)
loss = x.mean()
loss.backward()
print(A.grad)
</code></pre>
Build pytorch-gpu for different gpu archs

FantasyJXF: <p>I build <code>torch-1.6.0a0+b31f58d-cp36-cp36m-linux_x86_64.whl</code> on my Tesla V100 machine, when I install the <code>wheel</code> package on my Tesla T4 machine, I met the warning like this:</p>
<pre><code class="lang-auto">Tesla T4 with CUDA capability sm_75 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_70.
</code></pre>
<p>I wonder how should I build pytorch-gpu for different gpu archs?</p>
<p>Github issue <a href="https://github.com/pytorch/pytorch/issues/47268" rel="noopener nofollow ugc">here</a>.</p>

ptrblck: <p>I’ve answered in the GitHub issue.</p>
Commercial use of Torch Vision Models

VISHNU_A.P: <p>Hi Team,</p>
<p>I’m really impressed with torch vision pretrained models.<br>
I have a small question that is can we make use of any of these pretrained models especially<br>
Person Keypoint Detection (trained on coco image set ) in a commercial application?<br>
is there any license issue ?<br>
or is there any license or something i can purchase to make it happen??</p>
'Linear' object has no attribute 'log_softmax'

Ethereal: <p>Hi, this is my first time writing a Neural Network using PyTorch and I encountered the following error</p>
<pre><code class="lang-auto">'Linear' object has no attribute 'log_softmax'
</code></pre>
<p>Here’s my code:</p>
<pre><code class="lang-auto">class ConvNet(nn.Module):
  def __init__(self):
    super(ConvNet, self).__init__()

    self.conv1 = nn.Conv2d(in_channels=3,       # 224 x 224 -&gt; 222 x 222
                           out_channels = 64,
                           kernel_size = 3)
    self.maxpool1 = nn.MaxPool2d(kernel_size=2)   # 222 x 222 -&gt; 111 x 111
    self.batchnorm1 = nn.BatchNorm2d(64)
    self.dropout1 = nn.Dropout(p = 0.5)

    self.conv2 = nn.Conv2d(in_channels = 64,         # 111 x 111 -&gt; 106 x 106
                           out_channels = 128,
                           kernel_size = 5)
    self.maxpool2 = nn.MaxPool2d(kernel_size = 2)  # 106 x 106 -&gt; 53 x 53
    self.batchnorm2 = nn.BatchNorm2d(128)
    self.dropout2 = nn.Dropout(p = 0.3)

    self.conv3 = nn.Conv2d(in_channels = 128,     # 53 x 53 -&gt; 46 x 46
                           out_channels = 256,
                           kernel_size = 7)
    self.maxpool3 = nn.MaxPool2d(kernel_size=2)  # 46 x 46 -&gt; 23 x 23
    self.batchnorm3 = nn.BatchNorm2d(256)
    self.dropout3 = nn.Dropout(p = 0.2)

    self.fc1 = nn.Linear(256*23*23, 1000)
    self.dropout4 = nn.Dropout(p = 0.1)

    self.fc2 = nn.Linear(1000, 2)

  def forward(self, x):
    x = self.conv1(x)
    x = self.maxpool1(x)
    x = self.batchnorm1(x)
    x = self.dropout1(x)
    x = F.relu(x)

    x = self.conv2(x)
    x = self.maxpool2(x)
    x = self.batchnorm2(x)
    x = self.dropout2(x)
    x = F.relu(x)

    x = self.conv3(x)
    x = self.maxpool3(x)
    x = self.batchnorm3(x)
    x = self.dropout3(x)
    x = F.relu(x)

    print(x.shape)
    x = x.view(x.size(0), -1)
    print(x.shape)

    x = self.fc1(x)
    x = self.dropout4(x)
    x = F.relu(x)
    x = self.fc2

    return x
</code></pre>
<p>Here’s the code for my training loop</p>
<pre><code class="lang-auto">In_epochs = 10
lr = 0.001
cec_loss = nn.CrossEntropyLoss()
params = model.parameters()
optimizer = optim.Adam(params = params, lr = 0.001)

n_epochs = 3
n_iterations = 0

losses = []
for e in range(n_epochs):
  counter = 0
  for i, (images, labels) in enumerate(train_loader):
    images = Variable(images)
    labels = Variable(labels)
    output = model(images)

    model.zero_grad()
    loss = cec_loss(output, labels)
    loss.backward()

    optimizer.step()
    n_iterations +=1 
    counter += 1

    if counter % 100 == 0:
      print(loss.item())
</code></pre>

ptrblck: <p>You are accidentally returning the <code>self.fc2</code> layer in your model:</p>
<pre><code class="lang-python">x = self.fc2
return x
</code></pre>
<p>instead of calling it with <code>x</code> and returning the activation.<br>
Change it to:</p>
<pre><code class="lang-python">x = self.fc2(x)
return x
</code></pre>
<p>and it should work. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
NLP. RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

vorlket: <p>Hi I have the following code: <a href="https://gist.github.com/vorlkets/f38c91aa52cb4e68976e1f638d251a85" rel="noopener nofollow ugc">https://gist.github.com/vorlkets/f38c91aa52cb4e68976e1f638d251a85</a>. When run, I get the following error:</p>
<blockquote>
<p>python hw2main.py<br>
Using device: cpu</p>
</blockquote>
<p>Traceback (most recent call last):<br>
File “hw2main.py”, line 142, in <br>
main()<br>
File “hw2main.py”, line 76, in main<br>
loss.backward()<br>
File “C:\Users\Cliff\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\tensor.py”, line 195, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “C:\Users\Cliff\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\autograd_<em>init</em>_.py”, line 97, in backward<br>
Variable._execution_engine.run_backward(<br>
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</p>
<p>What does the error tell me? and how to fix this?</p>

ptrblck: <p>I haven’t checked the complete code, as it’s quite long, but if you are using your custom <code>loss</code> definition note that the operations are not differentiable and will thus detach the loss value from the computation graph:</p>
<pre><code class="lang-python">x = torch.randn(10, requires_grad=True)
ratingOutput = torch.round(x).long()
categoryOutput = torch.argmax(x)
print(ratingOutput.grad_fn)
&gt; None
print(categoryOutput.grad_fn)
&gt; None
</code></pre>
<p>As you can see, the <code>.grad_fn</code> are <code>None</code> after the operations used in your <code>loss()</code> definition.</p>
LSTM Size mismatch

Oguzhan: <p>Hi, I am trying to implement an LSTM model to predict coronavirus cases and I got this error : <strong>RuntimeError: Expected hidden[0] size (2, 134, 14), got (2, 14, 14)</strong></p>
<p>Here is some informations :<br>
Len test : 47<br>
Len validation : 37<br>
Len train : 149<br>
seq_len = 14<br>
input_size = 1<br>
output_size = 1<br>
hidden_dim = 14<br>
n_layers = 2<br>
dropout = 0.4<br>
num_epochs = 10</p>
<p>and the model is that :</p>
<pre><code class="lang-auto">class LSTM(nn.Module):
    def __init__(self, input_size, output_size, hidden_dim, n_layers,seq_len,dropout):
        super(LSTM, self).__init__()
        
        self.hidden_dim=hidden_dim
        
        
        self.lstm = nn.LSTM(input_size, hidden_dim, n_layers, batch_first=True, dropout = dropout)
        
        self.fc = nn.Linear(hidden_dim, output_size)
        
    def reset_hidden_state(self):
        self.hidden = (
        torch.zeros(n_layers, seq_len, hidden_dim),
        torch.zeros(n_layers, seq_len, hidden_dim))
        
    def forward(self, sequences):
        lstm_out, self.hidden = self.lstm(
      sequences.view(len(sequences), seq_len, -1),
      self.hidden)
        
        last_time_step = lstm_out.view(seq_len, len(sequences), self.n_hidden)[-1]
        y_pred = self.linear(last_time_step)
        return y_pred
</code></pre>
<p>What can I do to solve this problem ?</p>

ptrblck: <p>There are a few issues in your code:</p>
<ul>
<li>the states should have the shape <code>[num_layers * num_directions, batch_size, hidden_size]</code>, while your <code>self.hidden</code> tuple contains the <code>seq_len</code>.</li>
<li>if you want to permute the dimensions in <code>sequences</code> using <code>sequences.view(len(sequences), seq_len, -1)</code>, note that you would interleave the data, so use <code>sequences.permute</code> instead.</li>
<li>the same applies for <code>lstm_out.view</code>. Swapping dimensions should be done with <code>permute</code>. Since you are using <code>batch_first=True</code>, you can instead index the output via <code>lstm_out[:, -1, :]</code> without any permutation.</li>
<li>
<code>self.linear</code> is undefined and should be replaced with <code>self.fc</code>.</li>
</ul>
<p>The <a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html?highlight=lstm#torch.nn.LSTM">docs</a> provide also additional information about the expected shapes. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
GAN training - How to strengthen my discriminator?

island99: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/0/d0f15cd7aa405429bf7de88f3e21f883593c5cab.png" alt="image" data-base62-sha1="tOoqIGAjpLFp3UaI4cLCTBu0Xft" width="482" height="213"><br>
I use GAN in a NLP task. The loss of discriminator is approximately stable at 0.8, and the generator is only 0.3. This pic shows the loss of my training, does that mean my discriminator is weak? How to strengthen my discriminator?</p>
NAN while training the model, RuntimeError: Function 'PowBackward0' returned nan values in its 0th output

kunasiramesh: <p>Hi All,</p>
<p>I am using the custom loss function:<br>
<strong>loss = torch.mean(torch.square(torch.sqrt(y_true + 1e-10) - torch.sqrt(y_predict + 1e-10)) + 10*torch.square(torch.square(torch.sqrt(y_true + 1e-10) - torch.sqrt(y_predict + 1e-10))))</strong></p>
<p>After some iteration, I am getting below error.</p>
<p>[W python_anomaly_mode.cpp:60] Warning: Error detected in PowBackward0. Traceback of forward call that caused the error:<br>
File “main.py”, line 38, in <br>
main()<br>
File “main.py”, line 34, in main<br>
train(dataloader_train=train_dl, dataloader_eval=valid_dl, model=model, hyper_params=train_params, device=‘cuda’)<br>
File “train_model.py”, line 81, in train<br>
loss = my_cost(outputs,labels)<br>
File “train_model.py”, line 15, in my_cost<br>
loss = torch.mean(torch.square(torch.sqrt(y_true + 1e-10) - torch.sqrt(y_predict + 1e-10)) + 10*torch.square(torch.square(torch.sqrt(y_true + 1e-10) - torch.sqrt(y_predict + 1e-10))))<br>
(function print_stack)<br>
Traceback (most recent call last):<br>
File “main.py”, line 38, in <br>
main()<br>
File “main.py”, line 34, in main<br>
train(dataloader_train=train_dl, dataloader_eval=valid_dl, model=model, hyper_params=train_params, device=‘cuda’)<br>
File “train_model.py”, line 83, in train<br>
loss.backward()<br>
File “/usr/local/lib/python3.6/dist-packages/torch/tensor.py”, line 185, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “/usr/local/lib/python3.6/dist-packages/torch/autograd/<strong>init</strong>.py”, line 127, in backward<br>
allow_unreachable=True)  # allow_unreachable flag<br>
RuntimeError: Function ‘PowBackward0’ returned nan values in its 0th output.</p>
<p>My final output is after relu activation, so I am sending only +ve values to the sqrt function</p>

tom: <p>Maybe you could output the value of y_true and y_predict when the exception happens to be sure.</p>

kunasiramesh: <p>Thank you the reply Tom.</p>
<p>I have printed the estimated output when the exception happened.</p>
<p>y_true::tensor([[[0.9508, 0.9464, 0.9941,  …, 0.1872, 0.4230, 0.4505],<br>
[0.9412, 0.9590, 0.9167,  …, 0.0199, 0.0446, 0.0476],<br>
[1.0088, 0.9939, 1.1853,  …, 0.0752, 0.1353, 0.1411],<br>
…,<br>
[1.0073, 1.0330, 1.0652,  …, 0.3139, 0.7555, 0.8156],<br>
[0.9773, 0.9773, 0.9945,  …, 0.6462, 0.8663, 0.8789],<br>
[1.0590, 1.0328, 1.0088,  …, 0.8305, 0.9132, 0.9175]]],<br>
device=‘cuda:0’)</p>
<p>y_predict::tensor([[[0.2288, 0.0000, 0.0000,  …, 0.0000, 0.0000, 0.1711],<br>
[0.2288, 0.0000, 0.0000,  …, 0.0000, 0.0000, 0.1711],<br>
[0.2288, 0.0000, 0.0000,  …, 0.0000, 0.0000, 0.1711],<br>
…,<br>
[0.2288, 0.0000, 0.0000,  …, 0.0000, 0.0000, 0.1711],<br>
[0.2288, 0.0000, 0.0000,  …, 0.0000, 0.0000, 0.1711],<br>
[0.2288, 0.0000, 0.0000,  …, 0.0000, 0.0000, 0.1711]]],<br>
device=‘cuda:0’, grad_fn=)</p>

tom: <p>So if you take min() of them, that’s positive (well non-negative), too, right?</p>

kunasiramesh: <p>Sorry Tom, I didn’t understand. Can you elaborate?</p>
Loss function based on similarity of a patterns

NadiaMe: <p>Given two patterns/images A and B, is there any loss function that would compute how these patterns are similar?</p>

jeremysalwen: <p>There are lots of different loss functions you could use, depending on how simple you want to make things, and what aspect of image similarity you care about.  It’s a huge field with lots of ongoing active research.  Some keywords to search are “perceptual similarity” “image similarity metric” “structural similarity”.</p>
<p>The simplest thing you could do is use MSE (Mean Squared Error).</p>
Same weights different output for 2D conv - PyTorch vs TF2.x/Keras

xian_kgx: <p>Hi, I’m attempting to port a model from PyTorch to TF 2.x/Keras. I was able to load PyTorch weights in Keras, but the outputs does not seem to be equal, even for a simple conv 2D layer.</p>
<p>The following is the code that I used.</p>
<pre><code class="lang-auto">def filter_dict_by_name(name, state_dict):
    return dict(filter(lambda t: t[0].startswith(name),  state_dict.items()))


def strip_dict_key_prefix(state_dict, cut=-2):
    """Strip parameter key prefix."""
    return {f"{'.'.join(k.split('.')[cut:])}": v for k, v in state_dict.items()}


def conv2d_weights_pth2tf(params_dict):
    """Convert conv2d weights from Pytorch to TensorFlow."""
    
    weights = []
    
    params_dict = strip_dict_key_prefix(params_dict, -1)
    
    # gamma, beta, moving_mean, moving_variance
    for key in ["weight", "bias"]:
        if key not in params_dict:
            continue
            
        if key == "weight":
            weights.append(np.transpose(params_dict[key].detach().numpy(), [2, 3, 1, 0]))
        else:
            weights.append(params_dict[key].detach().numpy())
    
    return weights


def get_weight_conversion_fn(layer):
    """Get the appropriate weight conversion function given the layer."""
    
    if isinstance(layer, keras.layers.Conv2D):
        fn = conv2d_weights_pth2tf
    elif isinstance(layer, keras.layers.BatchNormalization):
        fn = bn_weights_pth2tf
    else:
        raise Exception(f"Unknown layer type: {type(layer)}")
        
    return fn

# Create two conv 2D layers, one in TF, another in PyTorch
conv1_tf = keras.layers.Conv2D(
            64,
            kernel_size=7,
            strides=2,
            padding="same",
            use_bias=False
        )

conv1 = torch.nn.Conv2d(
    3, 
    64, 
    kernel_size=7, 
    stride=2, 
    padding=3, 
    bias=False
)

conv1_state_dict = strip_dict_key_prefix(filter_dict_by_name("conv1", state_dict), -1)
conv1.load_state_dict(
    conv1_state_dict
)
# outputs: &lt;All keys matched successfully&gt;
conv1.eval()

conv1_tf.set_weights(
    get_weight_conversion_fn(conv1_tf)(
        conv1_state_dict
    )
)

conv1_out = conv1(torch.from_numpy(fixed_input).permute(0, 3, 1, 2)) \
                .permute(0, 2, 3, 1) \
                .detach() \
                .numpy()
conv1_out_tf = conv1_tf(fixed_input)\
                .numpy()

np.allclose(conv1_out, conv1_out_tf)
# outputs: False
</code></pre>

virg_hr: <p>Hi, did you manage to fix the problem in the end and get a similar output for the Keras Conv2D? I am in a similar conversion process and I observe this behaviour as well. Thanks.</p>

xian_kgx: <p>Yeah, in the end I had to do a custom tf.Layer Conv2D class mimicking PyTorch’s Conv2D, most notably the padding part. Was able to get the output of the two close.</p>
NLP. RuntimeError: bool value of Tensor with more than one value is ambiguous

vorlket: <p>I have the following code:<a href="https://gist.github.com/vorlkets/f38c91aa52cb4e68976e1f638d251a85" rel="noopener nofollow ugc">https://gist.github.com/vorlkets/f38c91aa52cb4e68976e1f638d251a85 </a>. When run I get RuntimeError: bool value of Tensor with more than one value is ambiguous. Can someone tell me what’s going on? The following is the error trace:</p>
<blockquote>
<p>python hw2main.py<br>
Using device: cpu</p>
</blockquote>
<p>Traceback (most recent call last):<br>
File “hw2main.py”, line 142, in <br>
main()<br>
File “hw2main.py”, line 73, in main<br>
loss = lossFunc(ratingOutput, categoryOutput, rating, businessCategory)<br>
File “C:\Users\Cliff\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\nn\modules\module.py”, line 532, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “F:\Master of Information Technology at UNSW\COMP9444 Neural Networks and Deep Learning\Assignment\Assignment 2\hw2\student.py”, line 120, in forward<br>
if categoryOutput[0] == torch.max(categoryOutput): buscat = 0<br>
RuntimeError: bool value of Tensor with more than one value is ambiguous</p>

Abhilash_Srivastava: <p>The error Message <code>Bool value of Tensor with more than one value is ambiguous</code>  appears when you try to cast a tensor into a bool value. This happens most commonly when passing the tensor to an <strong>if</strong>  condition.<br>
A better way of doing this could be to use <code>torch.argmax</code> instead.</p>
C++ API for tensor construction

m75: <p>How do you construct a tensor from data in C++ API?</p>
<p>Python:<br>
<code>x = torch.Tensor([[1,2],[2,3],[3,4],[4,-1],[5,-5]])</code></p>
<p>C++:</p>
<pre><code class="lang-auto">std::vector&lt;torch::jit::IValue&gt; inputs;
auto input = torch::zeros({5,5});
input[0][0] = 1; // manually copy values one by one
input[0][1] = 2;
inputs.push_back(input);
</code></pre>

tom: <p>The problem is in the Python. The <a href="https://pytorch.org/blog/pytorch-0_4_0-migration-guide/#creating-tensors" rel="noopener nofollow ugc"><code>torch.Tensor</code> constructor has been deprecated in PyTorch 0.4</a> or so. The new way is using <code>torch.tensor</code> instead, and that has an equivalent in C++. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Parallel architectures not registered

YovelRom: <p>Hi!<br>
I’m trying to create a network with multiple parallel architectures, that in the end are joined by common layers. However, the parallel architectures’ parameters are not registered with the model. What’s the best way to do that so they are registered automatically, or register them manually?<br>
The number of the parallel architectures may vary, so I can’t just give each Sequential its own field.</p>
<pre><code class="lang-python">class Dummy(nn.Module):
    def __init__(self):
        super(Dummy, self).__init__()
        self.model = []
        for _ in range(3):
            layers = []
            for i in range(5):
                layers.append(nn.Linear(28**2, 28**2))
                layers.append(nn.Tanh())
            self.model.append(nn.Sequential(*layers))
        self.linear = nn.Linear(28**2 * 5, 10)
    
    def forward(self, x):
        channels_out = []
        for c in self.model:
            channels_out.append(c(x))
        out = self.linear(torch.cat(channels_out, dim=-1))
        return out
</code></pre>

jeremysalwen: <p>Anything that is a field on your Dummy class will be registered.  So in your case you could just replace <code>layers</code> with <code>self.layers</code> and make sure to only initialize it once.  (you could also create nested lists instead of putting all layers from all models into a single list).</p>
Why is audio classification tutorial removed?

TinfoilHat0: <p>I was looking to do some audio classification with PyTorch.</p>
<p>I’ve found <a href="https://pytorch.org/tutorials/beginner/audio_classifier_tutorial.html?highlight=audio" rel="noopener nofollow ugc">this</a> tutorial but its code is removed.<br>
Why’s that?</p>

ptrblck: <p>The tutorial was removed in <a href="https://github.com/pytorch/tutorials/commit/ad0dd80eae88af57f73587da7ea71534071b0af9#diff-0dff437e3599fdb7a3fe8c59409303dc6f5bd3d7dc851fc45e025cb279512239">this PR</a> over a year ago and I guess it might have been outdated by then.<br>
CC <a class="mention" href="/u/vincentqb">@vincentqb</a> who might know more.</p>

TinfoilHat0: <p>I’ve noticed the tutorial uses some deprecated methods from torchaudio. This could be the reason.<br>
However, it’s almost trivial to convert the old code (can be found <a href="https://colab.research.google.com/github/pytorch/tutorials/blob/gh-pages/_downloads/audio_classifier_tutorial.ipynb" rel="noopener nofollow ugc">here</a>), to a compatible version (which I did).</p>
Using MSE loss on batch

Idang: <p>I’m trying to use MSE loss on a batch the following way:<br>
My CNN’s output is a vector of 32 samples. So, for example, if my batch size is 4, I’ll have an output of 4X32 samples. Each output vector needs to be loss- calculated with another vector. Then, I want to take each vector and apply on it backward function and so on.<br>
the code as it is right now:</p>
<pre><code>loss_criterion = nn.MSELoss(reduction= 'none')
...    
...
for batch_idx, data in enumerate(training_loader, 0):
        optimizerS.zero_grad()
        simulated_spec = netS(data['image_tensor'], batch_size)
        S_loss = loss_criterion(data['metadata_tensor'], simulated_spec)
        S_loss.backward()
        optimizerS.step()
</code></pre>
<p>S_loss is now a tensor of (batch size, 32) , and I get a run time error:<br>
raise RuntimeError(“grad can be implicitly created only for scalar outputs”)</p>
<p>how can it be solved and how do i get the MSE loss per every 32 samples?</p>

jeremysalwen: <p>You would just take the sum or the mean over the batch to get the total loss.  i.e. <code>S_loss.mean()</code> or <code>S_loss.sum()</code>.  I’m guessing if your total loss is supposed to be MSE you would be taking the mean.</p>
Mobile optimization error

valerianG: <p>I am trying to execute the retinanet model included in torchvision on an android mobile with Pytorch Mobile. When using the following snippet :</p>
<pre><code class="lang-auto">import torch
import torchvision
from torch.utils.mobile_optimizer import optimize_for_mobile

model = torchvision.models.detection.retinanet_resnet50_fpn(pretrained=True)
model.eval()

traced_script_module = torch.jit.script(model)
traced_script_module_optimized = optimize_for_mobile(traced_script_module)
</code></pre>
<p>I get the following error :</p>
<pre><code class="lang-auto">  File "xxx.py", line 28, in &lt;module&gt;
    traced_script_module_optimized = optimize_for_mobile(traced_script_module)
  File "yyy/torch/utils/mobile_optimizer.py", line 43, in optimize_for_mobile
    optimized_cpp_module = torch._C._jit_pass_optimize_for_mobile(script_module._c, optimization_blocklist, preserved_methods)
RuntimeError: node-&gt;kind() == prim::GetAttr INTERNAL ASSERT FAILED at "/opt/conda/conda-bld/pytorch_1603729096996/work/torch/csrc/jit/passes/freeze_module.cpp":22, please report a bug to PyTorch. Expected prim::GetAttr nodes
</code></pre>
<p>Is retinanet incompatible with optimize_for_mobile() or is it a bug ? Is there any workaround ?</p>

ptrblck: <p>Could you create an issue on <a href="https://github.com/pytorch/pytorch/issues">GitHub</a> so that we can track it, please?</p>

valerianG: <p>Of course, it has been done.</p>
Normal inference works fine, but captum (IntegratedGradients) throws a size missmatch

brix: <p>Hi,<br>
I am trying to use captum for the frist time and would like to use integrated gradients a analyze my model.</p>
<p>Nomal inference works fine, but  trying to use integrated gradients with captum leads to an error:</p>
<blockquote>
<p>RuntimeError: size mismatch, m1: [1 x 9800], m2: [196 x 512] at /opt/conda/conda-bld/pytorch_1579022027550/work/aten/src/TH/generic/THTensorMath.cpp:136</p>
</blockquote>
<p>I have no clue why the size is 1 x 9800.<br>
The tensor I use has a single dimension with 192 elements and the output of the fist model layer is 512 elements.</p>
<p>This code I tried:</p>
<pre><code class="lang-auto"># Initial imports
from captum.attr import IntegratedGradients
from captum.attr import LayerConductance
from captum.attr import NeuronConductance

import torch
import torch.nn as nn

# Define model
class MLP(torch.nn.Module):
    def __init__(self):
        super().__init__()

        self.model = nn.Sequential(
            # Add input layer 
            nn.Linear(196, 512),
            # Add ReLU activation
            nn.ReLU(),
            # Add Another layer
            nn.Linear(512, 512),
            # Add ReLU activation
            nn.ReLU(),
            # Add Output layer
            nn.Linear(512, 12)
        )

    def forward(self, x):
        # Forward pass
        return self.model(x)

# Prepare data and model
sample = torch.rand(196)
label = 7
model  = MLP()

# Normal inference (works fine):
score = model(sample)
prob = nn.functional.softmax(score, dim=0)
y_pred =  prob.argmax()
print("Predicted class {} with probability {}. True label is: {}".format(y_pred, prob[y_pred], label))

# Usage of captum (does not work)
ig = IntegratedGradients(model)
sample.requires_grad_()
attr, delta = ig.attribute(sample,target=label, return_convergence_delta=True) # ERROR!
attr = attr.detach().numpy()
</code></pre>
<p>Has anyone an idea where the wrong dimension is coming from and how to solve this?</p>
<p>Thanks!</p>

brix: <p>I edited the first post and provided a full minimum working example.</p>
<p>Any help would be appreciated <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

googlebot: <p>I think you need to add a leading batch dimension to input. Additionally,  internal_batch_size argument may be needed to limit cuda memory requirements.</p>

brix: <p>Thank you very much!<br>
I added the line</p>
<blockquote>
<p>sample.unsqueeze_(0)</p>
</blockquote>
<p>before</p>
<blockquote>
<p>sample.requires_grad_()</p>
</blockquote>
<p>and not I am not getting the error anymore <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Loading Pretrained Weights

Hacking_Pirate: <p>Hi,<br>
I pretrained a custom model with targets 402. Now I need to transfer those weights to a model with 206 targets. How can I do that? This is what I am doing right now <code>    model.load_state_dict(torch.load(f"FOLD{fold}_.pth"), strict=False)</code><br>
But it is not working, showing a size mismatch error.</p>

Hacking_Pirate: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> do you have any idea?</p>

googlebot: <p>if you mean a subset of targets, something like that:</p>
<pre><code class="lang-auto">d = torch.load(f"FOLD{fold}_.pth")
for suffix in (".weight",".bias"):
  key = PREFIX + suffix
  d[key]=map_402_to_206(d[key])
model.load_state_dict(d)
</code></pre>
<p>PREFIX is output linear layer’s key. map_402_to_206 should reflect how you select a subset, simplest case would be a subrange: d[key] = d[key][:206]</p>
<p>if you want to reuse hidden layers on a different set of targets, delete above keys instead and use strict=False</p>

Hacking_Pirate: <p>Thanks for your response, I actually want to use for a different set of targets and I am using strict =False but it is throwing a size mismatch error.</p>

Hacking_Pirate: <pre><code class="lang-auto">    for i in range(len(model_source.layers[:-1])):
        model_dest.layers[i].set_weights(model_source.layers[i].get_weights())
    return model_dest
</code></pre>
<p>This is the Keras equivalent of what I want to do.</p>

ptrblck: <p>Did you follow <a class="mention" href="/u/googlebot">@googlebot</a>’s suggestion and deleted the key(s) before using <code>strict=False</code>?<br>
The <code>strict=False</code> argument ignores unexpected or missing keys, not shape mismatch errors.</p>

Hacking_Pirate: <p>How can I delete the keys?</p>

ptrblck: <p>You can delete key-value pairs from an <code>OrderedDict</code> using <code>del</code> as seen here:</p>
<pre><code class="lang-python">d = OrderedDict()
d['a'] = 1
d['b'] = 2
print(d)
del d['a']
print(d)
</code></pre>
<p>However, <a class="mention" href="/u/googlebot">@googlebot</a> already posted a solution where you would copy some of the pretrained parameters into the <code>state_dict</code>.</p>

Hacking_Pirate: <p>Thanks, bur what would <code>PREFIX</code> be?</p>

googlebot: <p>you can just explore python directory, e.g. list(d.keys()), it will contain [sub]module names as assigned in <code>__init__</code>, and deleting is something like: del d[“output_layer.weight”]</p>
Image output is different for matplotlib Imread and ImageFolder

szahan: <p>Hi<br>
I am reading  an image using both matplotlib imread and imagefolder dataloader.<br>
When plotting the output looks okay for imread but the one with imagefolder output is multiplied to 9 for one image though the shape looks okay (9 of the same images with different channels I suppose as its not color anymore).</p>
<p>TEST_DATA_PATH = “./data”<br>
TRANSFORM_IMG = transforms.Compose([<br>
transforms.ToTensor(),<br>
])</p>
<p>test_data = torchvision.datasets.ImageFolder(root=TEST_DATA_PATH, transform=TRANSFORM_IMG)<br>
test_data_loader  = data.DataLoader(test_data, batch_size=2, shuffle=False)</p>
<p>for i, data in enumerate(test_data_loader):<br>
images,labels = data<br>
plt.imshow(images[0])<br>
plt.show()</p>

RaLo4: <p>you sure the shape is ok?</p>
<aside class="quote no-group" data-username="szahan" data-post="1" data-topic="101472">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/szahan/48/39021_2.png" class="avatar"> szahan:</div>
<blockquote>
<p>TEST_DATA_PATH = “./data”<br>
TRANSFORM_IMG = transforms.Compose([<br>
transforms.ToTensor(),<br>
])</p>
<p>test_data = torchvision.datasets.ImageFolder(root=TEST_DATA_PATH, transform=TRANSFORM_IMG)<br>
test_data_loader = data.DataLoader(test_data, batch_size=2, shuffle=False)</p>
<p>for i, data in enumerate(test_data_loader):<br>
images,labels = data</p>
</blockquote>
</aside>
<p>this will give you an image of shape [C, H, W]</p>
<p>while <code>plt.imread(img_path)</code> will give you an image of shape [H, W, C]</p>
<p>If you want the image tensor you get from your datalaoder to be of the same shape, then you have to use <code>.permute(1, 2, 0)</code> on your tensor.<br>
On your example this could be done here:</p>
<pre><code class="lang-auto">plt.imshow(images[0].permute(1, 2, 0))
</code></pre>
<p>Please also keep in mind that <code>.ToTensor()</code> puts your image data into the range [0, 1] by dividing by 255</p>

szahan: <aside class="quote no-group" data-username="RaLo4" data-post="2" data-topic="101472">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/65b543/48.png" class="avatar"> RaLo4:</div>
<blockquote>
<p><code>.permute(1, 2, 0)</code></p>
</blockquote>
</aside>
<p>Thanks a lot. I was trying to change the shape using view/reshape but permute did the trick.</p>
Slow backward using nn.Conv2d as substitution for nn.Linear

dang-qi: <p>I am trying to implement Faster RCNN by myself. For roi_head, I used nn.Conv2d with 7*7 kernal as the substitution for nn.Linear. However, I found that the backward is very slow with nn.Conv2d. It took around 15s backward time with 100 iteration with nn.Conv2d, but it took 3.7s backward time for 100 iteration with nn.Linear. (The backward time is for whole faster rcnn network since it is hard to just time module backprop time).</p>
<p>Why is the time difference so big?</p>
<p>The two code are listed here</p>
<pre><code class="lang-auto">class FastRCNNHead(nn.Module):
    def __init__(self, cfg):
        super().__init__()
        self.class_num = cfg.class_num
        class_num = cfg.class_num
        self.cfg = cfg
        feature_num = 1024
        pool_h = cfg.roi_pool.pool_h
        pool_w = cfg.roi_pool.pool_w
        self.feature_head = nn.Sequential( 
            nn.Linear(cfg.out_feature_num*pool_h*pool_w, feature_num),
            nn.ReLU(inplace=True),
            nn.Linear(feature_num, feature_num),
            nn.ReLU(inplace=True)
        )
        self.label_head = nn.Linear(feature_num, class_num+1)
        self.bbox_head = nn.Linear(feature_num, 4*(class_num+1))

        for m in self.modules():
            if isinstance(m, nn.Linear):
                nn.init.normal_(m.weight, std=0.01)
                nn.init.constant_(m.bias, 0)

    def forward(self, rois ):

        rois = rois.flatten(start_dim=1)
        features = self.feature_head(rois)
        label_pre = self.label_head(features)
        bbox_pre = self.bbox_head(features)

        return label_pre, bbox_pre
</code></pre>
<pre><code class="lang-auto">class FastRCNNHeadConv(nn.Module):
    def __init__(self, cfg):
        super().__init__()
        self.class_num = cfg.class_num
        class_num = cfg.class_num
        self.cfg = cfg
        feature_num = 1024
        pool_h = cfg.roi_pool.pool_h
        pool_w = cfg.roi_pool.pool_w
        self.feature_head = nn.Sequential( 
            nn.Conv2d(in_channels=cfg.out_feature_num, out_channels=feature_num, kernel_size=(pool_h, pool_w), padding=0 ),
            nn.ReLU(inplace=True),
            nn.Conv2d(in_channels=feature_num, out_channels=feature_num, kernel_size=1, padding=0 ),
            nn.ReLU(inplace=True)
        )
        self.label_head = nn.Conv2d(in_channels=feature_num, out_channels=class_num+1, kernel_size=1, padding=0 )
        self.bbox_head = nn.Conv2d(in_channels=feature_num, out_channels=4*(class_num+1), kernel_size=1, padding=0 )

        for m in self.modules():
            if isinstance(m, nn.Conv2d):
                nn.init.normal_(m.weight, std=0.01)
                nn.init.constant_(m.bias, 0)

    def forward(self, rois ):

        features = self.feature_head(rois)
        label_pre = self.label_head(features)
        label_pre = label_pre.view(label_pre.shape[:2])
        bbox_pre = self.bbox_head(features)
        bbox_pre = bbox_pre.view(bbox_pre.shape[:2])

        return label_pre, bbox_pre
</code></pre>
Smaller network takes longer time per iteration? is this possible?

qcyza: <p>Hi, I am doing my training on two networks of different structures, one is much larger than another (have more parameters),  why the larger one actually takes much less time to train (time per iteration)? Is this possible? Or you would suggest that this might be bugs. Is there any way that I can verify that my architectures are implemented correctly?</p>
<p>Thanks all!</p>

ptrblck: <p>If you are using the GPU, did you synchronize the code before starting and stopping the timer?<br>
This would be necessary, since CUDA operations are asynchronous and thus non-blocking.</p>
<p>If so, what does small/large refer to? Is it the depth of the model or also the number of parameters?</p>

qcyza: <p>I will check the synchronization, thanks! I have torch.cuda.synchronize() at the beginning of the script. Is this what you refer to?</p>
<p>Even though the ‘time per iter’ might not be accurate here, it also takes a much longer time to train over the same number of epochs with the same batch size, the same dataset.</p>
<p>By the size of the model, I meant to be the number of parameters.</p>
<p>Thanks again!</p>

ptrblck: <aside class="quote no-group" data-username="qcyza" data-post="3" data-topic="101382">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/qcyza/48/30541_2.png" class="avatar"> qcyza:</div>
<blockquote>
<p>I have torch.cuda.synchronize() at the beginning of the script. Is this what you refer to?</p>
</blockquote>
</aside>
<p>Yes, but it would be needed before starting and stopping all timers.<br>
Otherwise your profile will accumulate the times in the next blocking operation and will yield wrong results.</p>
How to add `target labels` with `ImageFolder images` for `DataLoader tensor`

akib62: <p>I am loading some images from a folder using <code>ImageFolder</code> and then using <code>transform</code> to convert into <code>tensor</code>.</p>
<p>Then I load the labels from another <code>CSV</code> file and converted into <code>tensor</code>.</p>
<p>Finally, I want to combine these 2 tensors and want to use them for the <code>DataLoader</code>. But getting the error <code>AttributeError: 'ImageFolder' object has no attribute 'size'</code></p>
<p>Code</p>
<pre><code>train_data_dir = '/home/dataset/'
train_labels = pd.read_csv('/home/HousePrice.csv')
transform = transforms.Compose([
    transforms.Resize((256, 256)),
    transforms.ToTensor(),
    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

train_data_tensor = datasets.ImageFolder(train_data_dir, transform=transform)
train_labels_tensor = torch.tensor(train_labels.values)

train_tensor = TensorDataset(train_data_tensor, train_labels_tensor)
train_dataloader = torch.utils.data.DataLoader(train_tensor, batch_size=1, shuffle=True)

return train_dataloader
</code></pre>
<p>How can I add the <code>target labels</code> with my <code>images</code> and make a <code>single tensor</code> to use in the <code>DataLoader</code></p>

ptrblck: <p><code>ImageFolder</code> already creates the data-target mapping internally and loads each sample lazily in its <code>__getitem__</code>.<br>
I think the cleanest approach would be to write a custom <code>Dataset</code> by reusing parts of <a href="https://github.com/pytorch/vision/blob/6b071be9da221998c9ba4173fb5e4186dabdc9fe/torchvision/datasets/folder.py#L65"><code>DatasetFolder</code></a> and add your target tensor manually to your custom class.</p>
Replace SI-SNR loss with MSE loss

Mohamed_Nabih: <p>I have a loss function called SI-SNR loss function implemented as follows:</p>
<pre><code class="lang-auto">def si_snr_loss(ests, egs):
    refs = egs["ref"]
    num_spks = len(refs)

    def sisnr_loss(permute):
        return sum([sisnr(ests[s], refs[t]) for s, t in enumerate(permute)]) / len(permute)
    N = egs["mix"].size(0)
    sisnr_mat = torch.stack([sisnr_loss(p) for p in permutations(range(num_spks))])
    max_perutt,_ = torch.max(sisnr_mat, dim=0)
    return -torch.sum(max_perutt) / N
</code></pre>
<p>I want to replace this loss function with an MSE loss function I write a simple equation as follows:</p>
<p>for egs in val_dataloader:</p>
<pre><code>            current_step += 1

            egs = to_device(egs, self.device)

            ests = data_parallel(self.net, egs['mix'], device_ids=self.gpuid)
            #loss = si_snr_loss(ests, egs)
            loss = (ests - torch.Tensor(np.array(egs.values())))**2
            losses.append(loss.item())
</code></pre>
<p>Unfortunately, this gives me an error:<br>
TypeError: can’t convert np.ndarray of type numpy.object_. The only supported types are: float64, float32, float16, int64, int32, int16, int8, uint8, and bool.</p>

ptrblck: <p>The error seems to be raised by <code>numpy</code> in:</p>
<pre><code class="lang-python">np.array(egs.values())
</code></pre>
<p>Based on the error message it seems that <code>egs.values()</code> might return <code>objects</code> with variable shapes or any other objects which numpy cannot transform to an array.<br>
Could you print this object and check what is stored in it?</p>