akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Pytorch latest update(1.4) broke CosineAnnealingWarmRestarts: T_cur is not define

Liron_Mor_Yosef: <p>Hello,<br>
maybe i missing something, but my tests failed on  CosineAnnealingWarmRestarts creation,</p>
<p>the problem is:</p>
<pre><code>    super(CosineAnnealingWarmRestarts, self).__init__(optimizer, last_epoch)

    self.T_cur = self.last_epoch
</code></pre>
<p>the super class call the override step function that use self.T_cur that is not defined.</p>
<p>i open a bug report(<a href="https://github.com/pytorch/vision/issues/2119" rel="nofollow noopener">https://github.com/pytorch/vision/issues/2119</a>), but if i missed something please let me know<br>
thanks</p>

tom: <p>Two quick comments:</p>
<ul>
<li>
<p>Is this something regarding PyTorch itself or torchvision? As far as I can tell, the scheduler is defined in <code>torch.optim</code>, which would make it a PyTorch thing.</p>
</li>
<li>
<p>In any bug, it would be helpful if you provided code to produce the error.</p>
<ul>
<li>If you found a bug, it makes digging into it much easier.</li>
<li>If it is something with your particular use of the class - which may be uncommon yet something that should work, you avoid someone making up an example, saying “works for me” and move on.</li>
<li>If it is something where your use is not supported (i.e. bug in your code), it’ll help finding that, too.</li>
</ul>
<p>So it would seem to be a win in any case. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
</li>
</ul>
<p>Best regards</p>
<p>Thomas</p>

Liron_Mor_Yosef: <p>Hi Tom,<br>
you are right, it’s torch 1.4.</p>
<p>the reproduce step is easy, simply create the class and run step:<br>
from torch import optim<br>
from torchvision.models import resnet50<br>
from torch.optim import lr_scheduler</p>
<p>model = resnet50()<br>
optimizer = optim.SGD(model.parameters(), lr=0.05)<br>
for param_group_dict in optimizer.param_groups:<br>
param_group_dict[‘initial_lr’] = param_group_dict[‘lr’]<br>
lr_scheduler.CosineAnnealingWarmRestarts(optimizer=optimizer,  T_0=50, T_mult=1, eta_min=0, last_epoch=0)</p>
<p>thanks for your replay.</p>
<p>the use case can be found at <a href="https://discuss.pytorch.org/t/pytorch-latest-update-1-4-broke-multisteplr-wrong-lr-after-step-from-get-closed-form-lr/77514" class="inline-onebox">Pytorch latest update(1.4) broke MultiStepLR: wrong LR after step from _get_closed_form_lr</a></p>
Pytorch latest update(1.4) broke MultiStepLR: wrong LR after step from _get_closed_form_lr

Liron_Mor_Yosef: <p>Hi,<br>
the setting are simple,<br>
using simple SDG optimizer with start lr of 0.05, and using MultiStepLR lr scheduler with milestone of 2,4.</p>
<p>code for reproduce:<br>
#####################################<br>
from torch import optim<br>
from torchvision.models import resnet50<br>
from torch.optim import lr_scheduler</p>
<p>model = resnet50()<br>
optimizer = optim.SGD(model.parameters(), lr=0.05)<br>
ms_scheduler = lr_scheduler.MultiStepLR(optimizer=optimizer, milestones=[2,4])<br>
lrs = [param_group[‘lr’] for param_group in optimizer.param_groups]<br>
print(f"lr after lrs= {lrs}")<br>
ms_scheduler.step(epoch=0)<br>
lrs = [param_group[‘lr’] for param_group in optimizer.param_groups]<br>
print(f"lr after lrs= {lrs}")<br>
######################################</p>
<p>The bug happens if epoch is not None(i noted that that pytorch community tried to remove the option to pass epoch at step, but i will open a separate issue about that) at the  <strong>init</strong>  of _LRScheduler there is a call to self._get_closed_form_lr() and reset the value of the optimizer. In my case the excepted value in epoch 0 is 0.05 and not 0.005(it’s seems that he perform all the steps in the multistep in the init!!)</p>
<p>I see a code that handle this case when the epoch is None but buggy code for the other case.</p>
<p>I always pass the epoch when calling step( i have scheduler sequence wrapper that i wrote in order to change scheduler during training (most use cases are warm up and tear down lr scheduling ,but not only)</p>
How to show image from output of model (RNN)?

sornpraram: <p>I am learning RNN with pytorch from this github.</p>
<p><a href="https://github.com/yunjey/pytorch-tutorial/blob/master/tutorials/02-intermediate/recurrent_neural_network/main.py" rel="noopener nofollow ugc">Pytorch RNN Tutorial</a></p>
<p>I’m a little bit confused, because the code didn’t show result of the training. Only show the accuracy. Now I want to show image that output from model.</p>
<p>I try to save image from output, but the result I get is a compressed representation (below is the image I got from output of the model)<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/0/501fdb3b461321372dfc9576dd5b2bfc8833e69b.png" alt="de98" data-base62-sha1="bqOrSgrjnRiZs6tKdBh0VB9R5RF" width="10" height="100"></p>
<pre><code class="lang-auto"># Train the model
total_step = len(train_loader)
for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader):
        images = images.reshape(-1, sequence_length, input_size).to(device)
        labels = labels.to(device)

        # Forward pass
        outputs = model(images)
        loss = criterion(outputs, labels)

        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if (i+1) % 100 == 0:
            print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}' 
                    .format(epoch+1, num_epochs, i+1, total_step, loss.item()))
</code></pre>
<p>I am trying to show the image of the output by adding this cod</p>
<pre><code class="lang-auto">plt.imshow(outputs.reshape(28, 28))
</code></pre>
<p>But I got this error</p>
<pre><code class="lang-auto">RuntimeError: shape '[28, 28]' is invalid for input of size 1000
</code></pre>
<p>So, I assume that the output is 1000 size. but I want it to show as a mnist image (784 size (28*28)).</p>
Combining model weight

jaugust: <p>Hi All,</p>
<p>I have a question on combining model weights accurately.<br>
I made one NN and trained the model separately on two datasets. Now I am trying to obtain a single model out these two models by combining the weights.</p>
<p>What are different ways to combine the weights in anyone’s opinion?</p>
<p>Thanks in advance for answering this question</p>

ptrblck: <p>Hi Jees,</p>
<p>let me try to make sure I understand your use case completely.<br>
You have one model architecture and you initialized two models using it.<br>
These two model were trained on two different datasets and converged.</p>
<p>Now you would like to “combine” all parameters of both models and create a single one.<br>
Using this new single model you would now want to predict both datasets or just one of them?<br>
Do both datasets contain the same classes or are the targets completely different?</p>

jaugust: <p>Thanks a lot <a class="mention" href="/u/ptrblck">@ptrblck</a> for the queries.</p>
<p>You have one model architecture and you initialized two models using it.</p>
<blockquote>
<p>Yes that is very true</p>
</blockquote>
<p>These two model were trained on two different datasets and converged.</p>
<blockquote>
<p>that is also correct</p>
</blockquote>
<p>Now you would like to “combine” all parameters of both models and create a single one.</p>
<blockquote>
<p>that is the idea, yes</p>
</blockquote>
<p>Using this new single model you would now want to predict both datasets or just one of them?</p>
<blockquote>
<p>I want it to predict both of them with reasonable accuracy</p>
</blockquote>
<p>Do both datasets contain the same classes or are the targets completely different?</p>
<blockquote>
<p>completely different (we are not sure but let me consider that case)</p>
</blockquote>
<p>Thanks for queries I hope this can help you to give me some pointers.</p>

Dhirendra_Jha: <p>Hi,<br>
Can we club two weight file into one weight file without loss of catastrophic loss with data A and data B,<br>
In our case we trained initial some data and after some time I get some more new data. So I train only new data and club both weight file into one.<br>
Number of targets is same class and both data file in same domain.<br>
I club both model but I found catastrophic loss will be happen<br>
Please suggest me what is the best way</p>

ptrblck: <p>I don’t think combining different trained parameters will automatically result in a good new model, as both models might (and most likely) have converged to different local minima.<br>
The mean (I assume you are taking the average of all parameters) will not necessarily yield another minimum on the loss surface.</p>
<p>You could try to fine tune your model by retraining with the new samples and a low learning rate and check the validation accuracy again on the updated dataset.</p>
Pytorch not computing the right gradient for division

rzhangdl: <p>I’m trying to normalize an array(tensor) so that they sum to 1.</p>
<p>I did the following computation, however, the output gradient seems to be wrong. ( why are they all the same?? they should be different with respect to each element in  <code>a</code></p>
<p>code:</p>
<pre><code class="lang-auto">a = torch.rand(10, requires_grad=True)
print(a)
x = torch.sum(a)
y = a / x
y.backward(torch.ones_like(a))
pp(a.grad)
</code></pre>
<p>output:</p>
<pre><code class="lang-auto">tensor([0.6022, 0.6065, 0.0329, 0.3639, 0.2053, 0.1006, 0.6993, 0.1843, 0.0400,
        0.2939], requires_grad=True)
tensor([-5.9605e-08, -5.9605e-08, -5.9605e-08, -5.9605e-08, -5.9605e-08,
        -5.9605e-08, -5.9605e-08, -5.9605e-08, -5.9605e-08, -5.9605e-08])
</code></pre>

albanD: <p>Why do you expect them to be all different? You backward a Tensor full of <code>1</code>s.</p>

rzhangdl: <p>thanks for the quick reply!</p>
<p>Let’s say a = (a1, a2, a3)</p>
<p>then y = (y1, y2, y3) = (a1, a2, a3) / (a1 + a2 + a3).<br>
then d(y1) / d(a1) = (a2 + a3) / (a1 + a2 + a3) ^ 2</p>
<p>and similarly, d(y2) / d(a2) = (a1 + a3) / (a1 + a2 + a3) ^ 2</p>
<p>So they should be different rather than same?</p>

albanD: <p>Yes but this is not what you get here.<br>
Since you backward a Tensor full of 1, you get for the grad of a1: <code>1*d(y1)/d(a1) + 1*d(y3)/d(a1) + 1*d(y3)/d(a1)</code>.<br>
And I guess the same term will appear if you do the same for a2 and a3 hence the constant result that you see.</p>

rzhangdl: <p>I see, PyTorch is computing the jacobian matrix… my bad my bad.</p>
<p>Thanks for the help!</p>
5-fold Cross Validation

sampa: <pre><code class="lang-auto">merge_data = datasets.ImageFolder(data_dir + "/train", transform=train_transforms)
fold_counts= 5
kfold = KFold(n_splits=fold_counts, random_state=777, shuffle=True)
num_workers = 0
#
    #--------------------------------------------------------------
for i, (train_index, validate_index) in enumerate(kfold.split(merge_data)):
    #print("train index:", train_index, "validate index:", validate_index)
    train = torch.utils.data.Subset(merge_data, train_index)
    validation = torch.utils.data.Subset(merge_data, validate_index)
    train_loader = torch.utils.data.DataLoader(train, batch_size=batch_size, shuffle=True, num_workers=num_workers, pin_memory=False)
    valid_loader = torch.utils.data.DataLoader(validation, batch_size=batch_size, shuffle=True, num_workers=num_workers, pin_memory=False)
        
    print("Number of Samples in Train: ",len(train))
    print("Number of Samples in Valid: ",len(validation))
    model, train_loss, valid_loss, train_acc, valid_acc=train_model(model, train_loader, valid_loader, criterion, optimizer, num_epochs, patience)
</code></pre>
<p>Is it possible to get 5 models using 5 validation sets?</p>
In sound event detection, how to augment synthetic labels?

kmh7576: <p>Hi, I’m trying to participate in DCASE 2020 Task4, a sound event detection.<br>
And while modifying the baseline code, I could find augmentation in training process.</p>
<p>As you know, synthetic data have event label and onset, offset time.<br>
So, when I augment data with time transform, I should adjust onset, offset labels to fit this.</p>
<p>I’m stuck in here.<br>
So my question is:</p>
<ol>
<li>
<p>When data have been augmented and have two data in tuple, how to modify labels?</p>
</li>
<li>
<p>When tuple have multi-data features, how pytorch link them with labels?<br>
(ex. Data in Tuple : 4, label : 2)</p>
</li>
</ol>
<p>Thank you for your help in advance.</p>
Operations on gradient before summing in autograd

pheonixashes11: <p>Hello Everyone,</p>
<p>From the documentation of pytorch, I come to understand that torch.autograd.grad(Y, X, G) computes the following:  sum of partial derivative of ‘Y’ wrt ‘X’ multiplied by G.</p>
<p>I would like to do the following instead:<br>
sum of (partial derivative of ‘Y’ wrt ‘X’)^2 multiplied by G.</p>
<p>Is this possible to do in pytorch? Do we have to override the torch.autograd.grad? Or is there another way?<br>
A crude way may be to do torch.autograd.grad(Y[i,j], X, grad_outputs=torch.ones_like(Y[i,j]) ), then square each entry of the this result and multiply with G. But that is too much computationally expensive.</p>

albanD: <p>Hi,</p>
<p>Actually the sum is not there, it’s a matrix product: G * J where J is the Jacobian of the function that given X produced Y.<br>
Unfortunately, this is a limitation of automatic differentiation itself, not pytorch. So you won’t be able to change the jacobian before doing the multiplication with G.</p>

pheonixashes11: <p>Ok. In that case, can we then get the Jacobian J by using G = I, the Identity matrix for 2D output Y?</p>

albanD: <p>I’m afraid it’s not possible. This notation with matrix multiply and 2D Jacobian works if you consider 1D inputs and 1D outputs.<br>
AD only performs vector jacobian product. You need special function if you want to be able to do this.</p>
<p>If you are using the latest version of pytorch, you can find such function <a href="https://pytorch.org/docs/master/autograd.html#functional-higher-level-api">here</a>.</p>

pheonixashes11: <p>As a workaround I get the Jacobian for each index of Y. This is computationally intensive. But does the work for now.</p>
<pre><code class="lang-auto">def squared_grad(self, Y, X, G):
        result = torch.zeros_like(X)
        g = torch.zeros_like(G)
        g_np = g.data.cpu().detach().numpy()
        gout = torch.ones(1).cuda()[0]
        for index, e in np.ndenumerate(g_np):
            result += torch.autograd.grad(Y[index], X, grad_outputs=gout, retain_graph=True)[0]**2*G[index]
        return result
</code></pre>
<p>Any comments?</p>
<p>Thanks.</p>
Problem c10/utils/variant.h

alepack: <p>Hi</p>
<p>Maybe I’m doing something wrong but I’ve got a problem with libtorch 1.4.0 (I’m using the pre-built binaries you can find at <a href="http://pytorch.com" rel="nofollow noopener">pytorch.com</a> - Ubuntu 16.04 with gcc-5.4)<br>
The code I’m trying to build is the following one</p>
<pre><code class="lang-auto">//Include standard libraries
#include &lt;iostream&gt;

//Define namespace data
//This can be included for example by another header file
namespace data
{
}

//Include torch library
#include &lt;torch/torch.h&gt;

int main(int argc, char* argv[])
{
  std::cout &lt;&lt; torch::randn({3, 3}) &lt;&lt; std::endl;
  return 0;
}
</code></pre>
<p>The CMakeLists.txt is</p>
<pre><code class="lang-auto">cmake_minimum_required(VERSION 3.0 FATAL_ERROR)
project(test_variant)

find_package(Torch REQUIRED)

add_executable(test_variant main.cpp)
target_link_libraries(test_variant "${TORCH_LIBRARIES}")
set_property(TARGET test_variant PROPERTY CXX_STANDARD 14)
</code></pre>
<p>I’ve added an empty “data” namespace to better explain the problem.<br>
When I build the example I get a lot of errors like</p>
<pre><code class="lang-auto">~/test_variant/libtorch/include/c10/util/variant.h:2519:7: error: invalid use of ‘void’
       AUTO_RETURN(v &amp;&amp; holds_alternative&lt;I&gt;(*v)

</code></pre>
<p>I’ve looked inside the file and it seems that the problem is the statement (line 1165 c10/utils/variant.h)</p>
<pre><code class="lang-auto">AUTO_REFREF_RETURN(recursive_union::get_alt(
              data(lib::forward&lt;V&gt;(v)), in_place_index_t&lt;I&gt;{}))
</code></pre>
<p>Where “data” is incorrectly detected as the namespace I’ve added before the &lt;torch/torch.h&gt; inclusion.<br>
I think it should refer to the “data” function which can be found  in the same file at line 1743 (I tried to change its name and indeed the error is solved).<br>
Could this be an issue or it’s something I’m doing wrong in my code?</p>
<p>Thank you very much for your help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>Thanks for the code snippet and debugging.<br>
Could you additionally create an issue on <a href="https://github.com/pytorch/pytorch/issues">GitHub</a> to discuss this error?</p>

alepack: <p>Sure, I’ll do that</p>
<p>Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Multiprocessing for inference

divyesh_rajpura: <p>I am using simple BLSTM architecture with 3 layers and I am trying to use multiprocessing.Pool to make inference faster. But it is not working. Below is the snippet of the code. Please suggest solution.</p>
<pre><code class="lang-auto">def do_enroll_step(batch_index, batch_data, model):
    model.eval()
    batch_feature, _, _, batch_spkid = batch_data
    with torch.no_grad():
        batch_feature = batch_feature.contiguous()
        hidden = model.init_hidden(len(batch_feature))
        batch_output, _ = model(batch_feature, hidden)
        
        with h5py.File(enroll_result_file, 'a') as f:
            f.create_dataset(batch_spkid[0], data = batch_output)

def do_enroll():
    starttime = time.time()
    num_processes = 5
    pool = mp.Pool(processes = num_processes)
    pool.starmap_async(do_enroll_step, [(batch_index, batch_data, model) for batch_index, batch_data in enumerate(enroll_loader)])
    pool.close()
    pool.join()
    print('That took {} seconds'.format(time.time() - starttime))
</code></pre>

divyesh_rajpura: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, can you please help me with this?</p>
Error in looping through the data_path

saty_101: <p>I am trying to load in MNIST dataset and although I know that there are many different ways to do so, I am having trouble in loading the dataset via the following method -</p>
<pre><code class="lang-auto">#image_location is already there in the variable path_digit[]
for i in range(10):
  print("check")
  stacked_training[i] = torch.stack([tensor(Image.open(o)) for o in (path_digit[i])]).float()/255
  print(stacked_training[i].shape, '\n')
</code></pre>
<p>below is the error code -</p>
<pre><code class="lang-auto">check
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-10-bb2269508aed&gt; in &lt;module&gt;()
----&gt; 4   stacked_training[i] = torch.stack([tensor(Image.open(o)) for o in (path_digit[i])]).float()/255
      5   print(stacked_training[i].shape, '\n')

RuntimeError: expand(torch.FloatTensor{[5923, 28, 28]}, size=[28, 28]): the number of sizes provided (2) must be greater or equal to the number of dimensions in the tensor (3)
</code></pre>

ptrblck: <p>How did you define <code>path_digit</code>?<br>
The <code>torchvision.datasets.MNIST</code> dataset uses the internal <code>data</code> attribute to load all images from the downloaded binary files, so I’m not sure how you extracted the paths.</p>
<p>EDIT: Seems to be solved in <a href="https://discuss.pytorch.org/t/workaround-for-appending-a-tensor-list/77449/2">this topic</a>.</p>

saty_101: <p>Yes this has been solved. Sorry for not editing this post.</p>
Expected object of backend CUDA but got backend CPU for argument #2 'mat2'

PrudviRaj_J: <p>I encountered this error for the below code snippet.</p>
<p>def <strong>init</strong>(self, original_dim, projection_dim, n_modalities=2):<br>
super().<strong>init</strong>()</p>
<pre><code>    self.C = []
    self.n_modalities = n_modalities

    for _ in range(n_modalities):
        # C tensor performs the mapping of the h vector and stores the s vector values as well
        C = torch.zeros(original_dim, projection_dim)
        for i in range(original_dim):
            C[i, np.random.randint(0, projection_dim-1)] = 2 * \
                np.random.randint(0, 2) - 1  # s values

        self.C.append(C)

def to(self, device):
    for i in range(len(self.C)):
        self.C[i].to(device)

def forward(self, *x):
    feature_size = x[0].size()
    y = [0]*self.n_modalities

    for i, d in enumerate(x):
        y[i] = d.mm(self.C[i]).view(feature_size[0], -1) # Error line</code></pre>

ptrblck: <p>If <code>self.C[i]</code> returns a tensor, you would have to reassign the result of <code>to</code>:</p>
<pre><code class="lang-python">self.C[i] = self.C[i].to(device)
</code></pre>

PrudviRaj_J: <p>Thank you for quick response,</p>
<p>model is working for below snippet but is it appropriate to define “device” in forward block?</p>
<p>def forward(self, *x):<br>
device = torch.device (“cuda” if torch.cuda.is_available() else “cpu”)<br>
feature_size = x[0].size()<br>
y = [0]*self.n_modalities</p>
<pre><code>for i, d in enumerate(x):
self.C[i] = self.C[i].to(device)
    y[i] = d.mm(self.C[i]).view(feature_size[0], -1)</code></pre>

ptrblck: <p>If you need to create a tensor in the <code>forward</code> method, I would recommend to use the <code>device</code> attribute of any parameter of the model. Otherwise you might run into problems e.g. if you are using data parallel.</p>
<p>However, in your approach you are pushing each tensor in <code>self.C</code> to the same device in each forward pass, which seems not to be necessary, as you could also do it in the <code>__init__</code> method once.<br>
Is <code>self.C</code> containing tensors with a variable shapes? If not, you could just use <code>torch.stack</code> to create a single tensor.</p>
How could I use sync-bn correctly?

coincheung: <p>Hi,</p>
<p>Do I need to convert each batchnorm layer to its sync-bn version or Do I only need to convert the model?<br>
I mean:</p>
<pre><code class="lang-python">class Model(nn.Module):
def __init__(self):
     self.conv = nn.Conv2d(...)
     self.bn = nn.SyncBatchNorm.convert_sync_batchnorm(nn.BatchNorm2d(...))
...
model = Model()
</code></pre>
<p>Or this:</p>
<pre><code class="lang-python">class Model(nn.Module):
def __init__(self):
     self.conv = nn.Conv2d(...)
     self.bn = nn.BatchNorm2d(...)
...
     model = nn.SyncBatchNorm.convert_sync_batchnorm(Model())
</code></pre>
<p>Which one is the correct usage ? Could I simply call the convert function on the whole model ?</p>

ptrblck: <p>The <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.SyncBatchNorm.convert_sync_batchnorm">docs</a> show an example, where it’s called on the complete module, which should work.<br>
Are you seeing any issues in this approach?</p>

coincheung: <p>Yes, I found that the training becomes quite slow and the converging time gets longer. As for the final results, the second method is worse than the first method in my experiments.</p>

coincheung: <p>I have figured out my problem, it has nothing to do with the way of using convert_sync_bn. The solution is that if I use apex, I should use convert_sync_bn before initializing the amp. There will be problem if I use convert_sync_bn after it.</p>
How to tackle the class imbalance problem during training in PyTorch

spaul13: <p>Can someone please tell me if I have a different number of images per class during training on how to tackle the class imbalance problem in PyTorch?</p>
<p>How to solve this “class imbalance problem” in PyTorch?</p>

ptrblck: <p>You could over/undersample the classes using a <code>WeightedRandomSampler</code>.<br>
Have a look at <a href="https://discuss.pytorch.org/t/dataloader-using-subsetrandomsampler-and-weightedrandomsampler-at-the-same-time/29907/2">this post</a> for a small example.</p>
How to increase a loss

lepoeme20: <p>Hi,</p>
<p>I want to increase the specific loss while learning the model.</p>
<p>I have the following code:</p>
<pre><code class="lang-auto">criterion_MSE = nn.MSELoss()
criterion_CE = nn.CrossEntropyLoss()

...
loss_MSE = criterion_MSE(predict1, target1)

...
loss_CE = criterion_CE(predict2, target2)

total_loss = loss_MSE + loss_CE

total_loss.backward()
optimizer.step()
</code></pre>
<p>If I want to maximize the loss_CE here (with minimize loss_MSE), just multiply the loss_CE by negative one?<br>
If not, please tell me how I can do it.</p>

simaiden: <p>Yes, remember that you are solving a optimization problem, so <code>max f(x) &lt;=&gt; min -f(x)</code></p>
RuntimeError: Calculated padded input size per channel: (2 x 18). Kernel size: (3 x 3). Kernel size can't be greater than actual input size

urban_fantast: <p>I was trying to implement the NVDIA’s paper on self-driving cars and had the following error<br>
RuntimeError: Calculated padded input size per channel: (2 x 18). Kernel size: (3 x 3). Kernel size can’t be greater than actual input size</p>
<p>My conv_layer is as follows:<br>
self.conv_layers = nn.Sequential(<br>
nn.Conv2d(3, 24, 5, stride=2),<br>
nn.ELU(),<br>
nn.Conv2d(24, 36, 5, stride=2),<br>
nn.ELU(),<br>
nn.Conv2d(36, 48, 5, stride=2),<br>
nn.ELU(),<br>
nn.Conv2d(48, 64, 3, stride=2),<br>
nn.ELU(),<br>
nn.Conv2d(64, 64, 3, stride=2),<br>
nn.MaxPool2d((1,3)),<br>
nn.Dropout(0.5)<br>
)</p>
<p>and the input.size() to this layer is torch.Size([32, 3, 70, 320]).<br>
I am trying to understand how the the the size of ([32,3,70,320]) gets converted to a 3@66*220 input layer to it<br>
Help me out</p>

ptrblck: <p>Your input size is just too small for this model.<br>
I’ve added the output shapes of the conv layers as comments:</p>
<pre><code class="lang-python">conv_layers = nn.Sequential(
nn.Conv2d(3, 24, 5, stride=2), # [N, 24, 33, 158]
nn.ELU(),
nn.Conv2d(24, 36, 5, stride=2), # [N, 36, 15, 77]
nn.ELU(),
nn.Conv2d(36, 48, 5, stride=2), # [N, 48, 6, 37]
nn.ELU(),
nn.Conv2d(48, 64, 3, stride=2), # [N, 64, 2, 18]
...
</code></pre>
<p>As you can see, the spatial size if the last mentioned layer is <code>2x18</code>, which is too small for a kernel size of 3 with stride=2.</p>

mathematics: <p><del>This was happening to me in pretrained model, As above I had used small input size for inception_v3,<br>
After i resize image to (229, 229)  as mention on docs . oh It didn’t worked. Help!!<br>
Throwing same error<br>
<code>RuntimeError: Calculated padded input size per channel: (3 x 3). Kernel size: (5 x 5). Kernel size can't be greater than actual input size</code></del></p>
<p>problem was solved <a href="https://discuss.pytorch.org/t/error-in-training-inception-v3/23933">inception v3 calculated padded input…</a></p>
Detectron2 maskrcnn implementation question

jackck: <p>I am studying Detectron2 implementation of Mask-RCNN.  I am trying to understand why there is a max( , 0.1) needed in <a href="https://github.com/overjetdental/detectron2/blob/master/detectron2/structures/masks.py#L66" rel="nofollow noopener">masks.rasterize_polygons_within_box</a></p>
<pre><code class="lang-auto">    # 2. Rescale the polygons to the new box size
    ratio_h = mask_size / max(h, 0.1)
    ratio_w = mask_size / max(w, 0.1)
</code></pre>
<p>Here, h and w are the height and width of the box. Where does the number 0.1 come from?  Is that just to prevent a divide by zero error?<br>
Thanks.</p>

ppwwyyxx: <p>Yes, it’s to prevent division by a too small number.<br>
0.1 is quite arbitrarily set. Since (h, w) means the size of the object, if it’s &lt; 0.1 it wouldn’t be meaningful anyway.</p>
Relation between t-SNE and Overfitting

davidlee: <p>Hi guys,</p>
<p>Is there any overfitting related with t-SNE visualization?<br>
For example, If I get more divided and separate map from t-SNE than the previous,<br>
can I say that it’s more overfitted than the previous?</p>
<p>I hope you answer any thoughts about it!</p>
<p>Thanks.</p>

ptrblck: <p>I think while a t-SNE plot might give you some information about the class separation, I think the training vs. validation losses should be better used to claim less overfitting.<br>
t-SNE might output a lot of “good” and “bad” visualizations in my experience depending on the arguments you are passing to it, so I wouldn’t rely on its outcome regarding overfitting.</p>

davidlee: <p>Hi ptrblck,<br>
Thank you for your reply. I’ve been always motivated from your answers.</p>
<p>To make it clear, I have one more question.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/f/5fdb7d4343fe6c1627cba3375b8636e0a8ab87d2.png" alt="image" data-base62-sha1="dFZC1MaEsA557EO6wIrqQoTEmKC" width="508" height="221"></p>
<p>If there are two t-sne plots from two different models trained individually,<br>
and I cannot say that which one is more generalizable?<br>
I know the right one is more trained and yields better accuracy but,<br>
if the features are more entangled, is there any chance to be more overfitted?</p>

ptrblck: <aside class="quote no-group" data-username="davidlee" data-post="3" data-topic="77471">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/davidlee/48/12812_2.png" class="avatar"> davidlee:</div>
<blockquote>
<p>I know the right one is more trained and yields better accuracy but,<br>
if the features are more entangled, is there any chance to be more overfitted?</p>
</blockquote>
</aside>
<p>I think your general statement is true.<br>
If the features (in the high dimensional feature space) are more entangled, then the model would have a harder time to separate the classes.<br>
My only concern is how to capture this “entanglement”.<br>
While t-SNE can certainly create your nice plots, I’m always skeptical about the chosen parameters for the t-SNE call.</p>
<p>If you are using <a href="https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html"><code>sklearn.manifold.TSNE</code></a>, are you using the default arguments and are other arguments always yielding a comparable output for both models?</p>
<p>That being said, I don’t want to discourage you using t-SNE, as it’s a great technique, and would recommend to also use the losses to describe the overfitting of a model.</p>

davidlee: <p>I totally agree with you.<br>
It yields sometimes good and sometimes bad plots, while changing the arguments.<br>
I think I just refer to the distribution approximation, but not the exact criteria about like generalization or something.<br>
Thank you for your time <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Is it possible to save a config dict along with the model parameters?

shamoons: <p>I am doing:</p>
<pre><code class="lang-auto">            torch.save(reconstruct_model.state_dict(), path.join(wandb.run.dir, 'best-model.pt'))
</code></pre>
<p>which saves my model parameters. Separately, I’m doing:</p>
<pre><code class="lang-auto">    open(path.join(wandb.run.dir, 'args.json'), 'w').write(json.dumps(vars(args)))
</code></pre>
<p>I’m wondering if I can just save those <code>args</code> in the <code>best-model.pt</code> file? Subsequently, when I load, currently, I do:</p>
<pre><code class="lang-auto">    spec = importlib.util.spec_from_file_location(
        "ReconstructionModel", saved_reconstruction_model_filepath)
    model_spec = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(model_spec)

    model = model_spec.ReconstructionModel(kernel_size=masked_args['kernel_size'], kernel_size_step=masked_args['kernel_size_step'])

    state_dict = torch.load(model_path, map_location=device)

    model.load_state_dict(state_dict)
</code></pre>
<p>where the <code>masked_args</code> are read from the <code>args.json</code> file.</p>

JuanFMontesinos: <p>You can save objects in general.<br>
You just need to create a dict with the parameters together with you dictionary.<br>
Like</p>
<pre><code class="lang-auto">torch.save({'model':model.state_dict(),'my_params':...}
</code></pre>

shamoons: <p>What about reading the params?</p>

JuanFMontesinos: <p>you just need to read it as if it were a dictionary.<br>
Your params will be in torch.load(path)[‘my_params’]<br>
and the model in model.load_state_dict(tocrch.load(path)[‘model’])</p>
How to make CUDA automatically find required data piece from matrix in CPU side, then only feed these data into CUDA

themoonboy: <p>When using CUDA device on pytorch, we normally need to send whole piece of data block (like matrix) to CUDA side, but when a matrix is very large, it would exceed CUDA’s capacity and we have to use torch.sparse methods to handle with it.</p>
<p>Now the thinking is, besides the sparse format, is there any pytorch functionalities that could automatically find which piece (row, column) of data from a matrix, and then only send these pieces into CUDA? Even though the original matrix is locating in CPU side.</p>
<p>For example, we are calculating torch.mm(A, B), where both A and B are extremely large and in dense format. Each time when multiple a row in A and a col in B, the expected pytorch functionalities could automatically find now who is in calculating and send them into CUDA. After getting some results, send them back to CPU and then turn head to next pieces of data who need CUDA travel.</p>
<p>The overall efficiency maybe low since it has frequent CUDA-CPU interaction, but it may worthy to do that for some very complex operations. Thus, is there any pytorch methods support to do such an automatic travel?</p>
Pytorch cross-compilation

Peter_Peter: <p>Hello,</p>
<p>I’m looking for to cross-compile PyTorch for an embedded device (aka RPi), please is there information and procedure to cross-compile PyTorch?</p>

tom: <p>As far as I know, there isn’t per se and I for one compile PyTorch on my RPi4, even if it takes quite a while. You could probably use distcc to offload the compilation to somewhere faster.<br>
That said, the Android compilation works as a cross-compilation, so libtorch can be cross-compiled and it probably is a question of adapting some cmake bits to do the right thing.</p>
<p>Best regards</p>
<p>Thomas</p>
Is there a way to compute matrix trace in batch broadcast fashion?

Melissa: <p>For example, I have a tensor of size(batch_size, hidden_size, hidden_size). Is there a way to compute the trace of matrix(hidden_size, hidden_size) for every sample in this batch without a loop? So that the output tensor is a vector of size(batch_size).</p>
<p>Any tips will help! Thanks!</p>

jmaronas: <p>Did you find a solution? (apart from looping) I was looking for a way to do this operation but it seems there is no bathed trace function</p>

tom: <p>There are several ways to get this easily, for example</p>
<pre><code class="lang-python">b = torch.einsum('bii-&gt;b', a)
</code></pre>
<p>or</p>
<pre><code class="lang-python">b2 = torch.diagonal(a, dim1=-2, dim2=-1).sum(-1) 
</code></pre>
<p>Best regards</p>
<p>Thomas</p>
Writer.add_scalars error

zahra: <p>Hi,</p>
<pre><code class="lang-auto"> tmp = correct / total
 tmp = torch.as_tensor(tmp)
 writer.add_scalars('accuracy/' + label,
                              {'test': tmp},
                               global_step=(epoch + 1) * len(trainloader))
</code></pre>
<p>I got this error:</p>
<pre><code class="lang-auto">TypeError: add(): argument 'other' (position 1) must be Tensor, not str
</code></pre>
<p>While</p>
<pre><code class="lang-auto">&gt;&gt;tmp.type()
'torch.FloatTensor'
&gt;&gt;label.type()
'torch.LongTensor'
</code></pre>
<p>Would you have any advices?</p>
<p>Thanks</p>

zahra: <p>Hi,<br>
No body have any ideas?</p>

ptrblck: <p>Could you post a small executable code snippet, which yields this error, pleas?</p>

zahra: <p>Thanks for your reply.<br>
Fortunately, I found what caused this error. I redefined label as  a string in a place of code.</p>
Workaround for appending a tensor list

saty_101: <p>I am trying to append a tensor list with the pixel values of MNIST dataset. However, I am unable to concatenate all the values of the pixels by a single loop. Is there a workaround this problem?</p>
<pre><code class="lang-auto">for i in range(10):
  stacked_training = torch.stack([tensor(Image.open(o)) for o in (path_digit[i])]).float()/255
  train_x = torch.cat([stacked_training]).view(-1, 28*28)
  train_y_temp = tensor([i]*len(stacked_training))
  train_y = torch.cat([train_y_temp])
  print(f"Shape of training dataset for {i} is :", len(stacked_training))
</code></pre>
<pre><code class="lang-auto">Shape of training dataset for 0 is : 5923
Shape of training dataset for 1 is : 6742
Shape of training dataset for 2 is : 5958
Shape of training dataset for 3 is : 6131
Shape of training dataset for 4 is : 5842
Shape of training dataset for 5 is : 5421
Shape of training dataset for 6 is : 5918
Shape of training dataset for 7 is : 6265
Shape of training dataset for 8 is : 5851
Shape of training dataset for 9 is : 5949
</code></pre>
<p>As you can see in my code only the last loop values will be contained in the train_x and train_y. Is there a way to append the values of each loop or any other workaround this problem?</p>

saty_101: <p>I was able to solve this problem by using this code, putting the code here if anyone else is stuck.</p>
<pre><code class="lang-auto">train_x = torch.cat([torch.stack([tensor(Image.open(o)) for o in ((path_digits[i])]).float()/255 for i in range(10)])
</code></pre>
Comparison between deep Auto-Encoder (stacked) and RBM training?

rostom007: <p>Hi.</p>
<p>im trying to implement a recommander system using RBM and deep auto-encoder architectures separately with pytorch. After the training i got a better performance result in RBM over AE.however,i have seen many papers and studies which said that the AE is always better than RBM .i used the same dataset for both architectures. and im trying to do a comparison between these two methods of deep learning.</p>
<p>My question is why did i got a different results than excpected? should i change something or it is ok ?</p>
Effect of normalizing the output

satinder147: <p>I have been working on a binary segmentation task for months. I tried everything. My architecture was able to overfit a small dataset but never could converge on  the larger one. I even tried a pretrained encoder, but it also didn’t work, tried tuning other hyper-paramters, used learning rate schedulers and they also didn’t work. Today I tried Normalizing the output with mean 0 and std 0.5 and it worked although normalizing with mean 0.5 and std 0.5 does not work and the loss starts diverging. I wanted to ask why  normalizing the output has such a drastic effect and how can I find these value in future for other problems.</p>

googlebot: <p>That’s because all usual weight initializers are trying to preserve mean &amp; variance, but it is only possible if input moments are known - so they assume normalized inputs (0,1). With input mean != 0, depending on layer and weight shape, layer output’s mean can be pushed far away from zero, variance may increase, therefore you can get a bunch of well-known problems (saturation, flat loss region, etc)</p>
<p>If you mean that final output normalization helped you, this again means that before normalization network outputs were in a bad region (i.e. mean 100)</p>

satinder147: <p>What can I do so that these problems occur less often and how can I know which value of mean and standard deviation would work?</p>

googlebot: <p>hidden layers usually work best with mean=0,sd=1, final output shouldn’t be normalized (at least, not without affine=True) as your loss target should dictate best mean.</p>

satinder147: <p>What about the pixel values of the output, should they be in range 0 to 1 or -1 to 1</p>

googlebot: <p>inside network, you have pixels-&gt;features-&gt;features-&gt;predictors pipeline, so no reason to use 0…1 range</p>
Load weight from hdf5 file

WERush: <p>I have a weight file whose format is hdf5. I want to load the weight parameters in hdf5 into my model.</p>
<p>How can i make it happen in pytorch?</p>

fmassa: <p>Have a look at <a href="https://github.com/szagoruyko/functional-zoo/blob/master/resnet-18-export.ipynb">https://github.com/szagoruyko/functional-zoo/blob/master/resnet-18-export.ipynb</a><br>
<code>hickle</code> is actually an interface for <code>hdf5</code>, so the example is actually loading a hdf5 file (but using the functional interface of pytorch, but the same thing can be done for models defined using <code>nn.Module</code>)</p>

Rao_Shivansh: <p>Could you please tell me how to load weights to hdf5 file.<br>
I am quite new to this.<br>
Thanks !</p>

thida: <p>What if, reversely, I want to export a .h5 file out of an existing CNN constructed with pytorch?<br>
Is there any trick for this please?</p>
Get indices of repeated columns

MauroPfister: <p>Hi everyone,</p>
<p>I stumbled over the following problem that I couldn’t solve myself:</p>
<p>Given a 4 x n tensor <code>A</code> which might contain non-unique columns, I’m looking for a way to get the column index of a single (could be the first, but does not have to) occurrence of each column.</p>
<p>Fore example given the tensor <code>A</code> (note that column 1 and 5 are identical)</p>
<pre><code class="lang-auto">tensor([[10., 11., 11., 11., 11., 11., 13., 13., 14., 14.],
        [ 0.,  2.,  0.,  1.,  1.,  2.,  1.,  1.,  0.,  0.],
        [ 3.,  6.,  6.,  2.,  6.,  6.,  9.,  3.,  4.,  7.],
        [ 2.,  3., 10., 10., 10.,  3.,  5.,  6.,  6.,  9.]])
</code></pre>
<p>I would like to obtain the following indices</p>
<pre><code class="lang-auto">tensor([[ 0.,  1.,  2.,  3.,  4.,  6.,  7.,  8.,  9.]])  # Index of 1st occurrence
# or
tensor([[ 0.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9.]])  # Index of 2nd occurrence
</code></pre>
<p>If getting the indices of the columns is not possible, a mask would also do:</p>
<pre><code class="lang-auto">tensor([[ 1.,  1.,  1.,  1.,  1.,  0.,  1.,  1.,  1., 1.]])  # Masking 2nd occurrence
# or 
tensor([[ 1.,  0.,  1.,  1.,  1.,  1.,  1.,  1.,  1., 1.]])  # Masking 1st occurrence
</code></pre>
<p>I hope it is somewhat clear what I’m trying to achieve. Maybe I’m over-complicating things and there is an easier way around this (see background information below). In any case help is really appreciated =)</p>
<p>Cheers!</p>
<p><em>Background information</em><br>
The columns of the tensor <code>A</code> above represents the locations of targets in a 4D grid. The targets themselves are stored in a tensor of the same length of <code>A</code>. When constructing my loss, I need to make sure, that there is only one target in each grid cell. My idea is to search the tensor <code>A</code> for non-unique columns and then only keep the column index of a single occurrence. Using this list of indices I could then select the valid targets.</p>
Accessing a GPU tensor via different processes?

Muppet: <p>Hi, is it possible to access a tensor on the GPU from different processes? Passing via mp.Queue() is not possible. I know that model weights can be shared, but can I specify a tensor at the beginning and share it across all processes? I’d like to populate the GPU tensor from one process, and access it from another (and DataLoader is not an option for my situation).</p>
<p>Thank you</p>
Dataset out of range

Eta_C: <p>The following code can reproduce this error.</p>
<pre><code class="lang-python">import torch
from torch.utils.data import Dataset
import numpy as np


class TEST1(Dataset):
    def __init__(self):
        super().__init__()
        self.dataset = None
        self.data_len = 3
    
    def __len__(self):
        return self.data_len
    
    def __getitem__(self, idx):
        if self.dataset is None:
            self.dataset = np.random.rand(3, 4)
        return torch.tensor(idx)  # here i return the index
    
    
class TEST2(Dataset):
    def __init__(self):
        super().__init__()
        self.dataset = None
        self.data_len = 3
    
    def __len__(self):
        return self.data_len
    
    def __getitem__(self, idx):
        if self.dataset is None:
            self.dataset = np.random.rand(3, 4)
        return torch.tensor(self.dataset[idx])  # here i return the data
    

test1 = TEST1()
test2 = TEST2()
for i, x in enumerate(test1):
    if i &gt; 5:
        break
    print("test1", i, x, "len = ", len(test1))
for i, x in enumerate(test2):
    if i &gt; 5:
        break
    print("test2", i, x, "len = ", len(test2))
</code></pre>
<p>Then we can see that the output is</p>
<pre><code class="lang-python">test1 0 tensor(0) len =  3
test1 1 tensor(1) len =  3
test1 2 tensor(2) len =  3
test1 3 tensor(3) len =  3
test1 4 tensor(4) len =  3
test1 5 tensor(5) len =  3
test2 0 tensor([0.9721, 0.4401, 0.8097, 0.5749], dtype=torch.float64) len =  3
test2 1 tensor([0.3238, 0.0371, 0.4474, 0.3992], dtype=torch.float64) len =  3
test2 2 tensor([0.1020, 0.2405, 0.4203, 0.1879], dtype=torch.float64) len =  3
</code></pre>
<p>Actually, if I do not use <code>break</code>, the loop of <code>test1</code> would never stop. Why?<br>
PyTorch 1.4.0 and PyTorch 1.3.1 both have this problem.</p>

Eta_C: <p>It seems that I should throw a IndexError exception, so that PyTorch can catch it and stop the iteration.</p>
Feeding Multiprocessing IterableDataset from a single source

tsteffek: <p>I’m trying to train a BILSTM as language model.</p>
<p>I have a list of files, all containing varying length documents (1 line up to 1000 lines). I’ve been following <a href="https://medium.com/speechmatics/how-to-build-a-streaming-dataloader-with-pytorch-a66dd891d9dd" rel="nofollow noopener">this tutorial</a>, so I’m creating multiple <code>IterableDataset</code>s, each grabbing a file, streaming its contents until its empty, rinse and repeat. This works all fine.</p>
<p>Now since my files are of such different lengths, I’d like to avoid just splitting the list and passing shards to each dataset. My initial thought was to pass an <code>Iterator</code> (just gets copied, multiplying my data) or <code>queue.Queue</code> / <code>queue.SimpleQueue</code>, but that fails when pickling. (Is this a windows only error? But pickling seems to be required in CUDA environments anyway, so I guess I’ll have the same problem in Linux)</p>
<p><strong>TLDR;</strong> Is there a way to pass some sort of single source of truth into the <code>IterableDataset</code>s, or a way for them to communicate? Just sharing the information of passed files would be enough.</p>

tsteffek: <p>Nevermind, I just realised that it works with <code>torch.multiprocessing.Queue</code>.</p>