akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Confusion of CPU/GPU memory use

leemengwei: <p>On linux,<br>
I make a quite simple feed-forward network which consist of some fc/batchnorm/activation layer.<br>
I then save the model, the *.pt or *.pth file is about 40K given by du model.pt -sh.</p>
<p>In deployment, I load the model just as recommended:<br>
model = NN_model.NeuralNet()<br>
model.load_state_dict(torch.load(model_path).state_dict())<br>
Confusion here: Whether on CPU or CUDA, I noticed that the memory usage is about 1.5G on CPU memory given by free -h and ~900MB on GPU given by nvidia-smi.</p>
Libtorch glog doesn't print

dancedpipi: <p>I have build libtorch from source. Follow by <a href="https://github.com/pytorch/pytorch/tree/v1.0.1" rel="nofollow noopener">https://github.com/pytorch/pytorch/tree/v1.0.1</a></p>
<p>I also install glob. But when I use <code>LOG(INFO) &lt;&lt; "hello world!";</code>, it print nothing. Set the <code>minloglevel</code> to 0 doesn’t work.</p>
<p>What should I do to deal with it? I just want use glog…</p>

dancedpipi: <p><code>DLOG(INFO)</code> will cause compile error.<br>
<code>LOG(ERROR)</code> can work properly.<br>
so How can I use a normal glog?</p>

dancedpipi: <p>Any suggestion?<br>
Appreciate for any help~</p>

chegnyanjun: <p>Maybe you can try it: add add_definitions(-DC10_USE_GLOG) in your project’s cmakelists.txt.</p>

dancedpipi: <aside class="quote no-group" data-username="chegnyanjun" data-post="4" data-topic="63822">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chegnyanjun/48/19332_2.png" class="avatar"> chegnyanjun:</div>
<blockquote>
<p>add_definitions(-DC10_USE_GLOG)</p>
</blockquote>
</aside>
<p>it works!!! you are my god!!!</p>
Confused about the MSE loss implementation

George3d6: <p>Looking at the implementation of MSE loss: <a href="https://pytorch.org/docs/stable/_modules/torch/nn/functional.html#mse_loss" rel="nofollow noopener">https://pytorch.org/docs/stable/_modules/torch/nn/functional.html#mse_loss</a></p>
<p>I’m pretty confused about this bit here:</p>
<pre><code class="lang-auto">    if target.requires_grad:
        ret = (input - target) ** 2
        if reduction != 'none':
            ret = torch.mean(ret) if reduction == 'mean' else torch.sum(ret)
    else:
        expanded_input, expanded_target = torch.broadcast_tensors(input, target)
       ret = torch._C._nn.mse_loss(expanded_input, expanded_target, _Reduction.get_enum(reduction))
</code></pre>
<p>Why exactly is the  <code>torch._C</code> mse_loss being used if the tensor doesn’t need a gradient (so, presumably, if .backwards doesn’t need to be called on the loss) ?</p>
<p>If so, is it faster, or rather, fast enough to justify the complexity ? Even given that the broadcast operation is needed</p>

ptrblck: <p>Note that the different paths are triggered, if the <code>target</code> requires gradients, not the model output.<br>
Most likely the backward pass for the output and target are not yet implemented in the dispatched <code>torch._C</code> path, so that the loss will be calculated using the Python frontend.</p>

George3d6: <blockquote>
<p>Note that the different paths are triggered, if the <code>target</code> requires gradients, not the model output</p>
</blockquote>
<p>Ah, true… but, why would the targets require gradient ?</p>
<p>As in, shouldn’t get gradient be computed on the outputs of the model by comparing them to the targets (in this case via the MSE loss), what’s the point of having a gradient for the target vector since it’s already “correct” ?</p>

ptrblck: <p>You don’t need the gradients in the usual use case, which is also why it wasn’t implemented in the backend in the first place.<br>
However, there seem still to be some use cases as shown e.g. <a href="https://discuss.pytorch.org/t/nn-criterions-dont-compute-the-gradient-w-r-t-targets-error/16406/5">here</a>.</p>

George3d6: <p>Ah, alright, that makes more sense… also potentially explains why a version I implemented using a mask seemed significantly slower than the standard MSELoss.</p>
<p>Thanks a lot for the clarifications <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Read multiple .gz file and return it in one tensor

Mohamed_Nabih: <p>I am trying to read multiple .gz file and return its content in one tensor as follows:</p>
<pre><code class="lang-auto">value_list = [ ]

with ReadHelper('ark: gunzip -c /home/mnabih/kaldi/egs/timit/s5/exp/mono_ali/*.gz|') as reader:

    for i, b in enumerate(reader):

        value = numpy.asarray(b[1])

        value_list.append(value)

        value = torch.from_numpy(value)

        print(type(value))
</code></pre>
<p>it gives me multiple tensors, I tried this command to concatenate it</p>
<pre><code>    values = torch.cat(value, dim = 0, out=None)
But it gives me 
</code></pre>
<p>cat() received an invalid combination of arguments - got (Tensor, out=NoneType, dim=int), but expected one of:</p>
<ul>
<li>(tuple of Tensors tensors, name dim, Tensor out)</li>
</ul>

ptrblck: <p>Could you append tensors to the list and call <code>torch.tensor(value_list)</code>?<br>
This should work:</p>
<pre><code class="lang-python">value = np.random.randn(10, 10)
value_list = []
for _ in range(3):
    value_list.append(value)

res = torch.tensor(value_list)
print(res.shape)
&gt; torch.Size([3, 10, 10])
</code></pre>
Using forward hooks to modify the output of the neural network

dakshanand: <p>I have a simple neural network and I want to modify the value of the output layer for a given input ‘I’  to the input layer.</p>
<p>Pytorch documentation has this tutorial which says -</p>
<p>"<br>
We’ve inspected the weights and the gradients. But how about inspecting / modifying the output and grad_output of a layer?</p>
<p>We introduce hooks for this purpose.<br>
"<br>
The tutorial just says this can be done but does not show how to actually do it. Can someone please provide some code that shows how to change/set the outputs?</p>

SimonW: <p><a href="http://pytorch.org/docs/master/nn.html#torch.nn.Module.register_forward_hook" class="onebox" target="_blank">http://pytorch.org/docs/master/nn.html#torch.nn.Module.register_forward_hook</a></p>

karl-schulz: <p>tl;tr: torch &gt;=v1.2.0</p>
<p>Hey <a class="mention" href="/u/dakshanand">@dakshanand</a>,<br>
I just ran into the same issue. In case anybody else is wondering: Update torch to &gt;=1.2.0. Before this version, hooks were read-only, now they can modify the output.<br>
Best</p>
Mismatch error ,, jacobian gradient matrix

Sam3: <p>Hi,<br>
i have input image with size(1,1,28,28) and output image y with the same size. y= cnn(x)<br>
and i want to calculate the gradient between every output pixel image with respect to every pixel from input image but i have  : RuntimeError: Mismatch in shape: grad_output[0] has a shape of torch.Size([784]) and output[0] has a shape of torch.Size([]).<br>
my cod is :<br>
def jacobian(y, x, create_graph=False):</p>
<pre><code>jac = []
y = cnn(x)

x.requires_grad_(True)
flat_y = y.reshape(-1) 
flat_x = x.reshape(-1)
#y= cnn(flat_x)
grad_y = torch.zeros_like(flat_y) 

for i in range(len(flat_y)): 
    
    for j in range(len(flat_x)):
    
        grad_y[i] = 1. 
   
        grad_x,= torch.autograd.grad(flat_y[i] ,flat_x[j],grad_y,retain_graph=True,create_graph= create_graph,allow_unused=True)

        jac.append(grad_x.reshape(x.shape))  
        
        grad_y[i] = 0.

return torch.stack(jac).reshape(y.shape + x.shape)  
</code></pre>
<p>print( jacobian (y,x,create_graph = True) )</p>
<p>could someone help me why i have this error?</p>

albanD: <p>Duplicate of <a href="https://discuss.pytorch.org/t/jacobian-gradient-matrix/69244" class="inline-onebox">Jacobian gradient matrix</a></p>
<p>The issue here is that flat_y[i] is a single number.<br>
You should either give flat_y and keep doing what you do with the grad outputs so that they are both of the size of flat_y.<br>
Or you can remove the grad outputs and use flat_y[i] so that they are both of size 1.</p>
Tensor slicing and transpose

llefolgo: <p>Hi,</p>
<p>Is the following expected behaviour? (pytorch 1.4)</p>
<blockquote>
<p>a = torch.tensor([[[1,2],[3,4]]]) # a[0] is a 2*2 tensor with values 1,2,3,4<br>
a[0] = a[0].t()<br>
print(a) # returns [[[1,3],[3,4]]]</p>
</blockquote>
<p>Numpy returns [[[1,3],[2,4]]] in a similar scenario.</p>
<p>This kind of assignment is not very pytorch-like (not backprop’able I guess), but still I’m curious about what’s happening under the hood now. It’s as if the LHS assignment and RHS evaluation is happening one element at a time.</p>
<p>This is a minimal example, as</p>
<blockquote>
<p>b = torch.tensor([[1,2],[3,4]]) # 2d tensor<br>
b = b.t() # returns [[1,3],[2,4]]</p>
</blockquote>
<p>matches intuition (or doing a[0].clone().t()).</p>
<p>Thanks!</p>

albanD: <p>Hi,</p>
<p>The problem is that your right hand side and left hand side actually look at the same memory. so while you modify the right hand side before finishing to read it. Hence the result you see.<br>
You should not write and read the same expression in general.</p>
Do we have any places that we can deploy it, such as in flask for Free(🤑)?

JonathanSum: <p>Hi, I am thinking that I should deploy my Pytorch model(Maybe an image classifier or recoloring image).<br>
However, I want to deploy it to somewhere that I can run it with a pre-trained model with pth and pkl format.<br>
Thank You💖</p>

brighteningeyes: <p>hi,<br>
you can deploy on heroku and if you have a paid pythonanywhere account you can deploy it there as well.</p>

JonathanSum: <p>Thank You for telling me that. I just want to make sure.<br>
Why do people say it has sleeping issue for free version of heroku? Does it mean that I send a picture, and it will response to me like 5 mins later?</p>

brighteningeyes: <p>hi,<br>
no.<br>
it meanns when someone accesses heroku, the server will be up for about 30 mminutes (you will loose your 30 minutes of usage if you got only 1 request, or 100000).<br>
after that the app will be slept (you won’t use your usage until another request).</p>
How is it possible that I can't create a deep copy although all my parameters are leafs?

pinocchio: <p>I was playing around with deep copy <a href="https://discuss.pytorch.org/t/how-does-one-reproduce-the-deepcopy-error-of-tensors-made-by-code/69852/3">here</a> and eventually I got a very funny error were all my parameters are indeed leafs but I cannot do a deep copy regardless:</p>
<pre><code class="lang-auto">'''
'''
import torch
import torch.nn as nn

import copy
from collections import OrderedDict

model = nn.Sequential( OrderedDict( [ ('fc0', nn.Linear(3,1)) ] ) )
w = model.fc0.weight
print(f'w = {w.norm(2)}')
for i in range(5):
    w_new = w - 1000
print(f'w_new = {w_new.norm(2)}')
print(f'w_new is not a leaf?: {not(w_new.is_leaf)}')
assert not(w_new.is_leaf)
#model.fc0.weight = nn.Parameter( w_new )
setattr(model,'fc0.weight', w_new )
print(f'model.fc0.weight.norm(2) = {model.fc0.weight.norm(2)}')
print(model.fc0.weight.is_leaf)
print(model.fc0.bias.is_leaf)
model_copy = copy.deepcopy(model)
</code></pre>
<p>output:</p>
<pre><code class="lang-auto">w = 0.4955401122570038
w_new = 1732.4461669921875
w_new is not a leaf?: True
model.fc0.weight.norm(2) = 0.4955401122570038
True
True
.
.
.
__deepcopy__(self, memo)
     42     def __deepcopy__(self, memo):
     43         if not self.is_leaf:
---&gt; 44             raise RuntimeError("Only Tensors created explicitly by the user "
     45                                "(graph leaves) support the deepcopy protocol at the moment")
     46         if id(self) in memo:

RuntimeError: Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol at the moment
</code></pre>
<p>which seems really strange. What is going on? Is this a bug in pytorch?</p>

albanD: <p>You are not using the <code>setattr</code> function correctly. If you want to change the weights from fc0, you need to do <code>setattr(model.fc0,'weight', w_new )</code></p>

pinocchio: <aside class="quote no-group quote-modified" data-username="albanD" data-post="2" data-topic="69864">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>setattr(model.fc0,‘weight’, w_new )</p>
</blockquote>
</aside>
<p>What I want to avoid is exactly this <code>model.fc0</code> because I want it to work for a model in general (so I have the string names of the layers)…is that possible?</p>

albanD: <p>Then you need two call to <code>setattr</code>, one for fc0 and one for weight.</p>
How do pull predictions out for overall accuracy

Jordan_Howell: <p>Hello,</p>
<p>I’m trying to figure out the best way to pull accuracy out for future plotting.  Right now, I get a tensor of 10 predictions.  How do i pull each one out of the tensor to store in a list to calculate accuracy for later plotting?  Code below:</p>
<pre><code class="lang-auto">train_losses, test_losses = [], []
train_accuracy, test_accuracy = [], []

def train_model(model, criterion, optimizer, num_epochs=5, test_epochs = 5):
    liveloss = PlotLosses(series_fmt={'training': '{}', 'test':'test_{}'})
    max_accuracy = 0
    running_loss = 0.0
    running_corrects = 0
    for epoch in range(num_epochs):
        for b, (image, label, policy, categorical_data, numerical_data) in enumerate(train_loader):
            image = image.cuda()
            label = label.cuda()
            numerical_data = numerical_data.cuda()
            categorical_data = categorical_data.cuda()

            outputs = model(image, categorical_data, numerical_data)
            loss = criterion(outputs, label)
            print(loss.item())
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

            _, preds = torch.max(outputs, 1)
            print(preds)
            running_loss += loss.item()
            running_corrects += torch.sum(preds == label.data)

            b += 1
            if b % print_interval == 0:
                print(f"train_epoch: {epoch},train_batch: {b}")

            if b == max_trn_batch:
                train_losses.append(running_loss / max_trn_batch)
                train_accuracy.append(running_corrects/ max_trn_batch)
                print(f"epoch: {epoch}, loss: {train_losses}, accuracy: {train_accuracy}")
                break
</code></pre>

Jordan_Howell: <p>Nevermind.  I put a <code>.item()</code> on the end of <code>running_corrects += torch.sum(preds == label.data).item()</code>.</p>
Understanding gradients when .detach() is used

MahdiNazemi: <p>I am trying to understand the impact of <code>.detach()</code> on gradients and have set up an example as follows:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        
        self.features = nn.Sequential(
            nn.Linear(784, 100),
            nn.ReLU(inplace=True),
            nn.Linear(100, 100),
            nn.ReLU(inplace=True),
        )
        
    def forward(self, x):
        # The original forward function
        x = self.features(x)

        # The modified forward function including the .detach() method
        #x = self.features(x) + (1 - self.features(x)).detach()
        
        return x
</code></pre>
<pre><code class="lang-auto">torch.manual_seed(0)
x = torch.randn(784).view(1, 784).requires_grad_(True)

net = Net()
print(x.grad)

u = net(x)
loss = u.abs().sum()
print(loss)
loss.backward()

print(x.grad)
</code></pre>
<p>Regardless of whether I run the original <code>forward</code> function or the modified one (which is commented out above), I get the same values for <code>x.grad</code>. This is despite the fact the the loss is about 10 in one case and 100 in the other.</p>
<p>I understand that the way my <code>forward</code> method is defined, backward sees <code>self.features(x)</code> as the forward method in both cases. The part I’m confused about is <strong>why having two different loss values and the same backward function results in the same gradient values for <code>x</code></strong>.</p>
<p>Is this the expected behavior or a different loss value should result in a different <code>x.grad</code> value?</p>

albanD: <p>Hi,</p>
<p>As soon as you use detach, you don’t get “real” gradients and the whole thing becomes quite hard to reason about.</p>
<p>The main reason for you to get the same value, is that in the lower part of your function, the gradient does not depend on the value. Try changing your loss to a squared norm, and you will see a different behavior.</p>
<p>You can check this work in progress of the revamp of the autograd doc that tries to define detach in a more mathematical way: <a href="https://alband.github.io/doc_view/notes/autograd.html">here</a>.</p>

MahdiNazemi: <p>Thanks for your answer. I modified the loss function to <code>torch.norm()</code> and observed different gradients at the input.</p>
<p>However, I have difficulty understanding the following part of your answer:<br>
“The main reason for you to get the same value, is that in the lower part of your function, the gradient does not depend on the value.”<br>
Do you mind sharing pointers to resources that explain this part in more details?</p>
<p>Thank you!</p>

albanD: <p>The idea is that the gradient of <code>2*x</code> is <code>2</code>. So whatever is the value of <code>x</code> during the forward, it does not change the value of the gradient.<br>
On the other hand, for <code>x**2</code>, the gradient is <code>2*x</code> and so depending on the value <code>x</code> used in the forward, the gradient will change.</p>
Efficiently move data from torch::Tensor to std::vector

dyu: <p>In C++ I have a Tensor of shape <code>(32, 1211, 34)</code> on a GPU. I want to move its data to <code>std::vector</code>. Now I do it in this way:</p>
<pre><code class="lang-auto">typedef vector&lt;vector&lt;float&gt;&gt; EmissionT;
// ...
auto tensor = i_value_out.toTensor().to(torch::kCPU);
auto outShape = tensor.sizes();

vector&lt;EmissionT&gt; emissions(outShape[0], EmissionT(outShape[1], vector&lt;float&gt;(outShape[2])));
auto access = tensor.accessor&lt;float, 3&gt;();

for (size_t nBatches = 0; nBatches &lt; outShape[0]; ++nBatches) {
    for (size_t T = 0; T &lt; outShape[1]; ++T) {
        for (size_t N = 0; N &lt; outShape[2]; ++N) {
            emissions[nBatches][T][N] = access[nBatches][T][N];
        }
    }
}
</code></pre>
<p>Is there a way to avoid copying and just move data?</p>

tom: <p>No. The problem is that std::vector cannot be initialized with given data.<br>
If you only need array-like access, you can use ArrayRef instead, but you have to take care of keeping the underlying tensor around.</p>
<p>Alternatively, you could allocate the vector, use torch.from_blob on that and then use an <code>_out</code> variant to write to that. Depending on the final op, it would just move the copying deep into PyTorch though.</p>
<p>Best regards</p>
<p>Thomas</p>
Naming Tensors in TensorDataset

ab-10: <p>What are your thoughts on providing an option to name the individual tensors in the TensorDataset, similarly to how named tensors allow dimension naming?</p>
<p>The goal of this post is to form an early understanding of the sentiment towards the idea.</p>
<p>I believe that naming tensors in TensorDataset would be a logical extension to naming tensor dimension and would provide similar benefits to datasets as named dimensions provide to tensors.</p>

ptrblck: <p>Could you post an example, how this naming would be used in a <code>Dataset</code>, please?</p>

ab-10: <p>Of course!</p>
<p>I’m thinking of something along the lines of:</p>
<pre><code class="lang-auto">import torch
from torch.utils.data import dataset
a = torch.randn(5, 10)
b = torch.randn(5,15)
named_tensor_dataset = dataset.TensorDataset(a,b,names=('embeddings', 'labels'))

# printing tensor names helps understand what data
# I'm dealing with and the order of that data
print(named_tensor_dataset.names)
&gt;&gt;&gt; ('embeddings', 'labels')

for example in named_tensor_dataset:
    y = SampleNN(example.embeddings)
    err = criterion(y, example.labels)
</code></pre>

ptrblck: <p>Thanks for the example.<br>
An easy way of “named samples” would be to return a <code>dict</code> in your <code>Dataset</code> and use the <code>key</code> as the name.<br>
Not sure, if a custom class implementation would work, and I would need to verify it.</p>

ab-10: <p>I was rather thinking of returning a <code>collections.namedtuple</code>, since it would allow to keep the current access by index in addition to access by name.</p>
<p>What do you think about the value of adding the feature to PyTorch?</p>

ptrblck: <p>How would the batching work with a <code>namedtuple</code>?<br>
Wouldn’t the sampler be required to create the batch as a <code>namedtuple</code>, so that you could index it inside the <code>DataLoader</code> loop as e.g.:</p>
<pre><code class="lang-python">for batch in loader:
    x, y = batch.data, batch.target
</code></pre>

ab-10: <p>I’m not sure why would the sampler need to create a <code>namedtuple</code>, since obtaining the number of <code>namedtuples</code> in the <code>Dataset</code> would suffice for generating random indices over them.</p>
<p>For batching the <code>DataLoader</code> should created a new <code>namedtuple</code> such that each element in the orginal corresponds to a tensor of batch size in the new one (as you have illustrated in the example).</p>
Simple CNN returnDefaultCPUAllocator: not enough memory: you tried to allocate 377487360000 bytes. Buy new RAM!

R_Tux: <p>Hi all,</p>
<p>I am very new to Pytorch and am trying to set up a simple Convolutional Neural Network. I followed some of the online guides and set up this test network:</p>
<pre><code class="lang-python">import torch
class SimpleCNN(torch.nn.Module):
    def __init__(self):
        super(SimpleCNN, self).__init__()
        img_size = 50 # image size
        s_im_size = int(img_size/2/2) # image size after 2 times maxPool
        out1 = 2
        out2 = 2
        # Conv Layer 1
        self.layer1 = torch.nn.Sequential(
        torch.nn.Conv2d(1, out1, kernel_size=5, stride=1, padding=2),
        torch.nn.ReLU(),
        torch.nn.MaxPool2d(kernel_size=2, stride=2)
        )
        # Conv Layer 2
        self.layer2 = torch.nn.Sequential(
        torch.nn.Conv2d(out1, out2, kernel_size=5, stride=1, padding=2),
        torch.nn.ReLU(),
        torch.nn.MaxPool2d(kernel_size=2, stride=2)
        )  
        self.fc1 = torch.nn.Linear(out2*s_im_size**2, 3*img_size**2)
        self.fc2 = torch.nn.Linear(3*img_size**2, 3*img_size**2)
    def forward(self, x):
        img_size = 50 # image size
        s_im_size = int(img_size/2/2) # image size after 2 times maxPool
        out2 = 2
        out = self.layer1(x)   # Change from (1,50,50) to (5,25,25)
        out = self.layer2(out)   # Change from (5,25,25) to (7,12,12)
        out = out.view(-1,out2*s_im_size**2) # Change from (7,12,12) to (1,7*12*12)
        out = self.fc1(out)
        out = self.fc2(out)
        out = out.view(img_size**2,3)
        return out

model = SimpleCNN()
</code></pre>
<p>Where i should input images of shape (50,50) and get an output of shape (50*50,3). All well and good.</p>
<p>But if I change image size to take slightly larger images ,<code>img_size = 320</code>, and call <code>model=SimpleCNN()</code> i get the error:</p>
<pre><code class="lang-python">RuntimeError: [enforce fail at ..\c10\core\CPUAllocator.cpp:72] data. DefaultCPUAllocator: not enough memory: you tried to allocate 377487360000 bytes. Buy new RAM!
</code></pre>
<p>Which to my mind doesn’t quite make sense. I must have made a mistake in my network, can anyone help me figure it out?</p>
<p>cheers</p>

S_Ellis: <p>Going from 50x50 to 320x320 can’t really be described as ‘slightly larger’! For 50x50 images, the first FC has input size of 288 and output size of 7500. For 320x320 images, that layer has input size 12800 and output size 307200. So in the first case fc1 has 2160000 trainable parameters and for the larger images it has 3932160000 trainable parameters. You will likely need to change the architecture if you need to use larger images.</p>

R_Tux: <p>Thanks for your comment, I now see why that would be a problem.</p>
Python int too large to convert to C long with torchtext

Xin_Chen: <p>Hey guys,</p>
<p>I’m following the tutorial " Text Classification with TorchText" and get errors here:</p>
<p>“OverflowError: Python int too large to convert to C long”</p>
<p>I searched the forum and it seems someone fixed the error by change the the 129th line of “torchtext/utils.py” from “csv.field_size_limit(sys.maxsize)” to “csv.field_size_limit(maxInt)”, however it doesn’t work for me, is there anyone know why?</p>
<p>Thanks for your help!</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/0/305f9d2ff1985297e4a1ac113fb079bbc6490198.png" data-download-href="https://discuss.pytorch.org/uploads/default/305f9d2ff1985297e4a1ac113fb079bbc6490198" title="批注 2020-02-17 223840"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/305f9d2ff1985297e4a1ac113fb079bbc6490198_2_690x158.png" alt="批注 2020-02-17 223840" data-base62-sha1="6TVLMpFRqJpMjKmeuXQzq2oxm4g" width="690" height="158" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/305f9d2ff1985297e4a1ac113fb079bbc6490198_2_690x158.png, https://discuss.pytorch.org/uploads/default/original/3X/3/0/305f9d2ff1985297e4a1ac113fb079bbc6490198.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/0/305f9d2ff1985297e4a1ac113fb079bbc6490198.png 2x" data-dominant-color="EBF1F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">批注 2020-02-17 223840</span><span class="informations">991×227 21 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Best practice for structuring and loading multi-label classification data

carlstronaut: <p>Hello guys,</p>
<p>I’m wondering if there’s some standardized/best practice way of both storing and loading data from a multi-label dataset. For my use case, I’m generating data from an audio dataset, in which I cut up audio-tracks into spectrograms as input features, and each input is multi labeled with 18 different classes.</p>
<p>After preprocessing one audio-track I get a (for example) (33 x 96 x 86) numpy array of 33 inputs and a corresponding (33 x 18) numpy array with labels.</p>
<p>Right now I’m storing everything like:</p>
<pre><code class="lang-auto">└── Preprocessed
    ├── Track1
    │   ├── labels.npy
    │   └── spectrograms.npy
    ├── Track2
    │   ├── labels.npy
    │   └── spectrograms.npy
    └── Track3
        ├── labels.npy
        └── spectrograms.npy
</code></pre>
<p>The reason for this is so that I can keep track of which track the different input comes from, in case I want to balance it somehow in a train/test split.</p>
<p>However, it feels like kind of a hussle to build a <code>Dataset</code> and corresponding <code>__getitem__ </code> in PyTorch using this structure, so I’m wondering if there is some smarter standardized way to structure and load multi-labeled data?</p>
Inf and Nan loss in RNN

ayush1999: <p>Here is my RNN class:</p>
<pre><code class="lang-auto">import torch.nn as nn
from torch.autograd import Variable

class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super().__init__()
        self.hidden_size = hidden_size
        self.i2h = nn.Linear(input_size , hidden_size)
        self.h2o = nn.Linear(hidden_size, output_size)
        self.h2h = nn.Linear(hidden_size, hidden_size)
        self.softmax = nn.LogSoftmax(dim = 1)

    def forward(self, input, hidden):
        h = self.softmax(self.h2h(hidden)+  self.i2h(input))
        o = self.softmax(self.h2o(h))
        return o, h

    def init_hidden(self):
        return Variable(torch.zeros(1, self.hidden_size))

</code></pre>
<p>And I created an instance and trained as follows:</p>
<pre><code class="lang-auto">rnn = RNN(n_chars, 90, n_chars)
criterion = nn.MSELoss()
learning_rate = 0.05
optimizer = torch.optim.Adam(rnn.parameters(), lr = learning_rate)
hidden = rnn.init_hidden()
epochs = 5


for epoch in range(epochs):
    for i in range(len(X)):
        for ele in X[i]:
            output, hidden = rnn(Variable(ele.t()), hidden)
        loss = criterion(output, Y[i])
        
        loss.backward(retain_graph=True)
        optimizer.zero_grad()
        optimizer.step()

        if (i %1000 == 0):
            print('Current loss is: ', loss)
</code></pre>
<p>The result is an extremely large loss in the first step, an then the loss becomes Inf, after which it becomes Nan till the end. Why is this happening? How can I correct this?</p>

jpeg729: <p>An unrelated issue…</p>
<p><code>optimizer.zero_grad</code> should come before <code>loss.backward</code> or after <code>optimizer.step</code>. If you put it after <code>.backward</code> and before <code>.step</code> then you delete the gradients needed for the update before applying the update.</p>
<p>LogSoftmax outputs values between -inf and 0. I suspect that with each step your hidden state is getting closer and closer to -inf.</p>
<p>You might have better luck if you use a bounded activation for the hidden states, such as nn.Tanh.</p>

ayush1999: <p>Oh, I see. Can you also have a look at <a href="https://discuss.pytorch.org/t/rnn-weights-get-converted-to-nan-values/15755">RNN weights get converted to nan values</a> ? I opened it and it looks related to the current error.</p>

AbdulsalamBande: <aside class="quote no-group" data-username="ayush1999" data-post="1" data-topic="15707">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/ee59a6/48.png" class="avatar"> ayush1999:</div>
<blockquote>
<p>loss.backward(retain_g</p>
</blockquote>
</aside>
<ol>
<li>
<p><strong>Possible Problem 1</strong> -  <em>optimizer.zero_grad()</em> should come before <em>loss.backward(retain_graph=True)</em> Or else all of the computer gradients will be Zeroed  and all of the parameters will not be updated.</p>
</li>
<li>
<p><strong>Possible Problem 2</strong> - Inspect all of your input and make sure none of the sequences contains “NAN”</p>
</li>
</ol>
Plot several graphs in pytorch tensorboard

erap129: <p>Hello,</p>
<p>I am training a dynamic neural network, meaning that each epoch I tweak the architecture and get a different computational graph.<br>
I want to plot the graph for each epoch using tensorboard, but when I use SummaryWriter.add_graph() at the end of each epoch it simply overwrites the previous one.</p>
<p>Any ideas how to plot several graphs using pytorch + tensorboard? It seems achievable as each graph has a “tag” but I found no option to change this tag to plot several of them.</p>
<p>Thanks,<br>
Elad</p>
Conv1D outputs NaNs

Hippolyte_Dubois: <p>Hi,<br>
I have a network which outputs NaNs after some epochs.<br>
It looks like this:</p>
<pre><code class="lang-auto">self.filter = nn.Sequential(
    nn.Conv1D(...),
    nn.Sigmoid()
)
</code></pre>
<p>The input vector is valid, doesn’t contain any NaNs.</p>
<p>It used to work fine with the following loss function:</p>
<pre><code class="lang-auto">distrib = torch.distributions.MultivariateNormal(y, torch.eye(y.size()[0]*sigma)
loss = -distrib.log_prob(x)
</code></pre>
<p>but I modified it so that I could compare different models with different sigmas:</p>
<pre><code class="lang-auto">distrib = torch.distributions.MultivariateNormal(y, torch.eye(y.size()[0])
loss = -(distrib.log_prob(x)/distrib.log_prob(y))
</code></pre>
<p>I use Adam Optimizer, SGD doesn’t work either. What is wrong ?<br>
Thanks.</p>

Hippolyte_Dubois: <p>Fixed it:<br>
You can’t just normalize the log_prob, you have to exponentiate it first:</p>
<pre><code class="lang-auto">loss = -(distrib.log_prob(x).exp() / distrib.log_prob(y).exp()).log()
</code></pre>
<p>Works fine now !</p>
<p>Leaving the thread in case it can be of use to anyone.</p>
CNN 1d layers structure

Muhammad_Izaz: <p>please some one explain the 1d CNN layers and how to prepare your data of size 2968 row and 101 column to pass into the CNN layers, the last column is the target column and the first hundred are input data.</p>

Hippolyte_Dubois: <p>If you want to regress the last column from a 100-long time series, this is how you could write your dataset:</p>
<pre><code class="lang-auto">class CustomDataset(torch.utils.data.Dataset):
    def __init__(self, data):
        self.x = data[:,:-1]
        self.y = data[:,-1]

    def __getitem__(self, index):
       return (self.x[index,:], self.y[index])

   def __len__(self):
       return self.x.size()[0]
</code></pre>
<p>You can then access your data using either the naive <code>for x,y in MyDataset:</code> approach, or use a dataloader.</p>
<p>For more details on the Conv1D, you can refer to this post <a href="https://discuss.pytorch.org/t/1d-convolution-on-1d-data/54661">here</a></p>
CUDA out of memory, but it shows enough memory available in error expression

JoYeY: <p>Dear all,<br>
I train a CNN with GPU and get an error information as figure shows. 30.00 MiB data is to be allicated and 967.43 MiB free but a CUDA out of memory is given:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/c/3c409f39e5499c5fea294df373b32f84b997f90b.png" data-download-href="https://discuss.pytorch.org/uploads/default/3c409f39e5499c5fea294df373b32f84b997f90b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/c/3c409f39e5499c5fea294df373b32f84b997f90b_2_690x226.png" alt="image" data-base62-sha1="8B16d4nWixVmezuolYMi1mvLMGD" width="690" height="226" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/c/3c409f39e5499c5fea294df373b32f84b997f90b_2_690x226.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/c/3c409f39e5499c5fea294df373b32f84b997f90b_2_1035x339.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/c/3c409f39e5499c5fea294df373b32f84b997f90b.png 2x" data-dominant-color="302F32"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1319×433 77.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I have no idea on it</p>

ptrblck: <p>Could you check the memory usage via <code>nvidia-smi</code>, please?<br>
Are other processes using memory on your device?</p>

JoYeY: <p>I set <code>batch_size</code> to 1 and it works.<br>
When I set <code>batch_size</code> to 2, and the same error is reported again:<br>
“RuntimeError: CUDA out of memory. Tried to allocate <strong>30.00 MiB</strong> (GPU 0; 2.00 GiB total capacity; 784.67 MiB already allocated; <strong>551.43 MiB free</strong>; 31.33 MiB cached)”.<br>
I use <code>nvidia-smi</code> and only this process exists.<br>
Do you have other suggestions?<br>
Thanks!</p>

ptrblck: <p>In that case the error message might be misleading.<br>
How much memory is your training code using with <code>batch_size=1</code>?</p>

JoYeY: <p>It shows around 881MiB of 2048MiB as  <code>batch_size = 1</code></p>

JoYeY: <p>It occupies more than 1500MiB as <code>batch_size = 1</code> and I will modify my net and  data.<br>
Thank you!</p>
Create dataset of pair image

GoHan: <p>Hi there, i’m create siamese networks for face comparison, folder image have structure:</p>
<ul>
<li>face_db</li>
<li>
<ul>
<li>id1:</li>
</ul>
</li>
<li>
<ul>
<li>img11.jpg</li>
</ul>
</li>
<li>
<ul>
<li>img12.jpg</li>
</ul>
</li>
<li>
<ul>
<li>…</li>
</ul>
</li>
<li>
<ul>
<li>id2:</li>
</ul>
</li>
<li>
<ul>
<li>img21.jpg</li>
</ul>
</li>
<li>
<ul>
<li>img22.jpg</li>
</ul>
</li>
<li>
<ul>
<li>…</li>
</ul>
</li>
</ul>
<p>So, how to create dataset contain pair image? There is code i search but I found it missing, because, if i have n image then i have n*(n-1)/2 item, but this have n item? Can i help me please?</p>
<pre><code class="lang-auto">class SiameseNetworkDataset(Dataset):
    
    def __init__(self, imageFolderDataset):
        self.imageFolderDataset = imageFolderDataset    
        
    def __getitem__(self, index):
        img0_tuple = random.choice(self.imageFolderDataset.imgs)
        
        should_get_same_class = random.randint(0,1)
        
        if should_get_same_class:
            while True:
                img1_tuple = random.choice(self.imageFolderDataset.imgs) 
                if img0_tuple[1]==img1_tuple[1]:
                    break
        else:
            while True:
                img1_tuple = random.choice(self.imageFolderDataset.imgs) 
                if img0_tuple[1] !=img1_tuple[1]:
                    break

        img0 = embedding_img(img0_tuple[0])
        img1 = embedding_img(img1_tuple[0])
        
        return img0, img1, torch.from_numpy(np.array([int(img1_tuple[1]!=img0_tuple[1])],dtype=np.float32))
    
    def __len__(self):
        return len(self.imageFolderDataset.imgs)
</code></pre>
Unable to clear CUDA cache

vr25: <p>Hi,</p>
<p>I am trying to clear the CUDA cache using torch.cuda.empty_cache() but it is unable to clear it.</p>
<p>Following are the code and the error:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/b/fb4a1e78bd74536ccadc3f163dc3f0784ad476c6.png" data-download-href="https://discuss.pytorch.org/uploads/default/fb4a1e78bd74536ccadc3f163dc3f0784ad476c6" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/b/fb4a1e78bd74536ccadc3f163dc3f0784ad476c6_2_493x500.png" alt="image" data-base62-sha1="zR0DWXQ0lW3BvWv0ynFs8OCrst8" width="493" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/b/fb4a1e78bd74536ccadc3f163dc3f0784ad476c6_2_493x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/b/fb4a1e78bd74536ccadc3f163dc3f0784ad476c6_2_739x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/b/fb4a1e78bd74536ccadc3f163dc3f0784ad476c6.png 2x" data-dominant-color="331028"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">901×912 92.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/c/ec4f6e6c83e8a33d59db67fa096e1185863faf96.png" data-download-href="https://discuss.pytorch.org/uploads/default/ec4f6e6c83e8a33d59db67fa096e1185863faf96" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/c/ec4f6e6c83e8a33d59db67fa096e1185863faf96_2_690x384.png" alt="image" data-base62-sha1="xIuRyOajwfFxcbEXXwxWwXg0cnQ" width="690" height="384" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/c/ec4f6e6c83e8a33d59db67fa096e1185863faf96_2_690x384.png, https://discuss.pytorch.org/uploads/default/original/3X/e/c/ec4f6e6c83e8a33d59db67fa096e1185863faf96.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/c/ec4f6e6c83e8a33d59db67fa096e1185863faf96.png 2x" data-dominant-color="3D1C32"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">929×518 95.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thanks!</p>

ptrblck: <p><code>empty_cache</code> will only release the unused and cached memory.<br>
If the parameters and activations need more memory, you will run into an OOM error nevertheless.<br>
You could try to lower the batch size or alternatively use <a href="https://pytorch.org/docs/stable/checkpoint.html"><code>torch.utils.checkpoint</code></a> to trade compute for memory.</p>
How to slice 3D tensor to 2D slices and construct Dataloader

Xiaoyu_Song: <p>Hi, I have a 3D image, and I would like to slice it to coronal, axial and sagittal views, and train the CNN using these slices.<br>
I’m wondering how to construct a dataloader in this situation. i.e How to slice this 3D image and construct the corresponding data class. Because In my understand, I need to iterate the slices when I construct the dataloader.</p>

roaffix: <p>Hi,</p>
<p>Have you checked this topic <a href="https://discuss.pytorch.org/t/train-3d-networks-with-pytorch/611" class="inline-onebox">Train 3D Networks with Pytorch</a>?</p>

Xiaoyu_Song: <p>Hi, Yes, but as the data will be very large, so I prefer to slice the 3d data first</p>

DD0922: <p>Have you solved this problem?<br>
Since each 3D image is very large , I also need to choose some 2D slices from 3D image.<br>
I think we can write the path of 2D slice and read them in __getitem() function…</p>
<p>Do you have another better solution??</p>

Xiaoyu_Song: <p>hi, I just sliced them by the order of that annex. For example, if the 3d image is 128x256x512, in order to slice it in sagittal view, I sliced 512 times, so I got 512 128x256 images.</p>
Torch.load did not use the gpu specified for training

Xiaoyu_Song: <p>Hi, I’m loading pretrained model for training, and I specified the gpu id for training is 1, but during the training, I saw the gpu 0’s memory is also occupied. Someone could explain? Thank you</p>

ptrblck: <p>Most likely the CUDA context is also initialized on the default device.<br>
You could mask all devices and only pass the wanted one via:</p>
<pre><code class="lang-auto">CUDA_VISIBLE_DEVICES=1 python script.py args
</code></pre>

Xiaoyu_Song: <p>Thankyou ptrblck, I get this</p>
<pre><code class="lang-auto">THCudaCheck FAIL file=../torch/csrc/cuda/Module.cpp line=37 error=101 : invalid device ordinal
</code></pre>
<p>Just share here in case others encounter the same error.<br>
If set</p>
<pre><code class="lang-auto">CUDA_VISIBLE_DEVICES =3
</code></pre>
<p>then in the train.py code, the gpu_id should be set to 0, not to 3.</p>
I want to run my simple cnn on cuda but as I am new and I am unable to do so plz help , I am getting this error : " RuntimeError: Expected object of backend CPU but got backend CUDA for argument #2 'weight''

Sanjeevramukyadav: <p><strong>## This is my cnn</strong></p>
<p>import torch<br>
import torch.nn as nn<br>
import torch.nn.functional as F</p>
<p>class Net(nn.Module):</p>
<pre><code>def __init__(self):
    super(Net, self).__init__()
    # 1 input image channel, 6 output channels, 3x3 square convolution
    # kernel
    self.conv1 = nn.Conv2d(1, 6, 3)
    self.conv2 = nn.Conv2d(6, 16, 3)
    # an affine operation: y = Wx + b
    self.fc1 = nn.Linear(16 * 6 * 6, 120)  # 6*6 from image dimension
    self.fc2 = nn.Linear(120, 84)
    self.fc3 = nn.Linear(84, 10)

def forward(self, x):
    x.to('cuda')
    # Max pooling over a (2, 2) window
    x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
    # If the size is a square you can only specify a single number
    x = F.max_pool2d(F.relu(self.conv2(x)), 2)
    x = x.view(-1, self.num_flat_features(x))
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = self.fc3(x)
    return x

def num_flat_features(self, x):
    size = x.size()[1:]  # all dimensions except the batch dimension
    num_features = 1
    for s in size:
        num_features *= s
    return num_features
</code></pre>
<p>net = Net()<br>
print(net)</p>
<p>**#  And this is the part of code where i am passing a tensor to my **<br>
<strong>#   network</strong><br>
input = torch.randn(1, 1, 32, 32)<br>
net.cuda()<br>
input.cuda()<br>
output = net(input)<br>
target = torch.randn(10)  # a dummy target, for example<br>
target = target.view(1, -1)  # make it the same shape as output<br>
criterion = nn.MSELoss()</p>
<p>loss = criterion(output, target)<br>
print(loss)</p>

ptrblck: <p>The <code>.to()</code> and <code>.cuda()/.cpu()</code> operations are not inplace ops for tensors, so you need to assign the result:</p>
<pre><code class="lang-python">x = x.to('cuda')
input = input.cuda()
</code></pre>
How to split the dataset of images with labeled folder into scikit learn train_test_split?

Deb_Prakash_Chatterj: <p>Previously I was using PyTorch to split my dataset and train my classifier, but now I want to use Sci-Kit learn to train my SVM model. For that reason, I need to split my dataset into train and test set. Now, Sci-Kit learn uses this <code>xtrain, xtest, ytrain, ytest = X, y, test_size=0.3, random_state=42)</code> to split. I am using this to split my data -</p>
<pre><code class="lang-auto">from google.colab import drive
drive.mount('/content/drive')
data = "/content/drive/My Drive/AMD_new"
</code></pre>
<pre><code class="lang-auto">train_data = datasets.ImageFolder(data+"/train", transform=transform_train)
test_data = datasets.ImageFolder(data+"/val", transform = transform_test)
#n_classes = test_data.shape[1]
n_classes = len(test_data.classes)
print(n_classes)

batch_size = 32

dataloader_train = torch.utils.data.DataLoader(train_data, batch_size, shuffle=True, num_workers=2)
dataloader_test = torch.utils.data.DataLoader(test_data, batch_size, num_workers=2)
</code></pre>
<p>These are 4 folders, labeled, along with images, that are uploaded into Google Drive and I am doing it from Google colab. Can anyone please tell me that how can I split the data into xtrain, xtest and ytrain and so on. Should I connect xtest with my valid folder? and xtrain with my train folder? Then what about ytrain and ytest? I am confused a little bit. Please help me to solve this. Thanks.</p>

gokulan_vikash: <p><a class="mention" href="/u/deb_prakash_chatterj">@Deb_Prakash_Chatterj</a> I want to implement the same split for my dataset. Can you please tell me how you solved this?</p>

Deb_Prakash_Chatterj: <p>Hey, use Scikit Learn <code>train_test_split</code>, like this -</p>
<pre><code class="lang-auto">xtrain, xvalid, ytrain, yvalid = train_test_split(bow[:split_num,:-1], train['label'], test_size=0.3, random_state=42)
</code></pre>
<p>Learn from this site - <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html" rel="nofollow noopener">train_test_split</a></p>
<p>Thanks</p>

gokulan_vikash: <p>I wanted to use subsetRandomSampler. You have any idea how to use that batches?</p>

Deb_Prakash_Chatterj: <p>Nope. I don’t have any idea.</p>
Unsupervised Graph Embeddings

celviofos: <p>Hi!<br>
I’m trying to do graph embedding using pytorch geometric but i can’t manage to do it in an unsupervised way. For example, the SAGE model is supposed to be trained unsupervised but all i could find was supervised implemention. I saw hat Node2Vec was trained in an unsupervised way but i’d like to apply it to the GNN models.<br>
Thank you for the help!</p>