akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Dropout and inplace

username: Vidit_Agarwal
cooked: <p>When we are using dropout what will be the difference in performance of our model if we change the inplace parameter to true instead of false. Both in terms of training and validating the model.</p>

username: ptrblck
cooked: <p>The inplace operation (assuming it’s allowed and doesn’t raise an error) would save the memory for the intermediate output activation, but would prevent potentially fusing this dropout layer with other layers if I’m not mistaken.</p>

username: Vidit_Agarwal
cooked: <p>Sorry didn’t got it can you please explain in terms of testing and validating? What does “Fusing” the dropout layer refers to?</p>

username: ptrblck
cooked: <p>Multiple operations can be fused if you are scripting the model via <code>torch.jit.script</code>. If an operation can be fused with their neighbors depends on the actual operations as well as the used fuser backend (e.g. <code>nvfuser</code> should be able to fuse dropout-add-relu etc. or other pointwise operations).</p>
<p>I’m not sure if you are referring to the training and validation loop, but inplace ops would work there in the same way with the benefit that Autograd would not complain of disallowed inplace manipulations during the validation run if the forward pass is wrapped in a <code>torch.no_grad()</code> context.</p>
Calculating the gradient through a sparse matrix into a tensor of its values

username: Valerka
cooked: <p>Hello everybody!<br>
I want to write a custom nn.Linear layer, but with weights in the form of a sparse matrix.</p>
<p>Please tell me how expensive it is to create a sparse matrix “on the fly” in order to calculate the gradient to the tensor of its values?<br>
And how can this code below be optimized?</p>
<p>If I don’t create a new sparse matrix at each iteration of the loop, then an exception is thrown after the first iteration: “RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.”</p>
<p>And I can’t beat this error in any way</p>
<p>Code example:</p>
<pre><code class="lang-auto">import torch
import torch.optim as optim


if __name__ == '__main__':
    indices = [[1], [0]]
    values = torch.FloatTensor([10.0]).requires_grad_()
    target = torch.FloatTensor(
        [
            [1],
            [2]
        ]
    )

    inputs = torch.FloatTensor(
        [
            [1],
            [2],
            [3]
        ]
    )

    optimizer = optim.Adam([values], lr=0.1)
    for _ in range(200):
        tensor = torch.sparse_coo_tensor(indices, values, (2, 3))

        y = torch.sparse.mm(tensor, inputs)

        loss = ((target - y) ** 2).sum()
        print(y.tolist(), values)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

</code></pre>
Error with pytorch xla

username: chetan06
cooked: <p>I am trying to train my model on tpu on kaggle.<br>
I used the follwoing code for installing torch_xla</p>
<pre data-code-wrap="bash"><code class="lang-bash">!curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py
!python pytorch-xla-env-setup.py --version nightly --apt-packages libomp5 libopenblas-dev
</code></pre>
<p>But while importing I am getting error</p>
<pre data-code-wrap="python"><code class="lang-python">import torch_xla.core.xla_model as xm
import torch_xla.distributed.parallel_loader as pl
import torch_xla.distributed.xla_multiprocessing as xmp
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/9/497026d4c7f44b134f9bfbe674de84150c7c469d.png" data-download-href="https://discuss.pytorch.org/uploads/default/497026d4c7f44b134f9bfbe674de84150c7c469d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/9/497026d4c7f44b134f9bfbe674de84150c7c469d_2_690x194.png" alt="image" data-base62-sha1="atF8D9AyonNH5kpVv3IG6wQcS9T" width="690" height="194" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/9/497026d4c7f44b134f9bfbe674de84150c7c469d_2_690x194.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/9/497026d4c7f44b134f9bfbe674de84150c7c469d_2_1035x291.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/9/497026d4c7f44b134f9bfbe674de84150c7c469d.png 2x" data-dominant-color="482628"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1337×377 61.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Can somebody help me understand that why I am getting this error and how can I make it work</p>
Torch Tensor with same shape but different storage size

username: Toby
cooked: <p>I’m working on GANs model, the generator creates a tensor with size <code>(3,128,128)</code> which I dumped with the pseudo-code</p>
<pre><code class="lang-python">import torch
image = Generator(noise).clone()
tensor = image[0].detach().cpu()
torch.save(tensor, save_path)
</code></pre>
<p>The problem is that tensor costs more storage than the same tensor size, even they have the same <code>shape</code> and <code>dtype</code></p>
<pre><code class="lang-auto">&gt;&gt;&gt; import sys
&gt;&gt;&gt; import torch
&gt;&gt;&gt; tensor = tensor.load(save_path)
&gt;&gt;&gt; rand_t = torch.randn(tensor.shape)
&gt;&gt;&gt; print(tensor.shape, rand_t.shape)
torch.Size([3, 128, 128]) torch.Size([3, 128, 128])
&gt;&gt;&gt; print(tensor.dtype, rand_t.dtype)
torch.float32 torch.float32
&gt;&gt;&gt; print(sys.getsizeof(tensor.storage()))
9830472
&gt;&gt;&gt; print(sys.getsizeof(rand_t.storage()))
196680
</code></pre>
<p>I tried to dump those tensors, the output from the generator took <code>9.2MB</code> and the random tensor took <code>197.4kB</code>. I did read the pytorch’s documents but found nothing. Please help me to figure out what is the difference between them?</p>

username: Toby
cooked: <p>Extracting the sub-tensor directly from the origin will bring the whole storage of the origin.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import sys
&gt;&gt;&gt; import torch
&gt;&gt;&gt; tensor = torch.randn(10,3,128,128)
&gt;&gt;&gt; sys.getsizeof(tensor.storage())
1966144
&gt;&gt;&gt; sub1 = tensor[0]
&gt;&gt;&gt; sub1.shape
torch.Size([3, 128, 128])
&gt;&gt;&gt; sys.getsizeof(sub1.storage())
1966144
&gt;&gt;&gt; sub2 = tensor[0].clone()
&gt;&gt;&gt; sub2.shape
torch.Size([3, 128, 128])
&gt;&gt;&gt; sys.getsizeof(sub2.storage())
196672
</code></pre>
<p>Therefore, in my case, I need to <code>clone</code> the <code>image</code> from the generator.:</p>
<pre><code class="lang-python">import torch
image = Generator(noise).clone() 
tensor = image[0].detach().clone().cpu() # using clone()
torch.save(tensor, save_path)
</code></pre>
<ul>
<li>There are some mistakes in my questions that might confuse the reader, I apologized for that.</li>
</ul>
How to initialize a CNN with weights from flat numpy array

username: henrikgruner
cooked: <p>Hi</p>
<p>Does anybody have a way to go back from flattening all the weights and putting them into a numpy vector (as done in the code snippet below). I trained the network for 3 days, before my allotted time expired, causing the program to cancel before converging. I have the checkpoints saved as numpy vectors. The model architecture is Resnet18 (for CIFAR10).</p>
<pre><code>gg = np.array()
for name, weight in network.named_parameters():
    gg.extend(weight.cpu().detach().numpy().flatten())
np.save("name" + str(q)+str(n), gg)
</code></pre>
<p>I suppose it is possible to manually split the numpy vector and convert it into a tensor, but any help is appreciated.</p>

username: ptrblck
cooked: <p>Yes, manually unwrapping the data should work. Something like this might help:</p>
<pre><code class="lang-python"># flatten
gg = []
for name, weight in network.named_parameters():
    gg.extend(weight.cpu().detach().numpy().flatten())
    
# restore
sd = {}
i = 0
for name, weight in network.named_parameters():
    param_len = weight.nelement()
    param = torch.tensor(gg[i:i+param_len])
    param = param.view_as(weight)
    sd[name] = param
    i += param_len
    
for (name1), (name2, param2) in zip(sd, network.named_parameters()):
    param1 = sd[name1]
    print('name {} equal: {}'.format(name1, name1==name2))
    print('param equal {}'.format((param1 == param2).all()))
</code></pre>
Which is the labels' order of the output Tensor?

username: alessandro_mondin
cooked: <p>I was trying to figure out how Pytorch sorts the labels in the output tensor. I’ve understood that with Torchvision the class_to_idx creates the order given how elements are retrieved by the ImageFolder method and this will be the idx order in the output tensor.<br>
On the other hand, as regard NLP I am very confused.<br>
Is it done while defining the network here below in un-explicit way?</p>
<pre><code class="lang-auto">class CharRNN(nn.Module):
    
    def __init__(self, tokens, n_hidden=256, n_layers=2,
                               drop_prob=0.5, lr=0.001):
        super(CharRNN, self).__init__()
        self.drop_prob = drop_prob
        self.n_layers = n_layers
        self.n_hidden = n_hidden
        self.lr = lr
        
        # creating character dictionaries
        self.chars = tokens
        self.int2char = dict(enumerate(self.chars))
        self.char2int = {ch: ii for ii, ch in self.int2char.items()}
        
        ## TODO: define the LSTM
        self.lstm = nn.LSTM(len(self.chars), n_hidden, n_layers, 
                            dropout=drop_prob, batch_first=True)
    
        self.dropout = nn.Dropout(drop_prob)
        
      
        self.fc = nn.Linear(n_hidden, len(self.chars)+1000)
        
    
    def forward(self, x, hidden):
        ''' Forward pass through the network. 
            These inputs are x, and the hidden/cell state `hidden`. '''
  
        r_output, hidden = self.lstm(x, hidden)

        out = self.dropout(r_output)
        
        # Stack up LSTM outputs using view
        # you may need to use contiguous to reshape the output
        out = out.contiguous().view(-1, self.n_hidden)

        out = self.fc(out)
        
        # return the final output and the hidden state
        return out, hidden
    
    
    def init_hidden(self, batch_size):
        ''' Initializes hidden state '''
        # Create two new tensors with sizes n_layers x batch_size x n_hidden,
        # initialized to zero, for hidden state and cell state of LSTM
        weight = next(self.parameters()).data
        
        if (train_on_gpu):
            hidden = (weight.new(self.n_layers, batch_size, self.n_hidden).zero_().cuda(),
                  weight.new(self.n_layers, batch_size, self.n_hidden).zero_().cuda())
        else:
            hidden = (weight.new(self.n_layers, batch_size, self.n_hidden).zero_(),
                      weight.new(self.n_layers, batch_size, self.n_hidden).zero_())
        
        return hidden
        
</code></pre>

username: ptrblck
cooked: <p>In your code snippet you are creating th character dictionary and mapping via the <code>tokens</code> input so an index will be assigned to each character by iterating it as seen here:</p>
<pre><code class="lang-python">tokens = ['c', 'a', 'z', '0', 'b']

chars = tokens
int2char = dict(enumerate(chars))
char2int = {ch: ii for ii, ch in int2char.items()}
# &gt; {'c': 0, 'a': 1, 'z': 2, '0': 3, 'b': 4}}
</code></pre>
SOTA techniques in image segmentation & detection, *medical images

username: YASJAY
cooked: <p>Looking for inputs on state of the art work in the detection and segmentation work for medical images.</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>I would check if e.g. <a href="https://monai.io/">MONAI</a> provides some desired models, which is part of the PyTorch ecosystem and deals with medical datasets.</p>
Can you select a specific output from a layer with mutiple out nodes?

username: pumplerod
cooked: <p>I’m curious to know the best method for creating a “branch” from a linear layer.</p>
<p>For example… If I have:</p>
<pre><code class="lang-auto">root_node = torch.nn.Linear( 1, 2)
</code></pre>
<p>the node takes 1 input and produces 2 outputs.  Can I access each of those outputs independently to create a branch like structure?  Something like:</p>
<pre><code class="lang-auto">branch_1 = torch.nn.Linear( 1, 2)
branch_2 = torch.nn.Linear( 1, 2)

X = root_node( input_data)
b1 = branch_1( X[0].view(-1,1))
b2 = branch_2( X[1].view(-1,1))
</code></pre>
<p>or something to that effect?</p>
<p>This doesn’t result in an error, but I’m not sure if I’d be thoroughly messing up something with the gradient.</p>

username: ptrblck
cooked: <p>Yes, indexing is tracked by Autograd and you can pass the indexed slices to other modules.</p>