akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Split input sequence based on model training loss

username: leolcling
cooked: <p>Hi,</p>
<p>When processing long input sequences in seq2seq model, RNN/ transformer type models often perform worse. I want to implement a remedy by splitting the input sequences into smaller sequences (divide and conquer). And I want to find the splitting position by the loss of the seq2seq task.</p>
<p>So I want:<br>
Input -&gt; splitting layer -&gt; Output -&gt; encoder -&gt; decoder -&gt; loss</p>
<p>e.g.<br>
[[ [1,1,1], [2,2,2], [3,3,3], [4,4,4] ]] (1 x 4 x 3) [(batch x len x feature)] -&gt; splitting layer -&gt; [ [[1,1,1] , [2,2,2], [3,3,3]], [[4,4,4]] ] (2x3x3 after padding)</p>
<p>My idea is to use BiLSTM to find the splitting position, and optimize BiLSTM model by seq2seq model output, but I am having trouble understanding</p>
<ol>
<li>How the loss back-propagate to the splitting layer?</li>
<li>How to reshape and pad after finding the split position?</li>
</ol>
<p>Thanks!</p>
Attempting to reduce memory consumption by fusing Conv1d with Maxpool1d

username: antimora
cooked: <p>Currently I am working on a speech recognition project. I am currently experimenting with SincNet (<a href="https://github.com/mravanelli/SincNet" rel="nofollow noopener">https://github.com/mravanelli/SincNet</a>) architecture for extracting speech features. SincNet layer essentially a Conv1D with some kernel window param optimization. In this architecture the conv layer is followed by a max pool layer.</p>
<p>Here is a problem I am facing. The input for SinceNet is a waveform signal with 1 channel. I have about 15 seconds of wav with 16K bitrate, which amounts to 24K params. The SincNet produces 64 channels (window size and strides are 1). Having batch size of 32 would explode the number of params to 49M. The max pool layer would reduce this number of params but I would like to max pooling immediately after convolution is applied for each channel.</p>
<p>What is your recommended take on this problem?</p>
<p>One obvious solution is to operate plane by plane (channel) and stack the results. However, I am not familiar with Pytorch’s memory management and async nature of Cuda operations.</p>
RuntimeErrror: one of the variables needed for gradient computation has been modified by an inplace operation

username: Mactarvish
cooked: <p>Hi, I’m trying to add a ReLU layer to PART of my model’s output, here is my code:</p>
<pre><code class="lang-auto">import torch.nn as nn
import torch

input = torch.ones(2).cuda()
linear = nn.Linear(2, 2).cuda()
output = linear(input)
output[0] = nn.ReLU(inplace=False)(output[0])
loss = torch.sum(output)
loss.backward()

</code></pre>
<p>but I got the error:</p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor []], which is output 0 of SelectBackward, is at version 3; expected version 2 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>
<p>I’m sure it was caused by this row:</p>
<pre><code class="lang-auto">output[0] = nn.ReLU(inplace=False)(output[0])

</code></pre>
<p>but why is this an inplace operation? How should I modify part of the output in a differentiable way? Thanks in advance.</p>

username: ptrblck
cooked: <p>Since you are assigning the result to the same tensor, it’s considered an inplace operation.<br>
You could instead create a new result tensor, using <code>torch.cat</code> and slicing:</p>
<pre><code class="lang-python">input = torch.ones(2)
linear = nn.Linear(2, 2)
output = linear(input)
output1 = nn.ReLU(inplace=False)(output[0])
output = torch.cat((output1.unsqueeze(0), output[1:]))
loss = torch.sum(output)
loss.backward()
</code></pre>

username: Mactarvish
cooked: <p>Thanks for your reply, and your solution just works.<br>
But there’s another question, if I modify my code:</p>
<pre><code class="lang-auto">import torch.nn as nn
import torch

input = torch.ones(2).cuda()
linear = nn.Linear(2, 2).cuda()
output = linear(input)
# output[0] = nn.ReLU(inplace=False)(output[0])
output[0] = output[0] * 2 + 1
output[0] = output[0] + output[1]
loss = torch.sum(output)
loss.backward()
</code></pre>
<p>Similarly,  I assigned the result to the same tensor in line 8 and line 9, but this code works well. What’s the different between the two versions?<img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=9" title=":thinking:" class="emoji" alt=":thinking:"></p>

username: ptrblck
cooked: <p>The error is thrown, if the output of the computation is needed for the backward pass, as explained <a href="https://discuss.pytorch.org/t/when-would-the-error-one-of-the-variables-needed-for-gradient-computation-has-been-modified-occur/11976/4">here</a> (the linked topic doesn’t exactly use your model, but might give you an idea).</p>

username: Mactarvish
cooked: <p>Thanks. I’ve got it.</p>
Input for LSTMs or GRU

username: mojzaar
cooked: <p>Hi,</p>
<p>I have a 3d data as follows:</p>
<p>Batch_size, num_channels, num_time_series, h, w = data.size()</p>
<p>I want to pass this data through LSTM layer but it just take batch size, number of time series, and size of data. I can flat the data to get a 1d data but I don’t know how can I interpret the number of channels. Does it make sense to flat all h, w, and num_channels?</p>

username: matthias.l
cooked: <p>You don’t need to flat your data. LSTM can consume your 3d data.</p>
<p>See <a href="https://gist.github.com/lochbrunner/d9c77c3fbfd5301154c070561e294ed7#file-bieber_lstm-py-L129" rel="nofollow noopener">https://gist.github.com/lochbrunner/d9c77c3fbfd5301154c070561e294ed7#file-bieber_lstm-py-L129</a><br>
Where <code>x</code> is the 3d data. of dimension <code>batch_size x sequence_length x embedding_dim</code></p>

username: mojzaar
cooked: <p>Thanks for your reply.</p>
<p>Actually, I have a conv3d layer before this LSTM layer which the Conv’s output channel is not equal to one. You can assume that we have ‘n’ channels where each channel has a 3d data (1 dimension is for sequence and other dimensions are the number of row and column). The input of LSTM should be 3 dimensions. The first one is batch size, the second one is the sequential dimension, and the third one is the size of the input.  Batch size and sequence should be equal to what I have from the original data. It makes sense to flat the data in row and column direction. I cannot understand how can I deal with the corresponding dimension with the number of channels. Should I create a loop and for each channel assign an LSTM layer or I have to consider my LSTM input as input.size() = batch_size, seq_len, num_channels * w * h.</p>
Softmax dimension out of range with parallel training

username: woodsboy
cooked: <p>HI, I try to use F.softmax(x, dim=1) with x is a 2-dim tensor on GPU, the code works well while I use a single GPU, but I get an error message ‘IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)’ after I use multi-GPU devices to train the network. could you help me to fix the issue?</p>
Autoencoder latent features into softmax layers

username: SU801T
cooked: <p>Hi,</p>
<p>I have extracted the latent representation from an autoencoder from a test dataset. I was wondering whether it made sense to simply pass these features into a softmax layer with no further training. As a note, I used training data and validation data to train/test the autoencoder and only extracted the latent layer features from the test set.</p>
<p>Cheers,<br>
T</p>
Ignoring Channels in Loss for Transfer Learning

username: aksg87
cooked: <p>I would like to transfer learn (or perhaps I should say combine) one vision problem to another similar problem. For the first problem, I have segmentations for the liver and spleen (Dataset 1), and in the second problem, I have segmentations only for the kidneys (Dataset 2).</p>
<p>I was planning on having 4 channels: Liver, Spleen, Right_Kidney, Left_Kidney. I wanted to utilize a model trained on Dataset 1 ignoring the kidney class, and then train this model on Dataset 2 incorporating the kidney data. (Knowledge of the Liver and Spleen is certainly helpful to spatially locate kidneys so the added data should ideally enhance segmentation of the kidneys).</p>
<p>If anyone has a reference on combining problems or ignoring channels in the loss function selectively, please let me know!</p>
Validation accuracy drops after loading model from checkpoint

username: Danny1
cooked: <p>I use this code to save my VAE model every epoch:</p>
<pre><code class="lang-auto">torch.save({'epoch' : epoch,
            'encoder' : encoder.state_dict(),
            'decoder' : decoder.state_dict(),
            'property_predictor_model' : property_predictor_model.state_dict(),
            'train_ids' : train_ids,
            'valid_ids' : valid_ids,
            'validationQuality' : validationQuality
}, '{}/model_checkpoint{}.pt'.format(out_dir, epoch))
</code></pre>
<p>And when I restart training, I use this code to initialize the model and to load the previous state:</p>
<pre><code class="lang-auto">model_encode = VAE_encode(**encoder_parameter).to(device)
model_decode = VAE_decode(**decoder_parameter).to(device)
model_prop_predict = property_predictor_model(**prop_pred_parameter).to(device)

model_encode.load_state_dict(checkpoint['encoder'])
model_decode.load_state_dict(checkpoint['decoder'])
model_prop_predict.load_state_dict(checkpoint['property_predictor_model'])
</code></pre>
<p>The Validation accuracy saved in the checkpoint itself would be around 60% yet running validation immediately after loading the model returns an accuracy of &lt;10%. Training continues as if I just initialized the model randomly. The Validation accuracy before and after i load the state is different, so it’s loading something, just not the right thing. Any ideas why this is happening.</p>
<p>Interesting note: If I use my debugger and load the checkpoint right after it gets saved, the validation accuracy is maintained, but if I cancel my script and load up the same checkpoint, accuracy is &lt;10% again.</p>
This type of loss curve for training and validation is normal?or there is something wrong?

username: sampa
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/a/8a3e7ca0816e8eb4a7e3c44ed044f69eedac429a.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/8a3e7ca0816e8eb4a7e3c44ed044f69eedac429a" title="Loss1_Resnet50_1000"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/a/8a3e7ca0816e8eb4a7e3c44ed044f69eedac429a_2_690x409.jpeg" alt="Loss1_Resnet50_1000" data-base62-sha1="jIXLDrpuStkzSVoLHPz20Bjdlxw" width="690" height="409" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/a/8a3e7ca0816e8eb4a7e3c44ed044f69eedac429a_2_690x409.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/8/a/8a3e7ca0816e8eb4a7e3c44ed044f69eedac429a.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/a/8a3e7ca0816e8eb4a7e3c44ed044f69eedac429a.jpeg 2x" data-dominant-color="F4E6D7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Loss1_Resnet50_1000</span><span class="informations">805×478 82.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: alex.veuthey
cooked: <p>That looks very normal for a simple setup. You could apply moving average on your values, it might be easier to tell what is happening. Also look up <a href="https://pytorch.org/docs/stable/tensorboard.html" rel="nofollow noopener">tensorboard</a> for logging training scalars such as loss over time (it can also show you many different things).<br>
I think you do not use scheduling for the learning rate, correct? That might improve your results a lot.</p>

username: sampa
cooked: <p>Thanks for your reply. I did not use scheduling for the learning rate. How to apply moving average?</p>
Sparse tensors for training data

username: Subhankar_Ghosh
cooked: <p>My training data is like:</p>
<pre><code class="lang-auto">x_train = torch.tensor(torch.from_numpy(np.random.randint(100, size=(1000, 25))), dtype=torch.double)
</code></pre>
<p>where each row is a sample and thus we have 1000 samples.</p>
<p>Now I need to have the training data such that for each of the sample/row there can be at max 3 non-zero elements.</p>
<p>Can you all please suggest how I can implement that? Thanks!</p>
Fold patches of images back to single image

username: agadetsky
cooked: <p>I have function that translates image into puzzles using <code>unfold</code>:</p>
<pre><code class="lang-auto">def make_jigsaw_puzzle(x, grid_size=(2, 2)):
    # x shape is C x H x W
    C, H, W = x.size()

    assert H % grid_size[0] == 0
    assert W % grid_size[1] == 0

    C, H, W = x.size()
    x_jigsaw = x.unfold(1, H // grid_size[0], W // grid_size[1])
    x_jigsaw = x_jigsaw.unfold(2, H // grid_size[0], W // grid_size[1])
    x_jigsaw = x_jigsaw.contiguous().view(-1, C,  H // grid_size[0], W // grid_size[1])
    return x_jigsaw
</code></pre>
<p><code>x_jigsaw</code> shape is <code>grid_size[0] * grid_size[1] x C x H x W</code>.<br>
How can I get image from patches back?</p>

username: ptrblck
cooked: <p><a href="https://discuss.pytorch.org/t/fold-unfold-to-get-patches/53419/6">This post</a> might help.</p>

username: agadetsky
cooked: <p>Thank you! It works:</p>
<pre><code class="lang-auto">def jigsaw_to_image(x, grid_size=(2, 2)):
    # x shape is batch_size x num_patches x c x jigsaw_h x jigsaw_w
    batch_size, num_patches, c, jigsaw_h, jigsaw_w = x.size()
    assert num_patches == grid_size[0] * grid_size[1]
    x_image = x.view(batch_size, grid_size[0], grid_size[1], c, jigsaw_h, jigsaw_w)
    output_h = grid_size[0] * jigsaw_h
    output_w = grid_size[1] * jigsaw_w
    x_image = x_image.permute(0, 3, 1, 4, 2, 5).contiguous()
    x_image = x_image.view(batch_size, c, output_h, output_w)
    return x_image

</code></pre>
What is different between train=False and train=True

username: hunar
cooked: <p>what is different between the data in training.pt and test.pt ?<br>
does mean that (train=True) download a dataset  been trained by default , and (train=False) a un trained dataset ?</p>
<p>trainset = datasets.FashionMNIST("~/.pytorch/F_MNIST_data",download=True , train=True , transform=transform)</p>
<p>testset = datasets.FashionMNIST("~/.pytorch/F_MNIST_data" , download=True , train=False, transform=transform)</p>
<p>thanks</p>

username: beaupreda
cooked: <p>The split of the data is different i.e. not the same data is loaded as explained in the <a href="https://pytorch.org/docs/stable/torchvision/datasets.html#fashion-mnist" rel="nofollow noopener">docs</a></p>
Memory Consuption

username: aldeka12
cooked: <p>How to check GPU memory consumption of the model on a given input and also size of the model? torchsummary do not work properly in my case.</p>

username: albanD
cooked: <p>Hi,</p>
<p>Unfortunately, this is very hard to predict upfront.<br>
If you want to have an exact value, to know the max memory consumption (including autograd buffers), the best way is to send your model to the gpu and do a forward backward.</p>

username: aldeka12
cooked: <p>Hello, thanks for quick answer! In this situation, I only need GPU memory consumption of forward pass. Also, is there a way to know total size of the model (using special methods or by using number of parameters)?</p>

username: albanD
cooked: <p>Unfortunately, modules can save arbitrary elements during forward. So it is really hard to have a common api that computes this.<br>
If torchsummary does not work, you most likely will have to run a forward and check the actual consumption.</p>

username: aldeka12
cooked: <p>Thanks! Is there any tutorials to how manage this correctly (run a forward and check the actual consumption)?</p>

username: albanD
cooked: <p>You can use regular python tools to check this as you are looking for cpu memory.<br>
For example, this memory profiler gives very nice outputs: <a href="https://pypi.org/project/memory-profiler/">https://pypi.org/project/memory-profiler/</a></p>

username: TinfoilHat0
cooked: <aside class="quote no-group" data-username="aldeka12" data-post="3" data-topic="61374">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aldeka12/48/11871_2.png" class="avatar"> aldeka12:</div>
<blockquote>
<p>on, I only need GPU memory consumption of forward pass. Also, is there a way to know total size of the model (using special methods or by using number of parameters)?</p>
</blockquote>
</aside>
<p>You can monitor the status of your GPU via <code>nvidia-smi</code>.<br>
E.g., you can create a simple script that only passes your model to GPU and then waits. Monitoring the GPU usage of such script may give you an idea.</p>
What is the backward function for a max pooling layer?

username: MahdiNazemi
cooked: <p>I am trying to understand the behavior of max pooling during forward and backward propagation (especially backward). Unfortunately, I have trouble making sense of the <a href="https://github.com/pytorch/pytorch/blob/433baf1b9067774aecdeef8e32664827779ddb86/torch/csrc/api/src/nn/modules/pooling.cpp" rel="nofollow noopener"><code>C++</code> source</a>.</p>
<p>I would really appreciate if you explain what is stored during the forward pass (if anything) and how such information is used during the backward pass. A short snippet of code in Python would be very much appreciated.</p>
<p>Thanks!</p>

username: albanD
cooked: <p>Hi,</p>
<p>The max pooling layer uses the maximum value out of all the ones in the kernel. So the gradient is 1 for the selected value and 0 for all the others.<br>
So during the backward, the gradient of the output is (multiplied by 1 and) set to the selected value. All the others are set to 0.<br>
To do so, it uses the indices returned by the forward pass.</p>
<p>The cpu implementation is <a href="https://github.com/pytorch/pytorch/blob/81bf73643b6552a63794fb889238aaf0c2a7baa6/aten/src/ATen/native/DilatedMaxPool2d.cpp#L231-L267">here</a> with gradInput initialized to zeros.</p>

username: MahdiNazemi
cooked: <p>Thank you for your answer.</p>
<p>The only question that remains is how such indices help when training in batches? My understanding is that the total loss value at the output is the mean of individual loss values for each sample in the minibatch.</p>
<p>With your explanation, it sounds like we need to backpropagate the loss value for each individual sample in the minibatch so that we can decide which indices should have a zero gradient and which ones should be multiplied by one.</p>
<p>P.S. In the source code, the name of the function includes <code>single_out_frame</code>, which might mean that in fact, the samples are processed one at a time during the backward pass.</p>

username: albanD
cooked: <p>You have one set of indices per element in the batch.<br>
As you noted, the function I linked only processes one sample, the for loop that runs through the batches is just below <a href="https://github.com/pytorch/pytorch/blob/81bf73643b6552a63794fb889238aaf0c2a7baa6/aten/src/ATen/native/DilatedMaxPool2d.cpp#L283">here</a>.</p>
How can I get atttr like stride or padding of Conv2d in cpp ScriptModule

username: yy738686337
cooked: <p>I instantiated a model and transfrom it into ScriptMoudle using this:</p>
<pre><code class="lang-auto">alex_model = torchvision.models.AlexNet(num_classes=10)
f = torch.jit.script(alex_model)
f.save("./models/alexnet.pt")
</code></pre>
<p>I cannot get the attr like stride of Conv2d when I <code>dump()</code> the model in c++<br>
only a constant name like this:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/0/50b7501a238ae343e1737fc3787cb26cebbfcf5a.png" data-download-href="https://discuss.pytorch.org/uploads/default/50b7501a238ae343e1737fc3787cb26cebbfcf5a" title="12"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/50b7501a238ae343e1737fc3787cb26cebbfcf5a_2_690x120.png" alt="12" data-base62-sha1="bw2WvCBDE1UyVnPdVDzbZOJURoC" width="690" height="120" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/50b7501a238ae343e1737fc3787cb26cebbfcf5a_2_690x120.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/50b7501a238ae343e1737fc3787cb26cebbfcf5a_2_1035x180.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/50b7501a238ae343e1737fc3787cb26cebbfcf5a_2_1380x240.png 2x" data-dominant-color="222222"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">12</span><span class="informations">1922×336 15.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>How can I get the value like stride or padding of <code>Conv2d</code> when loading the model in c++?<br>
THANKS VERY MUCH!</p>

username: driazati
cooked: <p>Constant values are inlined directly into the code of the module, so we currently have no way of accessing them once compilation has run, and the modules in <code>nn.Module</code>s store all their values as constants. I opened <a href="https://github.com/pytorch/pytorch/issues/30019" rel="nofollow noopener">this issue</a> so you can track that for fixes.</p>
<p>As a temporary workaround, you could grab the values you care about in Python and stash them as <a href="https://pytorch.org/docs/master/jit.html#module-attributes" rel="nofollow noopener">attributes</a> instead of <a href="https://pytorch.org/docs/master/jit.html#python-defined-constants" rel="nofollow noopener">constants</a>:</p>
<pre><code class="lang-python">class M(nn.Module):
    def __init__(self):
        super().__init__()
        self.alex_model = torchvision.models.AlexNet(num_classes=10)
        self.first_conv_stride = getattr(self.alex_model.features, '0').stride

    def forward(self, x):
        return self.alex_model(x)


sm = torch.jit.script(M())
print(sm.first_conv_stride)
</code></pre>
<p>and in C++</p>
<pre><code class="lang-cpp">auto module = torch::jit::load("alexnet.pt");
std::cout &lt;&lt; module.attr("first_conv_stride") &lt;&lt; "\n";
</code></pre>
How much is learned from an image?

username: alx
cooked: <p>Hey guys I was wondering if it was possible to get a score of how much was learned from an image. This could help discard useless images or noisy ones where nothing is learned.</p>
<p>Thanks</p>

username: alx
cooked: <p>Any ideas? Couldn’t find anything online</p>

username: TinfoilHat0
cooked: <p>The methods they use in the following paper might help perhaps.</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://static.arxiv.org/static/browse/0.2.6/images/icons/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://arxiv.org/abs/1703.04730" target="_blank" rel="nofollow noopener">arXiv.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail">

<h3><a href="https://arxiv.org/abs/1703.04730" target="_blank" rel="nofollow noopener">Understanding Black-box Predictions via Influence Functions</a></h3>

<p>How can we explain the predictions of a black-box model? In this paper, we
use influence functions -- a classic technique from robust statistics -- to
trace a model's prediction through the learning algorithm and back to its
training data, thereby...</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: vipin14119
cooked: <p>but what if the images you mark as “not worthy” might by limitations of your model not the dataset ?</p>

username: alx
cooked: <p>I don’t mind! it’ll be model dependent. The question is, for my model, what are the important images?</p>
How do I calculate batches to pull full dataset when using a random weighted sampler?

username: Jordan_Howell
cooked: <p>Hello,</p>
<p>From this forum, I learned how to use the random weighted sampler to pull more of the minority class than the majority class.  How would I calculate the batches correctly to make sure all of my images are used?</p>
<p>Here is my code for the weighted sampler:</p>
<pre><code class="lang-auto">batch_size = 10
class_sample_count = [46699, 17739] 
weights = 1 / torch.Tensor(class_sample_count)
sample_weights = weights[target]
weights = weights.double()
sampler = torch.utils.data.sampler.WeightedRandomSampler(
    weights=sample_weights,
    num_samples=len(sample_weights),
    replacement=True)
#trainloader = torch.utils.data.DataLoader(trainDataset, batch_size = batch_size, sampler = sampler)

train_loader = DataLoader(roof, batch_size=10, sampler = sampler)
</code></pre>
<p>So with a batch size of 10, I would run 6444 batches per epoch to get all of the images through.  How would I do it with a random weighted sampler?</p>

username: albanD
cooked: <p>You want to see all images at least once?<br>
I’m not sure you can be sure of this. as you could set a very small probability on one sample which would make it very unlikely to be drawn.</p>

username: Jordan_Howell
cooked: <p>Thank you.  What if I want to see the positive class at least one (the minority class).  Is that essentially, what it’s doing?</p>

username: albanD
cooked: <p>I don’t think you can get such guarantees.<br>
But you can check the <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.WeightedRandomSampler">doc</a>. It will sample the number of elements you asked for with the given probabilities. So in theory anything can happen. But if you sample enough samples, you can be sure it will be enough. Note that <code>num_samples</code> does not have to be the size of <code>sample_weights</code>.</p>

username: Jordan_Howell
cooked: <p>Ok.  Last question then (on this topic <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"> )  Can I artificially set how I want the model to sample?  So if I wanted equally sampling amongst the positive and negative class, Do Set sample_weights to <code>[.5, .5]</code> and that should pull equal amounts of positive/negative samples regardless if I have 3 to 1 negative over positive samples?</p>

username: albanD
cooked: <p>You cannot provide weights for specific classes right?<br>
You need to set it for each sample. So you can give <code>1/num_samples</code> to each sample to get a uniform sampling.<br>
But with the weights you have in your first post, it should draw the same number of each classes as your weight is proportional to the number of elements available for that class.</p>
Pretrained model as encoder

username: LizLima
cooked: <p>Hi,<br>
I implemented a autoencoder , and use pretrained model resnet as encoder and the decoder is a series of convTranspose.<br>
Is necessary to apply “init_weights” to autoencoder?<br>
If I use “init_weights” the weights of pretrained model also modified?</p>
<p>Ty.</p>

username: luanpham
cooked: <p>You can create a separate class for a decoder and init_weights inside this class. Then import it to autoencoder class.</p>
<p>For examples,</p>
<pre><code class="lang-auto">class Encoder(nn.Module):
    def __init__(self):
        # load pretrained here

    def forward(self, x):
        # do the thing of encode 

class Decoder(nn.Module):
    def __init__(self):
        # define weigths
        # init weight as random

    def forward(self, x):
        # do the thing to decode


class AutoEncoder(nn.Module):
    def __init__(self):
        self.enc = Encoder()
        self.dec = Decoder()

    def forward(self, x):
        x = self.enc(x)
        return self.dec(x)
</code></pre>
Pytorch Deallocates Data in C++ Extension While Using It?

username: Christopher_Penn
cooked: <p>I’m using CUDAExtension from torch.utils.cpp_extension to build a CUDA extension. In the C++ CPU function called by Python (bound using Pybind), I allocate and initialize a device pointer, then use the counter in kernels, then free it.</p>
<p>Sometimes, the counter will become garbage while it’s being used, and print as garbage after use. Other times, it will execute correctly on the exact same data.</p>
<p>It seems more likely to become garbage when there is more GPU memory in use.</p>
<p>Can PyTorch garbage-collect GPU data while a pointer referencing it is still in-scope?</p>
<p>If so, how do I tell PyTorch not to deallocate that data? How can PyTorch call any external library (like CUDNN) without deallocating its GPU data?</p>
<p>This is not a minimal case, but a conceptual example:</p>
<pre><code class="lang-auto">std::vector&lt;torch::Tensor&gt; somefunction(some inputs){
    int* counter;
    cudaMalloc( (void **)&amp;counter, sizeof(int) );
    cudaMemset( counter, 0, sizeof(int) );

    // Use counter in kernels, where it is incremented

    int h_counter[1];
    cudaMemcpy( h_counter, counter, sizeof(int), cudaMemcpyDeviceToHost );

    printf("counter is now garbage: %i\n", *h_counter);

    cudaFree(counter);

    return {some tensor};
}
</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>Pytorch won’t touch any data that you allocated yourself.<br>
Are you sure that you use it properly in your kernels and that you have the proper sync before reading its value?</p>
CUDA Extension is Much Slower Than nn.Conv2d?

username: Christopher_Penn
cooked: <p>I’m writing a CUDA extension with <code>CUDAExtension</code> from <code>torch.utils.cpp_extension</code>. When I benchmark the write speeds for 1,000,000 objects, it seems very slow, where a million writes takes ~2ms:</p>
<pre><code class="lang-auto">__global__ void write_test( int* edges, int size)
{
    int thid = blockIdx.x * blockDim.x + threadIdx.x;
    if (thid &gt;= size){ return; }
    else { edges[thid] = 23; }
}

// Entry point cpu code for python
void some_func()
{
    int size = 1000000;

    int* edges;
    cudaMalloc((void **)&amp;edges, (size)*sizeof(int) );

    write_test&lt;&lt;&lt;dim3(1024), dim3(1024)&gt;&gt;&gt;( edges, size );
}
</code></pre>
<p>In contrast, timing a nn.Conv2d with similar numbers of inputs takes less than 0.5ms:</p>
<pre><code class="lang-auto">def test_conv(self):
    device = torch.cuda.current_device()
   
    mod = torch.nn.Conv2d(32,32,3).cuda().eval()

    data = t.randint(0,255,[40,32,32,32], dtype=t.float, device=device)

    torch.cuda.synchronize(device)
    tic = time.time()
    out = mod(data)
    t.cuda.synchronize(device)
    print(time.time() - tic)
</code></pre>
<p>The time to convolve and then write the outputs includes however many multiplications. Is there some way to write to global memory on the GPU faster?</p>

username: albanD
cooked: <p>How do you time your code?<br>
It contains a <code>cudaMalloc</code> which are known to be slow. This is why we actually have a custom cuda allocator in pytorch.<br>
Also, I am not sure your launch parameters are optimal. You might want to do a bit more in each thread and not launch a million of them.</p>
PyTorch C++ API missing headers?

username: RobbinDG
cooked: <p>Hello!</p>
<p>I am a newer user of pytorch, and am currently constructing a CNN. I am using a Sequential object with a number of convolutional layers, and as such want to add some max pooling. However, my editor (CLion) tells me that the <code>torch::nn::MaxPool1D</code> class does not exist. I looked at the docs, and apparently it is supposed to be defined in a file called <code>....nn/modules/pooling.h</code>, which does not appear in my copy of the sources. I have many others in the same directory, such as <code>conv</code>, <code>dropout</code> and more.<br>
I am able to use <code>torch::nn::Conv1d</code> and <code>torch::nn::Functional</code> successfully, so what could be the reason for missing the pooling layers?</p>
<p>Kind regards</p>

username: yf225
cooked: <p>Are you using PyTorch 1.3? <code>torch::nn::MaxPool1d</code> should be in the 1.3 version.</p>

username: RobbinDG
cooked: <p>The <code>build-version</code> file tells me I have version <code>1.3.0.dev20190820+cpu</code>, which is a slightly older version I believe, but its still 1.3.0.</p>

username: RobbinDG
cooked: <p>I just went through the trouble of upgrading to the most recent stable version which I got from the home-page (The one that I downloaded from the installation help page was nightly version which did not compile). I checked with the source code on github that it indeed contained the Pooling functions. Glad to confirm that it does indeed work now. Thanks for the help!</p>
Cuda runtime error when using nn.Embedding

username: matthias.l
cooked: <p>The GIST <a href="https://gist.github.com/lochbrunner/d9c77c3fbfd5301154c070561e294ed7" class="inline-onebox" rel="noopener nofollow ugc">Using pytorch, LSTM, mini-batches and DataSets to train a toy model. This GIST is inspired by https://gist.github.com/williamFalcon/f27c7b90e34b4ba88ced042d9ef33edd but trying to be complete, working and a bit more simpler than the orig. Additionaly it uses torch datasets. · GitHub</a> is (for CPU yet) a running example using DataSets, minibatches and a RNN.<br>
But when I want to run it on the GPU with CUDA I get the error message:</p>
<blockquote>
<p>Traceback (most recent call last):<br>
File “./main.py”, line 174, in <br>
x = model(x, l)<br>
File “/home/matthias/.local/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 493, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “./main.py”, line 126, in forward<br>
x = self.embedding(x)<br>
File “/home/matthias/.local/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 493, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/matthias/.local/lib/python3.7/site-packages/torch/nn/modules/sparse.py”, line 117, in forward<br>
self.norm_type, self.scale_grad_by_freq, self.sparse)<br>
File “/home/matthias/.local/lib/python3.7/site-packages/torch/nn/functional.py”, line 1506, in embedding<br>
return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)<br>
RuntimeError: Expected object of backend CUDA but got backend CPU for argument <span class="hashtag-raw">#3</span> ‘index’</p>
</blockquote>
<p>I just transfered the ignore_index variable to CUDA but it did not help.<br>
Does anyone have an Idea?</p>

username: albanD
cooked: <p>Hi,</p>
<p>The <code>.to()</code> operator on Tensors is not inplace.<br>
You need to do <code>x = x.to(device)</code> to get the gpu Tensor.</p>
Are inplace operations faster?

username: Beinan_Wang
cooked: <p>(1) x = a * x + b<br>
(2) x.mul_(a); x.add_(b)</p>
<p>is (2) faster than (1)?</p>

username: tom
cooked: <p>Mostly not and you don’t get JIT optimisations.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Beinan_Wang
cooked: <p>So (1) can get JIT optimizations?</p>

username: tom
cooked: <p>On the GPU (1) will get fused into a single kernel (instead of two) by the JIT, saving you 1 of 4 reads and 1 of 2 writes by not writing the intermediate a*x to memory. A computation like this on sizeable inputs is memory bound, so your saving significantly (if that is a significant part of your computation).</p>
<p>Best regards</p>
<p>Thomas</p>

username: Beinan_Wang
cooked: <p>Than you very much. Is there any guide on how to write fast PyTorch code like what you said?</p>

username: tom
cooked: <p>But so there are bits of advice in various places, and it might depend on what you want to achieve where the best place to look is.<br>
NVidia’s <a href="https://devblogs.nvidia.com/mixed-precision-training-deep-neural-networks/" rel="nofollow noopener">dev blog (to pick a random article)</a>, the <a href="https://pytorch.org/blog/optimizing-cuda-rnn-with-torchscript/" rel="nofollow noopener">PyTorch article on LSTMs</a>, I tried to blog a bit about <a href="http://lernapparat.de/more-jit-optimizations/" rel="nofollow noopener">optimizations</a> and <a href="http://lernapparat.de/sinkhorn-kernel/" rel="nofollow noopener">a common reduction code pattern</a>…</p>
<p>In the end, you probably want to benchmark optimizations and gather some experience around what works and does not work.</p>
<p>Also, you can always look here at the forums or try to ask with concrete code. We have some cuda questions here every once in a while and I certainly have learnt a lot about GPU programming from people around PyTorch.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Beinan_Wang
cooked: <p>Thank you very much!</p>
Modify the computation graph on-the-fly?

username: Liangzhi_Tang
cooked: <p>Dear all,<br>
How to achieve the following goal:</p>
<ol>
<li>Trace (or record) the computation graph (e.x, conv2d, element-wise add)</li>
<li>Modify the "element-wise add " to “whatever op” I want</li>
</ol>
<p>Please let me know if there is a way to accomplish this? Thank you very much!</p>

username: albanD
cooked: <p>Hi,</p>
<p>We do not provide any way for the user to modify the computational graph.<br>
This is mainly because it is merely an implementation detail of the autograd engine and can (and do) change regularly.</p>
<p>What is the application for which you want to be able to do such things?</p>

username: Liangzhi_Tang
cooked: <p>Thank you for the reply. Suppose we have z = y + x and we want to do z = f(y) + f(x) (f is a customized nn.Module) everywhere in the network forward. How can we achieve that?</p>

username: albanD
cooked: <p>It depends what your original network is. But you most likely only need to modify the <code>.forward()</code> method on the original network and add the appropriate calls to your new module.<br>
If you could share how the original function is defined, that would help.</p>
Adapting Pytorch tutorial “NMT from Scratch…” for dynamic RNN

username: Dean_Sumner
cooked: <p>I want to adapt the tutorial for a dynamic RNN. I have written about the issue extensively on stackexchange. However these forums may be more appropriate home for such a question:<br>
<aside class="onebox stackexchange">
  <header class="source">
      <a href="https://datascience.stackexchange.com/questions/63354/adapting-pytorch-tutorial-nmt-from-scratch-for-dynamic-rnn" target="_blank" rel="nofollow noopener">datascience.stackexchange.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://datascience.stackexchange.com/users/67703/dean" target="_blank" rel="nofollow noopener">
    <img alt="Dean" src="https://lh6.googleusercontent.com/--dHAAPcC2jo/AAAAAAAAAAI/AAAAAAAAABo/KKGfnOUxfy8/photo.jpg?sz=128" class="thumbnail onebox-avatar" width="128" height="128">
  </a>
<h4>
  <a href="https://datascience.stackexchange.com/questions/63354/adapting-pytorch-tutorial-nmt-from-scratch-for-dynamic-rnn" target="_blank" rel="nofollow noopener">Adapting Pytorch tutorial "NMT from Scratch..." for dynamic RNN</a>
</h4>

<div class="tags">
  <strong>neural-network, deep-learning, lstm, pytorch</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://datascience.stackexchange.com/users/67703/dean" target="_blank" rel="nofollow noopener">
    Dean
  </a>
  on <a href="https://datascience.stackexchange.com/questions/63354/adapting-pytorch-tutorial-nmt-from-scratch-for-dynamic-rnn" target="_blank" rel="nofollow noopener">03:41PM - 18 Nov 19 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>Since I am having heaps of trouble to get this done, I would politely like to ask for any help from the community</p>
<p>kind regards,</p>
<p>Dean</p>
Multi-view lstm

username: Debasish_Jyotishi
cooked: <p>Hello,<br>
I want to implement the multi-view lstm of," Extending Long Short-Term Memory for Multi-View Structured Learning."  The images of mv-lstm’s architecture and equations have been attached. This architecture uses the hidden output of all the views (e.g. In an emotion recognition system which uses lip movement, face image, and speech data; lip movement and face image is considered to update lstmcell parameters which have speech input) to update gate of a view. Main issue in implementing this is to get access to the back-propagation part through pytorch. Whether I can simply backprop using pytorch’s autograd or do I have to implement truncated backpropagation ? If I have to implement truncated backpropagation, then can I do it by using autograd function?<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/0/5022e8b456faa865d58110178b4768308e308b7a.png" data-download-href="https://discuss.pytorch.org/uploads/default/5022e8b456faa865d58110178b4768308e308b7a" title="back_mvlstm"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/5022e8b456faa865d58110178b4768308e308b7a_2_690x459.png" alt="back_mvlstm" data-base62-sha1="bqUZmTykP3T5QcdqtrQcu14vhzk" width="690" height="459" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/5022e8b456faa865d58110178b4768308e308b7a_2_690x459.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/5022e8b456faa865d58110178b4768308e308b7a_2_1035x688.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/0/5022e8b456faa865d58110178b4768308e308b7a.png 2x" data-dominant-color="F8F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">back_mvlstm</span><span class="informations">1286×857 55.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/9/f99d5203d73182d2dcb97d86888e1d1893ccca1b.png" data-download-href="https://discuss.pytorch.org/uploads/default/f99d5203d73182d2dcb97d86888e1d1893ccca1b" title="mv_arch"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/9/f99d5203d73182d2dcb97d86888e1d1893ccca1b.png" alt="mv_arch" data-base62-sha1="zCbWK7ArQXm77L0a0rDWK9uyO8P" width="690" height="433" data-dominant-color="EFEFEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">mv_arch</span><span class="informations">975×613 24.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Calculating attention weights in Chatbot Tutorial

username: Jinho_Park
cooked: <p>ref : <a href="https://pytorch.org/tutorials/beginner/chatbot_tutorial.html?highlight=chatbot" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/chatbot_tutorial.html?highlight=chatbot</a></p>
<pre><code class="lang-auto"># Forward through unidirectional GRU
rnn_output, hidden = self.gru(embedded, last_hidden)
# Calculate attention weights from the current GRU output
attn_weights = self.attn(rnn_output, encoder_outputs)
</code></pre>
<p>Based on Luong et al attention model, when calculate attention weights,<br>
don’t we have to put hidden(from the gru above) and enconder_outputs for self.attn? like below code</p>
<pre><code class="lang-auto"># Forward through unidirectional GRU
rnn_output, hidden = self.gru(embedded, last_hidden)
# Calculate attention weights from the current GRU output
attn_weights = self.attn(hidden, encoder_outputs)
</code></pre>
Question about memory usage of math operation

username: fanl
cooked: <p>Hi,<br>
In my case, I calculate loss as following code:</p>
<pre><code class="lang-auto">loss = 0
for A,B,C in sequence_of_tensors:
    loss += ((A - B)**2 * k1 * (A &gt; C).float() +
             (A - B)**2 * k2 * (A &lt; C).float() +
             (A - B)**2 * k3 * (A == C).float()).sum()

</code></pre>
<p>A and B are tensors. When I put all tensors on GPU, memory usage is about 5GB. But when the computation begins, I occur <code>CUDA out of memory</code> error. My GPU has 12GB memory, which I think is enough. The intermediate results of computation <strong>should not be</strong> larger than original tensors.<br>
I can’t understand why such computation is so memory expensive. I’ll appreciate any help.</p>

username: JuanFMontesinos
cooked: <p>I’d say it depends on the length of sequence_of_tensors.<br>
are all the tensors in sequence of tensors the same length? You can try calculate that batch-wise rather than list-wise.<br>
I would also store A-B as a variable, since you are performing that op 3 times I think it will create 3 graph nodes.<br>
So, in short, I would try to do something like:</p>
<pre><code class="lang-auto">A,B,C =stacked_sequence_of_tensors:
tmp = (A-B)**2
loss = (tmp * k1 * (A &gt; C).float() +
             tmp * k2 * (A &lt; C).float() +
             tmp * k3 * (A == C).float()).sum()
</code></pre>

username: fanl
cooked: <p>Thanks! Using <code>tmp</code>  alleviates the problem. I am surprised by the memory usage in calculation. Just the first execution of loop will lead to <code>out of memory</code>. It might be much more complicated than just storing intermediate results.</p>

username: JuanFMontesinos
cooked: <aside class="quote no-group" data-username="fanl" data-post="3" data-topic="61347">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/f/c5a1d2/48.png" class="avatar"> fanl:</div>
<blockquote>
<p>alleviates</p>
</blockquote>
</aside>
<p>I think it is also related to the for loop you are using. I would try to stack tensors inside if possible. That will save lot of memory.</p>
FC convolution operator

username: AlviseSembenico
cooked: <p>Hi, I had the idea to substitute the convolution operator with a FC operator. What I mean is not just to apply a FC layer to the whole image but to a subset of it as the convolution works but instead of just dot product of the weights and the value apply a proper shallow FC layer to catch better relations. How can I start tackling the problem of implementing it?<br>
Thank you a lot.</p>

username: InnovArul
cooked: <p>I sense that you are talking about an idea similar to “<a href="https://arxiv.org/pdf/1312.4400.pdf">Network in Network</a>” paper. One way of implementing it can be found here: <a href="https://github.com/yangqiongyongyu/Network-In-Network-Pytorch/blob/master/models/nin.py">https://github.com/yangqiongyongyu/Network-In-Network-Pytorch/blob/master/models/nin.py</a> . They realize the fully connected layer in terms of 1x1 convolution layer.</p>
Grad is very small

username: Mainul
cooked: <p>First I thought, there is no update after the loss calculation and update step but the follwing code gives me <code>false</code> which prove there is an update:</p>
<pre><code class="lang-auto">before = list(model.parameters())[0].clone()
loss.backward()
optimizer.step()
after = list(model.parameters())[0].clone()
logging.info(torch.equal(before.data, after.data))
</code></pre>
<p>Then I wanted to see the magnitude of chage by the following code:<br>
<code>list(model.parameters())[0].grad</code><br>
Which give me very small amaount of change:</p>
<pre><code class="lang-auto">tensor([[ 3.2294e-07,  7.3983e-06,  6.5637e-06,  ..., -1.3529e-06,
2019-11-16T17:21:03.747068480Z          -4.9979e-06,  4.9799e-06],
2019-11-16T17:21:03.747074966Z         [ 5.5463e-08,  3.0771e-06, -9.5087e-07,  ...,  1.9265e-06,
2019-11-16T17:21:03.747080687Z          -4.5251e-06, -7.1564e-07].....]
</code></pre>
<p>This is part of the output which shows how small they are.</p>
<p>Here is my model :</p>
<pre><code class="lang-auto">self.lstm_hidden_dim = lstm_hidden_dim
self.lstm = nn.LSTM(word_embedding_dim,lstm_hidden_dim, num_layers=1,batch_first=False,bidirectional=True)
self.firstHidden = nn.Linear(lstm_hidden_dim*2, 300)
self.relu=nn.ReLU()
self.secondlinear=nn.Linear(300, score_space_size)
self.softmax= nn.LogSoftmax(dim=1)
</code></pre>
<p>Loss function and optimizer:</p>
<pre><code class="lang-auto">loss_function = nn.NLLLoss()
optimizer = optim.SGD(model.parameters(), lr=0.1)
</code></pre>
<p>Any idea, why the grad is very small?</p>

username: vipin14119
cooked: <p>Try using an activation function on output of lstm layer, softsign , or tanh or relu ?</p>