akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
.detach() vs .cpu()?

username: antran96
cooked: <p>Which one is a better practice to detach a tensor without expanding CPU memory too much?<code> x.detach() or x.cpu() ?</code><br>
I found out that if I perform <code>x.cpu()</code>, my memory expand quickly and soon reach memory limit error in slurm.</p>

username: albanD
cooked: <p>Hi,</p>
<p>The two have very different (and non-overlapping) effect:</p>
<ul>
<li>
<code>x.cpu()</code> will do nothing at all if your Tensor is already on the cpu and otherwise create a new Tensor on the cpu with the same content as <code>x</code>. Note that his op is differentiable and gradient will flow back towards <code>x</code>!</li>
<li>
<code>y = x.detach()</code> breaks the graph between <code>x</code> and <code>y</code>. But <code>y</code> will actually be a view into <code>x</code> and share memory with it.</li>
</ul>
Help on indexing a tuple in libtorch please

username: Chen0729
cooked: <p>Hi, All,</p>
<p>I would like to convert the following from pytorch to libtorch. The pytorch code is<br>
acc = torch.mean((output_label.max(1)[1] == y).float())</p>
<p>I convert it as:<br>
float acc = torch::mean((output_label.max(1)[1] == y));</p>
<p>The building error is:<br>
error: no match for ‘operator[]’ (operand types are ‘std::tuple&lt;at::Tensor, at::Tensor&gt;’ and ‘int’)</p>
<p>Any suggestions to correct it please ?</p>
<p>Much appreciated</p>

username: ptrblck
cooked: <p>Try to use <code>std::get&lt;1&gt;(output_label.max(1))</code>.</p>

username: Chen0729
cooked: <p>Thanks a lot.<br>
auto acc = torch::mean(get&lt;1&gt;(output_label.max(1))== y); works</p>
<p>But<br>
float acc = torch::mean(get&lt;1&gt;(output_label.max(1))== y); doesnot work</p>
<p>How shall I convert acc from tensor to float in this case please ?</p>
<p>Thanks.</p>

username: ptrblck
cooked: <p>I guess <code>.item&lt;float&gt;()</code> should work.</p>

username: Chen0729
cooked: <p>Thanks, yep it works.</p>

username: Chen0729
cooked: <p>Another inquiry is that in the return,</p>
<p>I would like to return<br>
Loas.data.cpu() and acc</p>
<p>So I write</p>
<p>return{loss.dats.cpu(), acc}</p>
<p>It says “ no matching constructor for initialization of std::vector<a>torch::Tensor</a></p>
<p>Any suggestions to solve that please .</p>
<p>Much appreciated</p>

username: ptrblck
cooked: <p>Don’t use the <code>.data</code> attribute, as it might yield unwanted side effects, and try to push the tensor to the device with <code>tensor.to(at::kCPU);</code>.</p>

username: Chen0729
cooked: <p>Thanks,</p>
<p>Do you mean to use return{loss.to(at::kCPU), acc1},</p>
<p>I think the main difficulty is that the first return argument is tensor while the second return argument is float.</p>
<p>So just wonder if there is a way to return two arguements with different types or I need to convert them to be in the same type.</p>
<p>Thanks.</p>

username: Chen0729
cooked: <p>I find this is no error<br>
return{loss, acc};</p>
<p>I am tring to include data into the  return.<br>
Thanks,</p>

username: alanzhai219
cooked: <p>Actually, it is not C++ syntax. It is recommended to return a struct instead.</p>
Optimize transfer speed from CPU to GPU for simple computations

username: malteekj
cooked: <p>I have parameterized a filtering problem, so that I essentially only have to make a product and sum between two arrays. I want to do this on the GPU, using pytorch to accelerate this.</p>
<p>Right now my code is:</p>
<pre><code class="lang-auto">kernels_cuda = torch.cuda.FloatTensor(kernels) #  (32,32,159,21,21,21) float32 array
sub_matrices_cuda = torch.cuda.FloatTensor(sub_matrices) # (32,32,159,21,21,21) float32 array
out = (sub_matrices_cuda*kernels_cuda).sum(dim=[3,4,5]).cpu().numpy()
torch.cuda.synchronize()
</code></pre>
<p>This takes about 3.0s right now, and almost all the time is spent on sending the data to the GPU. Is there anyway I can speed up my code?</p>
<p>My hardware is: Titan RTX, Intel Xeon W2133 and 2666 MHz DDR4 RAM. I have the feeling, that I should be able to exceed this on this hardware?</p>
Number of workers with iter / next method

username: Rakshit_Kothari
cooked: <p>Hi,</p>
<p>According to this <a href="https://stackoverflow.com/questions/57729279/how-to-ensure-that-a-batch-contains-samples-from-all-workers-with-pytorchs-data" rel="noopener nofollow ugc">thread</a>, each worker loads a single batch. The more number of workers essentially means it preloads “future” data in memory.</p>
<p>If that’s the case, then is there any benefit of using the <a href="https://github.com/pytorch/pytorch/issues/1917#issuecomment-433698337" rel="noopener nofollow ugc">iter/next</a> approach to read batches from a loader?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/5/754581aee3fb8cb37330995636f76accf2c68807.png" data-download-href="https://discuss.pytorch.org/uploads/default/754581aee3fb8cb37330995636f76accf2c68807" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/5/754581aee3fb8cb37330995636f76accf2c68807.png" alt="image" data-base62-sha1="gJqMqU1jISv42xxLAbNr3qisbeT" width="690" height="150" data-dominant-color="503046"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">806×176 47.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>If you are using the <code>iter/next</code> approach, the <code>DataLoader</code> iterator might already start preloading the data once you call the <code>next</code> op on it, so you could potentially overlap the loading of the first batches with other operations in your main script.</p>
<p>Such an approach can be seen <a href="https://discuss.pytorch.org/t/enumerate-dataloader-slow/87778/4">here</a>.</p>

username: Rakshit_Kothari
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>, I can confirm that in the <code>iter/next</code>  approach, the dataloader behaves as expect and preloads future batches when number of workers &gt; 0. My observation is based on a vanilla dataloader from torch.</p>
Logarithm approximation CUDA intrinsics

username: Where
cooked: <p>Hello, community!</p>
<p>I’m writing pytorch code that uses a lot of logarithms, and it’s slower than ideal. When writing raw CUDA code, I’ve been able to speed things up using CUDA intrinsics, which gives approximate logarithms at the cost of a small amount of accuracy:</p>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://docs.nvidia.com/cuda/cuda-math-api/group__CUDA__MATH__INTRINSIC__SINGLE.html" target="_blank" rel="noopener nofollow ugc">docs.nvidia.com</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://docs.nvidia.com/cuda/cuda-math-api/group__CUDA__MATH__INTRINSIC__SINGLE.html" target="_blank" rel="noopener nofollow ugc">CUDA Math API :: CUDA Toolkit Documentation</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Is there any way to use these approximations within pytorch? I couldn’t see any options for torch.log, but maybe the functionality is elsewhere.</p>

username: ptrblck
cooked: <p>You could create a custom CUDA extension as described <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html">here</a> and use it in your PyTorch model.</p>
Callback in pytorch

username: mrz_naghdi
cooked: <p>Hello everyone. How can I use callback in pytorch for saving end of epoch in google colab?</p>

username: ptrblck
cooked: <p>Higher-level APIs such as Ignite or Lightning provide these type of hooks, so you should check them out. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Can anyone explain GRU in following code

username: ankurji
cooked: <p>class RNNnet(nn.Module):<br>
def <strong>init</strong>(self,input_size,hidden_size,output_size,n_layers=1):<br>
super(self).<strong>init</strong>()<br>
self.input_size=input_size<br>
self.hidden_size=hidden_size<br>
self.output_size=output_size<br>
self.n_layers=n_layers<br>
self.encoder=nn.Embedding(input_size,hidden_size)<br>
self.gru=nn.GRU(input_size,hidden_size,n_layers)<br>
self.decoder=nn.Linear(hidden_size,output_size)</p>
<pre><code>    def forward(self,input,hidden):
        input=self.encoder(input.reshape(1,-1))
        output,hidden=self.gru(input.view(1,1,-1),hidden)
        output=self.decoder(output.view(1,-1))
        return output,hidden
    
    def init_hidden(self):
        return torch.zeros(self.n_layers,1,self.hidden_size)</code></pre>

username: ptrblck
cooked: <p>What is your question about this module? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Note, you can find general information in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.GRU.html#torch.nn.GRU">docs</a>.</p>
Conv1D on word embeddings

username: aseemarora
cooked: <p>Let’s I have a mini-batch of 4 sentences. The number of words in the sentences are 9, 4, 10, and 7, respectively. I have to apply conv1D filter on the sentences. I am using embedding dimension of 300 for each word. I am using padding-index of 0 to form the mini-batch as sentences are of different lenghts. My input will shaped as [4, 10, 300]. To apply conv1D filter on it, it is reshaped to [4, 300, 10]. This way, the conv1D filter will not ignore the embeddings corresponding to the padding index.</p>
<p>How do I make sure that the effect of embeddings corresponding to the padding index is not taken into account? Like in GRU, we could explicitly pass the padding_idx to ignore it’s effect.</p>

username: Abhilash_Srivastava
cooked: <p>Do you mean how do you ignore the backprop through the padding sequence?<br>
Actually, you don’t need to do anything about that. If your neural net is trained well enough (for sufficient train data and enough epochs), it’ll learn to just ignore those extra paddings.<br>
A good way to test is to increase the sequence length to 100. You should still expect similar as you’d get with a sequence length of 10.</p>
CUDA freezes! Use RTX 3080 to train a python model

username: xiangyu_tang
cooked: <p>I have an RTX 3080 and I want to use it to train a deep learning model. Both CUDA and Pytorch I installed are version 10.2.  However, when I train this deep learning model, it freezes when loaded into CUDA.  Is there any solution?</p>

username: ptrblck
cooked: <p>If you’ve installed the CUDA10.2 binaries, the first CUDA operation would call into the JIT and compile the kernels for your compute architecture.<br>
You could use the nightly binaries, which are build with CUDA11 and support <code>sm_80</code> via:</p>
<pre><code class="lang-auto">conda install pytorch torchvision cudatoolkit=11.0 -c pytorch-nightly
</code></pre>

username: xiangyu_tang
cooked: <p>Thank you very much. I solved the problem this way</p>
Pytorch Model not initializing

username: Hacking_Pirate
cooked: <p>I created a model like resnet but for tabular data here is model architecture</p>
<pre><code class="lang-auto">class Model(nn.Module):      # &lt;-- Update
    def __init__(self, num_features, num_targets, hidden_size):
        super(Model, self).__init__()
        head1 = nn.Sequential(
        nn.BatchNorm1d(num_features),
        nn.Dropout(0.2),
        nn.utils.weight_norm(nn.Linear(num_features, hidden_size)),
        nn.ReLU()
        )
        
        head2 = nn.Sequential(
        nn.BatchNorm1d(hidden_size),
        nn.Dropout(0.25),
        nn.utils.weight_norm(nn.Linear(hidden_size, hidden_size)),
        nn.ReLU()
        )
        
        head3 = nn.Sequential(
        nn.BatchNorm1d(hidden_size),
        nn.Dropout(0.3),
        nn.utils.weight_norm(nn.Linear(hidden_size, num_targets))
        )
        
    def forward(self, x):
        head1_out = head1(x)
        concat_1 = torch.cat((x, head1_out), dim=1)
        head2_out = head2(concat)
        concat_2 = torch.cat((x, head2_out), dim=1)
        out = head3(concat_2)
        
        return out
</code></pre>
<p>When I initialize it it doesn’t give me any error but doesn’t show any output either.<br>
Any help or suggestions would be appreciated.<br>
Thanks.</p>

username: ptrblck
cooked: <p>Could you explain what kind of output you are expecting and how you are initializing the model?<br>
If you are using:</p>
<pre><code class="lang-python">model = Model(num_features, num_targets, hidden_size)
</code></pre>
<p>then <code>model</code> should be the object of your custom <code>Model</code> class.</p>

username: Hacking_Pirate
cooked: <p>Yes, this is how I am initializing it. And I am working with a multilabel classification problem</p>

username: Hacking_Pirate
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Any solution?</p>

username: ptrblck
cooked: <p>I don’t know exactly what the issue is.<br>
Could you explain, if you are seeing any errors or what you would expect, which is not working at the moment?</p>

username: Hacking_Pirate
cooked: <p>I figured it out. I had to write <code>self.head1</code>.</p>

username: ptrblck
cooked: <p>Oh sorry, I completely missed this issue and couldn’t figure out what the problem is. <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"><br>
I’m glad you’ve figured it out.</p>
Feature corelation

username: smani
cooked: <p>I have a feature map of size NxCxWxH, where N is the batch size, C is the number of channels, and W and H are width and the height respectively.</p>
<p>Lets consider features (a_i) from this feature map. Each feature has a dimension of Cx1x1. I need to find out the following for each of the feature: \sum_j a_i*a_j.</p>
<p>So the output I am expecting is Nx1xWxH, where each element represents the sum of the dot product between the feature at that location and all the other features. How to do it?</p>

username: alexgo
cooked: <p>Create a map sized CxWxH of the feature a_i repeated WxH times, then calculate a dot product along the channel dimension between that and your original feature map of size CxWxH (let’s call it A):</p>
<pre><code class="lang-auto">A_rep = a_i.repeat((1,W,H))
result = (A_rep*A).sum(axis=0)
</code></pre>
<p>I ignored N for simplicity but the concept is similar with N&gt;1</p>
Converting Tensorflow model (weights) to pytorch

username: YASJAY
cooked: <p>Hi,</p>
<p>Looking for ways to convert a custom tensorflow trained model to pytorch equivalent that can be fine-tuned further in pytorch.</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>I don’t know if there are tools to convert the TF model automatically to PyTorch and think you would have to rewrite it manually in PyTorch.<br>
A definition of a custom model can be found in <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#define-a-convolutional-neural-network">this tutorial</a> and might be a good starter.</p>
<p>Once the model architecture is created in PyTorch, you could convert the pretrained weights from TF to PyTorch. <a class="mention" href="/u/tom">@tom</a> and I did the same for the <code>StyleGAN</code> model in <a href="https://github.com/lernapparat/lernapparat/blob/master/style_gan/pytorch_style_gan.ipynb">this notebook</a> so you could take a look at the implementation.</p>

username: YASJAY
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, thanks that helped. I was able extract the sequence of layer information and corresponding weights using tf.all_variables().</p>
<p>The following are the components I am trying to transfer weights from tensorflow to pytorch now,</p>
<p>TF layers and weights:</p>
<ol>
<li>Batchnorm layer: BatchNorm/beta:0, BatchNorm/moving_mean:0 and moving_variance:0</li>
<li>Conv layer: Conv/weights:0 and Conv/biases:0</li>
<li>Conv2d_transpose layer: Conv2d_transpose_1/weights:0 and Conv2d_transpose_1/biases:0</li>
</ol>
<p>Can I just create the same sequence of layers in pytorch and assign these? Will there be computational flow differences between pytorch and tensorflow?<br>
Wondering if you have idea on conversion for those particular layers.</p>

username: ptrblck
cooked: <p>The notebook also shows how to transfer these parameter to PyTorch.<br>
In particular, check out <code>weight_translate</code>, which permutes the conv filters etc. to match the expected shape for PyTorch layers.</p>
Torch.cat for negative dimension

username: SAI_VARSHITTHA
cooked: <p>In the following,</p>
<pre><code> x_6 = torch.cat((x_1, x_2_1, x_3_1, x_5_1), dim=-3)
 Sizes of tensors x_1, x_2_1, x_3_1, x_5_1 are
 torch.Size([1, 256, 7, 7])
 torch.Size([1, 256, 7, 7]) 
 torch.Size([1, 256, 7, 7])
 torch.Size([1, 256, 7, 7]) respectively.
        
 The size of x_6 turns out to be torch.Size([1, 1024, 7, 7])
</code></pre>
<p>I couldn’t understand this concatenation along a negative dimension(-3 in this case).<br>
What exactly is happening here?<br>
How does the same go if dim = 3?<br>
Is there any constraint on dim for a given set of tensors?</p>

username: ptrblck
cooked: <p>Negative dimensions start from the end, so <code>-1</code> would be the last dimension, <code>-2</code> the one before etc.:</p>
<pre><code class="lang-python">        [1, 256, 7, 7]
pos dim: 0,   1,  2, 3
neg dim: -4, -3, -2, -1
</code></pre>
<p>If you are passing dimensions outside of this range, you should get an error:</p>
<pre><code class="lang-python">IndexError: Dimension out of range (expected to be in range of [-4, 3], but got -5)
</code></pre>
<p><code>dim=3</code> would correspond to <code>dim=-1</code>.</p>
Random core dumps and segmentation fault

username: jwillette
cooked: <p>On a single machine I started getting random segmentation faults which causes a core dump. I have seen some segmentation faults in other scenarios which were caused by invalid array indexing in the C++ backbone. I don’t think my problem is caused by this though because it is happening randomly in stably training models. I have seen it happen the same way while training many different models too so I am sure it is not something to do with a specific implementation. Here is an example of the output I am seeing…</p>
<pre><code class="lang-auto">d=1.01e+7, loss=2.15e+10]
 17%|██████████████████▌                                                                                            
| 8384/50000 [03:29&lt;17:17, 40.11it/s, bpd=9.33, loss=1.99e+4]Segmentation fault (core dumped)
</code></pre>
<p>that is a tqdm progress bar just followed with the core dump and no error message. I am not sure how to diagnose what is happening here. Could this be a hardware issue?</p>
<p>Versions:</p>
<p>Pytorch: 1.6.0<br>
OS:  Ubuntu 20.04<br>
GPU: GeForce GTX 1080TI 11GB</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="jwillette" data-post="1" data-topic="99863">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jwillette/48/17366_2.png" class="avatar"> jwillette:</div>
<blockquote>
<p>Could this be a hardware issue?</p>
</blockquote>
</aside>
<p>Might be the case, but I would start by trying to isolate a particular software setup which might be causing this issue.<br>
E.g. could you run the same workload in an Ubuntu18.04 docker container and check, if the behavior stays the same?<br>
If so, could you run it with:</p>
<pre><code class="lang-auto">gdb --args python script.py args
run
...
bt
</code></pre>
<p>and post the backtrace here?</p>
Inference demo for Detectron 2 not using any resources

username: Marceline
cooked: <p>I’m using Detectron2 with pretrained COCO-PanopticonSegmentation model on a GTX960. But as you can see, it’s barely using any resources. (That spike you see is me opening the screenshot utility)</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/2/e24b00b60f64fc3ea7809464dfca11083dfe4293.png" data-download-href="https://discuss.pytorch.org/uploads/default/e24b00b60f64fc3ea7809464dfca11083dfe4293" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e24b00b60f64fc3ea7809464dfca11083dfe4293_2_344x250.png" alt="image" data-base62-sha1="whSBgmOVFzwUx3gJWDAKjlC84tZ" width="344" height="250" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e24b00b60f64fc3ea7809464dfca11083dfe4293_2_344x250.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e24b00b60f64fc3ea7809464dfca11083dfe4293_2_516x375.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e24b00b60f64fc3ea7809464dfca11083dfe4293_2_688x500.png 2x" data-dominant-color="969898"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">810×587 24.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thus, the inference is taking ages, even on Colab.</p>
<p>What could be going wrong?<br>
What should I read to fix it?</p>

username: ptrblck
cooked: <p>The Windows Task Manager doesn’t show the compute workload by default, if I’m not mistaken, so you should either select another view from the drop down menu or use <code>nvidia-smi</code> to check the GPU utilization.</p>
How can I convert this tensorflow model to pytorch?

username: Hacking_Pirate
cooked: <pre><code class="lang-auto">def build_model(n_features, n_features_2, n_labels, label_smoothing = 0.0005):    
    input_1 = layers.Input(shape = (n_features,), name = 'Input1')
    input_2 = layers.Input(shape = (n_features_2,), name = 'Input2')

    head_1 = Sequential([
        layers.BatchNormalization(),
        layers.Dropout(0.2),
        layers.Dense(512, activation="elu"), 
        layers.BatchNormalization(),
        layers.Dense(256, activation = "elu")
        ],name='Head1') 

    input_3 = head_1(input_1)
    input_3_concat = layers.Concatenate()([input_2, input_3])

    head_2 = Sequential([
        layers.BatchNormalization(),
        layers.Dropout(0.3),
        layers.Dense(512, "relu"),
        layers.BatchNormalization(),
        layers.Dense(512, "elu"),
        layers.BatchNormalization(),
        layers.Dense(256, "relu"),
        layers.BatchNormalization(),
        layers.Dense(256, "elu")
        ],name='Head2')

    input_4 = head_2(input_3_concat)
    input_4_avg = layers.Average()([input_3, input_4]) 

    head_3 = Sequential([
        layers.BatchNormalization(),
        layers.Dense(256, kernel_initializer='lecun_normal', activation='selu'),
        layers.BatchNormalization(),
        layers.Dense(n_labels, kernel_initializer='lecun_normal', activation='selu'),
        layers.BatchNormalization(),
        layers.Dense(n_labels, activation="sigmoid")
        ],name='Head3')

    output = head_3(input_4_avg)
</code></pre>

username: ptrblck
cooked: <p>For a general introduction to writing custom PyTorch models, have a look at <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#define-a-convolutional-neural-network">this tutorial</a>.</p>
<p>To convert the TF model to PyTorch you should initialize all modules in the <code>__init__</code> method of your custom model and use these modules in the <code>forward</code> method.</p>
<p>The layers are almost equivalently named, i.e. <code>layers.BatchNormalization</code> (assuming it’s working on temporal data) would correspond to <code>nn.BatchNorm1d</code>, while e.g. <code>layers.Dense</code> corresponds to <code>nn.Linear</code>.</p>
<p>The “functional” layers, such as <code>layers.Concatenate</code> can be applied in the <code>forward</code> pass of your model using:</p>
<pre><code class="lang-python">input_3_concat = torch.cat((input_2, input_3))
</code></pre>
Creating custom loss function in CNN for regression

username: hbolandi
cooked: <p>I am trying to make my CNN regression network work, but output is far from ground truth. The input is a binary image (600x600) which the background is black, and foreground is white. The ground truth associated with each input is an image with color range from 0 t 255 which is normalized between 0 and 1.</p>
<p>Below, you can see one sample of Input (x), ground truth(y) and predicted output (y_hat), respectively. The network can almost detect edges and background but in foreground all the predicted values are almost same (see predicted one).</p>
<p>I believe that since all foregrounds in input are 1, network cannot update weights correctly and it may need custom loss function to penalize loss function.</p>
<p>I am wondering whether my idea is right or not, if yes what kind of custom function I need?</p>
<p>I tried to penalize the foreground by custom loss function below, but it didn’t improve the result.</p>
<blockquote>
<p>mse = nn.MSELoss(reduction=‘mean’)</p>
<p>def criterion(y, y_hat, x, loss, weight=0.1):<br>
y_hat_modified = torch.where(x[:,0:1]==1, weight*y_hat,y_hat)        # x[:,0:1] is input<br>
return loss(y,y_hat_modified)</p>
</blockquote>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/f/3fa204b3989fca0dece23fc95661625762a3e078.png" data-download-href="https://discuss.pytorch.org/uploads/default/3fa204b3989fca0dece23fc95661625762a3e078" title="610"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3fa204b3989fca0dece23fc95661625762a3e078_2_250x250.png" alt="610" data-base62-sha1="94VcBOpob3ofkooNGnUSEuMjxyg" width="250" height="250" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3fa204b3989fca0dece23fc95661625762a3e078_2_250x250.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3fa204b3989fca0dece23fc95661625762a3e078_2_375x375.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3fa204b3989fca0dece23fc95661625762a3e078_2_500x500.png 2x" data-dominant-color="8C8C8C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">610</span><span class="informations">600×600 1.75 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <img src="https://discuss.pytorch.org/uploads/default/original/3X/a/f/afe1c5a71678e4b22f291c52328f0c4dcf67b9b5.png" alt="real-n1000" data-base62-sha1="p5VgQIZ6GSA8zZRSx856AIwqrLT" width="213" height="177"> <img src="https://discuss.pytorch.org/uploads/default/original/3X/1/8/183e506d4f2f0ece66072e7cc561a0d699835c7f.png" alt="3layers-0res-epoch15-filter12-n1000" data-base62-sha1="3ssXWiHDRVAl5PRAf36187MtE8T" width="228" height="192"></p>
Cuda:out of Memory error

username: Anand_Khanna
cooked: <p>Hi,<br>
I am trying to train a model using k80.<br>
I get this error<br>
<code>RuntimeError: CUDA out of memory. Tried to allocate 1.38 GiB (GPU 0; 11.17 GiB total capacity; 10.05 GiB already allocated; 514.31 MiB free; 10.28 GiB reserved in total by PyTorch)</code><br>
I tried training the model with a much smaller batch size, but the training becomes extremely slow (2hrs per epoch)<br>
Is there a way to fix this issue?<br>
Following is my code:</p>
<pre><code class="lang-auto">def train_epoch(model, training_data, optimizer, pred_loss_func, opt):
    """ Epoch operation in training phase. """
    model.train()
    total_event_ll = 0  # cumulative event log-likelihood
    total_time_se = 0  # cumulative time prediction squared-error
    total_event_rate = 0  # cumulative number of correct prediction
    total_num_event = 0  # number of total events
    total_num_pred = 0  # number of predictions
    for batch in tqdm(training_data, mininterval=2,
                      desc='  - (Training)   ', leave=False):
        """ prepare data """
        event_time, time_gap, event_type = map(lambda x: x.to(opt.device), batch)
        """ forward """
        optimizer.zero_grad()
        enc_out, prediction = model(event_type, event_time) #BR: Output predictions
        """ backward """
        # negative log-likelihood
        event_ll, non_event_ll = Utils.log_likelihood(model, enc_out, event_time, event_type)
        event_loss = -torch.sum(event_ll - non_event_ll)
        # type prediction
        pred_loss, pred_num_event = Utils.type_loss(prediction[0], event_type, pred_loss_func)
        # time prediction
        se = Utils.time_loss(prediction[1], event_time)
        # SE is usually large, scale it to stabilize training
        scale_time_loss = 100
        loss = event_loss + pred_loss + se / scale_time_loss
        loss.backward()
        """ update parameters """
        optimizer.step()
        """ note keeping """
        total_event_ll += -event_loss.item()
        total_time_se += se.item()
        total_event_rate += pred_num_event.item()
        total_num_event += event_type.ne(Constants.PAD).sum().item()
        # we do not predict the first event
        total_num_pred += event_type.ne(Constants.PAD).sum().item() - event_time.shape[0]
        del event_time
        del time_gap
        del event_type
#         del se
#         del pred_loss
#         del pred_num_event
#         del event_loss
    rmse = np.sqrt(total_time_se / total_num_pred)
    return total_event_ll / total_num_event, total_event_rate / total_num_pred, rmse
</code></pre>
Using detach() while training a stateful LSTM

username: prash_857
cooked: <p>Hello, I am relatively new to Pytorch and I am trying to train a stateful LSTM Autoencoder. But while training I run into the following error</p>
<p>“RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time”</p>
<p>Even after setting retain_graph=True I run into another error</p>
<p>“RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation.”</p>
<p>A solution that I found on the forum is to use detach() at the end of every forward pass. However, I am unsure if detaching the hidden state from the computation graph would only retain the hidden state from the previous time step alone. If this is the case how do I backpropagate across batches?</p>
<pre><code class="lang-auto">class Encoder(nn.Module):
    def __init__(self, inputDimension, outputDimension):
        super(Encoder, self).__init__()
        self.inputDimension = inputDimension
        self.outputDimension = outputDimension
        #batch size (numLayers, batchSize, hidden_layer size)
        self.hidden1 = (torch.zeros(1,1,64), torch.zeros(1,1,64))
        self.hidden2 = (torch.zeros(1,1,self.outputDimension), torch.zeros(1,1, self.outputDimension))
        self.lstm1 = nn.LSTM(self.inputDimension, 64, num_layers=1)
        self.lstm2 = nn.LSTM(64, self.outputDimension, num_layers=1)

    def forward(self, inputs):
        x,self.hidden1= self.lstm1(inputs, self.hidden1)
        x,self.hidden2= self.lstm2(x, self.hidden2)
        return x

class Decoder(nn.Module):
    def __init__(self, inputDimension, outputDimension):
        super(Decoder, self).__init__()
        self.inputDimension = inputDimension
        self.outputDimension = outputDimension
        self.hidden3 = (torch.zeros(1, 1, 64), torch.zeros(1, 1, 64))
        self.lstm1 = nn.LSTM(self.inputDimension, 64, num_layers=1)
        self.lin1 = nn.Linear(64, self.outputDimension)

    def forward(self, inputs):
        #no need to reshape input as the output of Encoder is a 3D tensor
        x,self.hidden3= self.lstm1(inputs, self.hidden3)
        x = x.view(-1, torch.prod(torch.tensor(x.shape)[1:]))
        x = self.lin1(x)
        return x

class LSTM_Autoencoder(nn.Module):
    def __init__(self,inputDimension, outputDimension):
        super(LSTM_Autoencoder, self).__init__()
        self.inputDimension = inputDimension
        self.outputDimension = outputDimension
        self.encoder = Encoder(inputDimension, outputDimension)
        self.decoder = Decoder(outputDimension, inputDimension)

    def forward(self, inputs):
        inputs = inputs.view(len(inputs), 1, -1)
        x = self.encoder(inputs)
        x = self.decoder(x)

        return x
</code></pre>
<p>Any clarification provided is much appreciated.</p>
What is the behaviour if I don't use an output head in loss computation?

username: paganpasta
cooked: <p>Hi,</p>
<p>I have a network which outputs <code>2</code> values, <code>x</code> and <code>y</code>.  I intend to not use the output <code>y</code> for training. Will the backward pass add weights with garbage values due to <code>y</code> branch?</p>
<p>Below is just an illustration of the model at hand.<br>
<code>model = common_model -&gt; [head_x, head_y]</code></p>
<p>The optimisers for <code>x</code> and <code>y</code> head are initialized with <code>model.parameters()</code> field. And I only call <code>opt_x.zero_grad()</code> before each forward pass.</p>
<p>Any things to take care of for such situations? Like calling <code>zero_grad()</code> for <code>opt_y</code> as well?</p>

username: albanD
cooked: <p>Hi,</p>
<p>If <code>y</code> is not used, then no gradient will flow back from there. And only the contribution from x will count.</p>
Creating custom dataset C++

username: Charles_Muzhanye
cooked: <p>Im getting started with the C++ frontend, I wanted to ask how I could create a dataset from two tensors, one with labels and another with features</p>

username: tom
cooked: <p>The <code>torch::data::datasets::TensorDataset</code>, which is the equivalent of <code>torch.utils.data.TensorDataset</code> in Python, might work.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Chen0729
cooked: <p>Thanks, I have exactly same problem with how to use TensorDataset,</p>
<p>The python code is<br>
return TensorDataset(torch.FloatTensor(np.array(data), torch.LongTensor(np.array(labels)))</p>
<p>May I ask how to convert the above in C++ please ?</p>
<p>I write torch::data::datasets::TensorDataset T1[]{}；</p>
<p>How to send data and label to T1 in this case ?</p>
<p>Thanks.</p>
How to implement Pytorch equivalent of Keras' kernel weight regulariser

username: paganpasta
cooked: <p>Hi, I wanted to implement a pytorch equivalent of keras code mentioned below.</p>
<pre><code class="lang-auto">    self.regularizer = self.L2_offdiag(l2 = 1) #Initialised with arbitrary value

    Dense(classes, input_shape=[classes], activation="softmax", 
    kernel_initializer=keras.initializers.Identity(gain=1), 
    bias_initializer="zeros",kernel_regularizer=self.regularizer, 
    bias_regularizer=keras.regularizers.l2(l=1))

    class Regularizer(object):
        """
        Regularizer base class.
        """

        def __call__(self, x):
            return 0.0

        @classmethod
        def from_config(cls, config):
            return cls(**config)


    class L2_offdiag(Regularizer):
        """
        Regularizer for L2 regularization off diagonal.
        """

        def __init__(self, l2=0.0):
        
            """
            Params:
                l: (float) lambda, L2 regularization factor.
            """
            self.l2 = K.cast_to_floatx(l2)

        def __call__(self, x):
            """
            Off-diagonal regularization (complementary regularization)
            """

            reg = 0

            for i in range(0, x.shape[0]):
                reg += K.sum(self.l2 * K.square(x[0:i, i]))
                reg += K.sum(self.l2 * K.square(x[i+1:, i]))
                
            return reg


</code></pre>
<p>I understand the I can use <code>torch.nn.Linear(classes,classes)</code> to create the the layers matching the keras implementation. But how do I integrate regularisers and bias in the layer?</p>

username: ptrblck
cooked: <p>The <code>bias</code> parameter is automatically added to <code>nn.Linear</code>, if you don’t explicitly set <code>bias=False</code> in its creation.<br>
To add a regularization term for the <code>weight</code> parameter, you could manually add it to the loss:</p>
<pre><code class="lang-python">output = model(input)
loss = criterion(output, target)
loss = loss + torch.norm(model.layer.weight, p=2)
</code></pre>

username: paganpasta
cooked: <p>Thanks, is there a default L2 regularisation which I need to disable explicitly before doing the normalisation myself?</p>

username: ptrblck
cooked: <p>Optimizers accept the <code>weight_decay</code> argument, which adds the L2 penalty to the loss, so you might want to set it to <code>0</code>.</p>
TensorFlow's tf.scan or Theano's theano.scan equivalent (2020)

username: galerkin
cooked: <p><a href="https://discuss.pytorch.org/t/tf-map-fn-or-theano-scan-equivalent-in-pytorch/3383/2">This question has been asked in 2017</a>, but I’m wondering now (2020) if there is any way to do <a href="http://deeplearning.net/software/theano/library/scan.html" rel="noopener nofollow ugc">tf.scan</a> or <code>theano.scan</code> in PyTorch.</p>
<p>In PyTorch, there is <code>torch.cumsum</code>, which can be thought of as a special case of <code>scan</code>. Specifically, <code>cumsum</code> is tied to the addition operator, whereas in TensorFlow or Theano, <code>scan</code> can be used with any binary operator (passed in as a function), not just addition.</p>
<p>From the 2017 thread (linked above), <a class="mention" href="/u/jekbradbury">@jekbradbury</a> said:</p>
<blockquote>
<p>For equivalents of <code>theano.scan</code> , use Python <code>for</code> and <code>while</code> loops.</p>
</blockquote>
<p>However, I would prefer not to use <code>for</code> and <code>while</code> loops, in part because I want my computation to be done in parallel on the GPU. Scans, a.k.a. prefix sums, are a common GPGPU operation, which is why CUDA libraries like Thrust contain a variety of prefix sum functions.</p>
<p>Has anyone (officially or unofficially) implemented a <code>scan</code> in PyTorch, or am I stuck with Python <code>for</code> and <code>while</code> loops?</p>

username: ebarsoum
cooked: <p>Did you try loop with TorchScript?</p>

username: blade
cooked: <p>Did you find an answer for your question?</p>

username: galerkin
cooked: <p>Nope, I never found an answer. I ended up using a combination of <code>torch.cumsum</code> and <code>torch.cumprod</code> to implement the operation that I needed, but it would still be convenient for PyTorch to have a scan primitive that accepts an arbitrary binary operator, like in TensorFlow/Theano or even Thrust.</p>
TypeError: 'DistributedDataParallel' object does not support indexing

username: yhz_yhz
cooked: <p>I used LSTMCell for decoders .And my decoder module looks like this :decoders = nn.ModuleList([Decoder(args, gpu) for i in range(args.max_len)])</p>
<p>I  changeded it for parallel using<br>
decoders = nn.parallel.DistributedDataParallel(decoders,<br>
device_ids=[gpu])</p>
<p>And when I wrote this<br>
output = decoders [i] (input)</p>
<p>the error raised<br>
TypeError: ‘DistributedDataParallel’ object does not support indexing</p>
<p>How can I fix this?</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/yhz_yhz">@yhz_yhz</a>, if you would like to access the original module that you passed to <code>DistributedDataParallel</code> ctor, you can use <code>decoders.module</code>. See the code below.</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/172ed51a17e70aabfbdc096491fde79755de9d08/torch/nn/parallel/distributed.py#L384" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/172ed51a17e70aabfbdc096491fde79755de9d08/torch/nn/parallel/distributed.py#L384" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/172ed51a17e70aabfbdc096491fde79755de9d08/torch/nn/parallel/distributed.py#L384</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="374" style="counter-reset: li-counter 373 ;">
<li>        output_device = device_ids[0]</li>
<li>
</li><li>    self.output_device = _get_device_index(output_device, True)</li>
<li>
</li><li>if process_group is None:</li>
<li>    self.process_group = _get_default_group()</li>
<li>else:</li>
<li>    self.process_group = process_group</li>
<li>
</li><li>self.dim = dim</li>
<li class="selected">self.module = module</li>
<li>self.device = list(self.module.parameters())[0].device</li>
<li>self.broadcast_buffers = broadcast_buffers</li>
<li>self.find_unused_parameters = find_unused_parameters</li>
<li>self.require_backward_grad_sync = True</li>
<li>self.require_forward_param_sync = True</li>
<li>self.ddp_join_enabled = False</li>
<li>self.gradient_as_bucket_view = gradient_as_bucket_view</li>
<li>if hasattr(module, '_ddp_params_and_buffers_to_ignore'):</li>
<li>    self.parameters_to_ignore = module._ddp_params_and_buffers_to_ignore</li>
<li>else:</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
RPC - TensorPipe send/recieve

username: ItamarWilf
cooked: <p>Can we have communication primitives such as <code>send</code> and <code>recv</code> when using the TensorPipe backend? I want to use it for inference between processes, having TensorPipe figure out the best way of communications. Is there any other way to implement it using the current RPC - TensorPipe api?</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/itamarwilf">@ItamarWilf</a>,</p>
<blockquote>
<p>Can we have communication primitives such as  <code>send</code>  and  <code>recv</code>  when using the TensorPipe backend?</p>
</blockquote>
<p>Yep, it should be possible to use <code>ProcessGroup</code> <code>send/recv</code> in conjunction with RPC. But we don’t yet have a TensorPipe backend for ProcessGroups. So the send/recv API needs to choose from gloo, nccl, and mpi backends.</p>
<p>One caveat is that, ProcessGroup send/recv requires the orders to match on the sender and receiver. However, RPC by-itself does not guarantee orders, as it will grab threads from a pool to process requests concurrently. So, if you are using ProcessGroup send/recv within RPC functions, you might need to enforce the order in user functions.</p>
<p>cc <a class="mention" href="/u/lcw">@lcw</a></p>
<blockquote>
<p>I want to use it for inference between processes, having TensorPipe figure out the best way of communications. Is there any other way to implement it using the current RPC - TensorPipe api?</p>
</blockquote>
<p>Yep, this should be possible. You can, e.g., using the RPC to implement a queue, and then call enqueue on the sender and dequeue on the receiver. Then enqueue/dequeue will be similar to send/receive.</p>
<p>Please let us know if this would work for you.</p>

username: lcw
cooked: <p>Echoing <a class="mention" href="/u/mrshenli">@mrshenli</a>, if all you need is send and recv, you may be better off using the RPC primitives, which were designed for this type of point-to-point communication, rather than using the ProcessGroup interface. There are currently no plans to expose TensorPipe as a ProcessGroup backend.</p>
Compressing a Pruned `nn.Module`

username: yngtodd
cooked: <p>Hey everyone,</p>
<p>I understand that the pruning methods in Pytorch create masks of the network’s weights, and when we use <code>prune.remove</code>, we remove that reparametrization, leaving the same number of network weights, where the pruned weights are set to zero.</p>
<p>I was wondering how I might then be able to reparametrize the network to remove those zeroed weights, resulting in a smaller model for devices with limited memory. Can anyone provide some thoughts on what that would require?</p>
<p>Thanks!</p>
How does requires_grad compare to set_grad_enabled?

username: AAYUSH_LAKHANI
cooked: <p>In this transfer learning tutorial <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html</a>, first, requires_grad is set to False then in the training loop, while training we’re calling set_grad_enabled, I am getting confused, new to PyTorch, pls explain what’s going on here?</p>

username: albanD
cooked: <p>Hi,</p>
<p>The <code>requires_grad</code> is a Tensor property that tells us if it requires gradients or not.<br>
THe <code>set_grad_enabled</code> is a global flag that allows you to disable the autograd locally.</p>
Create a dictionnary of layers in pytorch

username: VictorBarolle
cooked: <p>Hi everyone,</p>
<p>I’m working on an electromagnetism project using neural network and I’m trying to create a nn.Module class for a neural network for which many layers are contained in a dictionnary. Like this:</p>
<p>class ComplexNet(nn.Module):</p>
<pre><code>def __init__(self):
    
    super(ComplexNet, self).__init__()
    
    self.R = dict() 

    for pp in ['00', '01', '10', '11']:
        for ii in ['x','y','z']:
            for jj in ['x','y','z']:  
                for kk in ['r', 'i']:
                    self.R[pp+'e'+jj+'e'+ii+'_'+kk] = HadamardProduct(Nky, Nkx, pp+'e'+jj+'e'+ii+'_'+kk, bias = False)
                    self.R[pp+'h'+jj+'e'+ii+'_'+kk] = HadamardProduct(Nky, Nkx, pp+'h'+jj+'e'+ii+'_'+kk, bias = False)
</code></pre>
<p>where HadamardProduct is a layer for dot product operation:</p>
<p>class HadamardProduct(nn.Module):</p>
<pre><code>def __init__(self, Nky, Nkx, string, bias=True):
    super().__init__()
    self.Nky =  Nky
    self.Nkx =  Nkx
    self.weight = nn.Parameter(torch.load(f'{path}/R{string}')).double()
    
def forward(self, input):
    return input *  self.weight
</code></pre>
<p>The problem arises when I want to insert model.parameters() in the optimizer. It doesn’t recognize the different layers. Do you have any clue how to fix this issue?<br>
Otherwise, I’ll need to create layers like this:</p>
<p>self.R00exexr  =  HadamardProduct(Nky, Nkx, ‘00exex_r’, bias = False)<br>
self.R00exeyr  =  HadamardProduct(Nky, Nkx, ‘00exey_r’, bias = False)<br>
self.R00exezr  =  HadamardProduct(Nky, Nkx, ‘00exez_r’, bias = False)<br>
…</p>
<p>But there are 4x3x6 different ones.</p>
<p>Any help would be greatly appreciated!</p>
<p>Victor</p>

username: VictorBarolle
cooked: <p>Ok, I think I found a trick using nn.ModuleDict</p>
<p>class ComplexNet(nn.Module):</p>
<pre><code>def __init__(self):
    
    super(ComplexNet, self).__init__()
    
    self.R = nn.ModuleDict()

    for pp in ['00', '01', '10', '11']:
        for ii in ['x','y','z']:
            for jj in ['x','y','z']:  
                for kk in ['r', 'i']:
                    self.R[pp+'e'+jj+'e'+ii+'_'+kk] = HadamardProduct(Nky, Nkx, pp+'e'+jj+'e'+ii+'_'+kk, bias = False)
                    self.R[pp+'h'+jj+'e'+ii+'_'+kk] = HadamardProduct(Nky, Nkx, pp+'h'+jj+'e'+ii+'_'+kk, bias = False)
</code></pre>
<p>Let me know if you have any other suggestion <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: albanD
cooked: <p>Hi,</p>
<p>ModuleDict is the right thing to use here indeed!</p>