akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Early stop - should i stop training

username: Gal_Co
cooked: <p>This is more of theoretical question.</p>
<p>Consider the following example:<br>
While training my model reaches the following stats:<br>
40% accuracy on validation set and 60% accuracy on training set.<br>
If i continue training the model, the accuracy on the validation slightly improves to 43% while the accuracy on the training set goes up to around 97%.</p>
<p>Should i stop the training on the first option? in other words, should i take the model that with the higher results although its seems like overfitted.</p>
<p>Thanks a lot.</p>

username: Usama_Hasan
cooked: <aside class="quote no-group" data-username="Gal_Co" data-post="1" data-topic="93787">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gal_co/48/16893_2.png" class="avatar"> Gal_Co:</div>
<blockquote>
<p>Should i stop the training on the first option? in other words, should i take the model that with the higher results although its seems like overfitted.</p>
</blockquote>
</aside>
<p>Hy <a class="mention" href="/u/gal_co">@Gal_Co</a>, I guess you should training over there. In my opinion the first and the second results both are meaning less, because it’s overfitting in both the iteration. That’s the whole concept of early stopping plus overfitted model are of no use. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>

username: pinazawa123
cooked: <aside class="quote no-group" data-username="Gal_Co" data-post="1" data-topic="93787">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gal_co/48/16893_2.png" class="avatar"> Gal_Co:</div>
<blockquote>
<p>the accuracy on the validation slightly improves to 43% while the accuracy on the training set goes up to around 97</p>
</blockquote>
</aside>
<p>IMHO, this awnser really depends on your data and your loss curves (training and validation) and not solely on accuracy. These are the ones that will tell you better whether you will have or not overfitted your data.</p>
<p>Only watching accuracy is not great, since validation loss sometimes do not actually reflex on a great accuracy. For exemple, in imbalanced problems, your model can have an excelent accuracy by only learning the most present class of the dataset and rejecting others. Your val loss would be high, but the accuracy would also be “good”.</p>
<p>At the end of the day, I would choose the model that has the lowest validation loss.</p>
How to make LSTM Multivariate?

username: Sam_badie
cooked: <p>Hi Guys,</p>
<p>I am new to PyTorch and I need help. I was trying to use the sample code here: <a href="https://github.com/spdin/time-series-prediction-lstm-pytorch/blob/master/Time_Series_Prediction_with_LSTM_Using_PyTorch.ipynb?fbclid=IwAR0pzLvhdhz0kl2_tvJEGAybzRoXuIFlzUOpCxSHUzP-slWh6RIgHhbBUV0" rel="nofollow noopener">https://github.com/spdin/time-series-prediction-lstm-pytorch/blob/master/Time_Series_Prediction_with_LSTM_Using_PyTorch.ipynb?fbclid=IwAR0pzLvhdhz0kl2_tvJEGAybzRoXuIFlzUOpCxSHUzP-slWh6RIgHhbBUV0</a></p>
<p>it works well, but when I tried to add more input features I got Error in the testing section for the following lines:</p>
<pre><code class="lang-auto">data_predict = sc.inverse_transform(data_predict)
dataY_plot = sc.inverse_transform(dataY_plot)
</code></pre>
<p>The error says “non-broadcastable output operand with shape (110,1) doesn’t match the broadcast shape”</p>
<p>so the model works well, but I can’t retrieve the original value of Y. So what else Should I modify in the code? I modified  “input_size” in training section and added so following line in Data loading section:</p>
<pre><code class="lang-auto">y = (y[:,0]).reshape(y[:,0].shape[0],1) 
</code></pre>
<p>and finally</p>
<pre><code class="lang-auto">training_set = df.iloc[:,1:3].values
</code></pre>
<p>Thanks in advance</p>

username: Usama_Hasan
cooked: <aside class="quote no-group" data-username="Sam_badie" data-post="1" data-topic="93727">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sam_badie/48/28075_2.png" class="avatar"> Sam_badie:</div>
<blockquote>
<p>“non-broadcastable output operand with shape (110,1) doesn’t match the broadcast shape”</p>
</blockquote>
</aside>
<p>Hy can you compare the output shape of your and the data_predict which your trying</p>

username: Sam_badie
cooked: <p>Thanks Usama for your reply,</p>
<p>the shape of data_predict and it is (110,1), so apparently it contains only the target values, while the orginal data before min_max_scaler is (110,4) target + 3 features.</p>

username: Usama_Hasan
cooked: <p>Hy <a class="mention" href="/u/sam_badie">@Sam_badie</a>, Your error is quite clear, min_max_scaler will only convert array that have the same shape say (no_samples,4), which was used to fit the min_max_scaler object.</p>
Can LSTM run multivariate time series?

username: morphism
cooked: <p>Hello, everyone.<br>
I want to run Deep Learning model for multivariate time series.</p>
<p>For example, below is the daily delivery amount of post office</p>
<p>delivery date, post office id, delivery amount, weekday, …</p>
<p>which is daily data, multivariate</p>
<p>I want to predict future delivery amount using data above.</p>
<p>Can I run this as deep learning model using LSTM??</p>
<p>That is, can LSTM run mutlivariate time series data for prediction??</p>
<p>If it is not possible, which deep learning model should I use??</p>
<p>Thank you in advance and have a nice day!</p>

username: Usama_Hasan
cooked: <aside class="quote no-group" data-username="morphism" data-post="1" data-topic="93779">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/morphism/48/28101_2.png" class="avatar"> morphism:</div>
<blockquote>
<p>Can I run this as deep learning model using LSTM??</p>
</blockquote>
</aside>
<p>Yes you can use LSTM for time series data prediction. You can find alot of resources for that purpose.</p>
<p>You can check this github repo for research papers and link for data resources.<br>
<a href="https://github.com/Alro10/deep-learning-time-series" rel="noopener nofollow ugc">https://github.com/Alro10/deep-learning-time-series</a></p>
<p>If you want to checkout for implementation you can also find that in below link.<br>
<a href="https://github.com/zhangxu0307/time_series_forecasting_pytorch" rel="noopener nofollow ugc">https://github.com/zhangxu0307/time_series_forecasting_pytorch</a><br>
<a href="https://github.com/KurochkinAlexey/ConvRNN" rel="noopener nofollow ugc">https://github.com/KurochkinAlexey/ConvRNN</a><br>
<a href="https://stackoverflow.com/questions/56858924/multivariate-input-lstm-in-pytorch" rel="noopener nofollow ugc">https://stackoverflow.com/questions/56858924/multivariate-input-lstm-in-pytorch</a></p>

username: morphism
cooked: <p>Dear Usama:</p>
<p>Thank you for your help and kindness.<br>
I believe your advise can solve my problems.<br>
I will read them in detail.</p>
<p>Have a nice day and see you, Usama.</p>
Interpolate does not preserve the memory structure

username: smujjiga
cooked: <p><code>F.interpolate</code> is not preserving the input memory structure when it is channel first. Is it expected ? Is there a way to preserve the channel first memory structure or explicitly converting it back after <code>interpolate</code> is the only way ?</p>

username: ptrblck
cooked: <p>Could you post a code snippet to demonstrate the behavior, please? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Data loader struct pack issue(overflow)?

username: rui_zhang_331
cooked: <p>I am using pytorch data loader and runs into some strange error as below, we generate training data in each hour. and I run this experiment to just read training data csv file and parse line by line, no real training stuff is going. we define the dataloader each item as a file in bunch of date/hour parsed folder. and run multi worker data loader</p>
<p>I checked it should not be <code>__get_item</code> index out of boundary, is it because some training file we loaded is too large? it is not always repeatable error, so I am not sure it may be related to some training data? we constantly generate training data every hour, so any hint?</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/miniconda/lib/python3.6/multiprocessing/queues.py", line 240, in _feed
    send_bytes(obj)
  File "/home/miniconda/lib/python3.6/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/home/miniconda/lib/python3.6/multiprocessing/connection.py", line 393, in _send_bytes
    header = struct.pack("!i", n)
struct.error: 'i' format requires -2147483648 &lt;= number &lt;= 2147483647
</code></pre>
A question about the details of backpropagation and grad_fn in pytorch

username: co_ni
cooked: <aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://blog.paperspace.com/favicon.png" class="site-icon" width="500" height="500">
      <a href="https://blog.paperspace.com/pytorch-101-understanding-graphs-and-automatic-differentiation/" target="_blank" rel="nofollow noopener" title="08:03PM - 29 May 2019">Paperspace Blog – 29 May 19</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/443;"><img src="https://blog.paperspace.com/content/images/2019/05/full_graph.png" class="thumbnail" width="690" height="443"></div>

<h3><a href="https://blog.paperspace.com/pytorch-101-understanding-graphs-and-automatic-differentiation/" target="_blank" rel="nofollow noopener">PyTorch Basics: Understanding Autograd and Computation Graphs</a></h3>

<p>In this article, we dive into how PyTorch's Autograd engine performs automatic differentiation.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
In this article, there is one explanation about backpropagation happens with a computation graph.
<pre><code class="lang-auto">def backward (incoming_gradients):
	self.Tensor.grad = incoming_gradients

	for inp in self.inputs:
		if inp.grad_fn is not None:
			new_incoming_gradients = //
			  incoming_gradient * local_grad(self.Tensor, inp)
			
			inp.grad_fn.backward(new_incoming_gradients)
		else:
			pass
</code></pre>
<p>However, here is the question.</p>
<pre><code class="lang-auto">import torch

x = torch.randn((3, 3), requires_grad=True)
w = torch.randn((3, 3), requires_grad=True)


t = w * x
loss = 10 - t.sum()
loss.backward()

print(f"{x.grad_fn=}, {x.requires_grad=}, {x.grad=}, {x.is_leaf=}")
# x.grad_fn=None, x.requires_grad=True, x.grad=tensor([[-0.1528, -0.5653,  0.0930],
#         [ 0.6547, -1.1224,  1.1258],
#         [-0.2397, -0.4652, -1.7134]]), x.is_leaf=True

</code></pre>
<p>When <code>inp = x</code>, then because <code>x.grad_fn</code> is None, so the code would straightly go to <code>else : pass</code>.<br>
Then <code>x.grad</code> should rationally be None anyway, since the code didn’t do the computation.<br>
But when we print it out, as we can see above, <code>x.grad</code> is not None.(see <code>x.grad=tensor([[-0.1528, -0...</code> )<br>
So does this mean the article I showed is wrong?</p>
<p>By the way, I’m curious about under which conditions a tensor’s <code>grad_fn</code> would be None?<br>
Would a tensor’s <code>grad_fn</code> set to <code>None</code> only if its ‘is_leaf’ is <code>True</code>?</p>
Saving and load model Intermediate Layer output the tensor object

username: vkthevallavan
cooked: <p>I am trying to save the intermediate output, the Tensor object output of a pre-trained model layer to a file, and loading it back using pickle dump/load. when I convert to a dump and loading it back, the grad_fn object in the tensor is getting removed and replaced by requires_grad.</p>
<p><strong>Original:</strong><br>
tensor([[[[ 7.3672e-01,  3.9696e-01,  8.4902e-01,  …, -5.8079e-01,<br>
4.0203e+00,  1.5609e+00],<br>
…,<br>
[         [-9.5835e-01,  5.4335e-01,  9.7559e-01,  …, -2.1748e+00,<br>
-3.4150e+00, -8.7405e+00]]]], grad_fn=)<br>
<strong>Converted:</strong><br>
tensor([[[[ 7.3672e-01,  3.9696e-01,  8.4902e-01,  …, -5.8079e-01,<br>
4.0203e+00,  1.5609e+00],<br>
…,<br>
[-9.5835e-01,  5.4335e-01,  9.7559e-01,  …, -2.1748e+00,<br>
-3.4150e+00, -8.7405e+00]]]], requires_grad=True)</p>
<p>Is there a way I can avoid this? or is there a way to add grad_fn to a tensor object manually. For Ex: I want to know if there a way to set MkldnnConvolutionBackward as grad_fn when i can reconstruct the tensor object matrix from data of the orginal tensor object</p>
<p>Thank You for any assistance</p>

username: ptrblck
cooked: <p>No, I don’t think you can manually recreate the computation graph.<br>
What’s your use case that you would like to serialize the tensor first and rebuild the graph later?</p>

username: vkthevallavan
cooked: <aside class="quote no-group quote-modified" data-username="ptrblck" data-post="2" data-topic="93431">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>RE</p>
</blockquote>
</aside>
<p>my use case is compute intermediate layer Output result and send to another server when another half of execution will be done something like client server environment. For example if an alexnet has 12 layers first machine will be having 6 layers running in it which will take the image input and we will get the intermediate output (tensor object) of it and then sent it through network and use it as input for next half of the model. Is there any way to do it? Even in my current scenario the numpy array is intact only the grad_fn is missing for future prediciton</p>

username: ptrblck
cooked: <p>This sounds like a distributed training setup. While the usual workflow would be to let each node compute a single forward and backward pass on its own, I think it should be possible to apply model sharding in this setup as well. Unfortunately, I don’t have an examplecode snippet ready for it and would recommend to look into the functional API of the <code>distributed</code> package.</p>

username: vkthevallavan
cooked: <p>Thank you for the information. I will try and see</p>
Is finfo() exposed in the libtorch?

username: Shisho_Sama
cooked: <p>Hello everyone.<br>
in Pytorch finfo is exposed but it seems either it doesnt exist in  libtorch or I cant find it!<br>
Is it also available in libtorch? I need to implement sth like <code>np.spacing(1)</code> and I dont know how I’m supposed to do it without finfo!<br>
Any help is greatly apprecaited</p>
Is Pytorch nn.Embedding supposed to give identical results with the same weights?

username: waynchi
cooked: <p>I’m having an issue where after loading from my checkpoint my model is not giving me the same results. I’ve tested the same exact code in training and that works as expected, but after loading from a checkpoint I get a completely different result.</p>
<p>I think I have narrowed it down to the nn.embedding layer; I’ve checked to ensure that the embedding layer gives the same weights, but somehow when I look for a specific vocab in the embedding layer, I get different results during training vs inference with the same code.</p>
<p>For example, here I am saving the weights<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/2/92ca47ec7f5002341228b47fcbe2f494787fc662.png" data-download-href="https://discuss.pytorch.org/uploads/default/92ca47ec7f5002341228b47fcbe2f494787fc662" title="saving_2"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/2/92ca47ec7f5002341228b47fcbe2f494787fc662_2_690x173.png" alt="saving_2" data-base62-sha1="kWz5VAFWmTdliS8kTqaNxfkGOUW" width="690" height="173" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/2/92ca47ec7f5002341228b47fcbe2f494787fc662_2_690x173.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/2/92ca47ec7f5002341228b47fcbe2f494787fc662_2_1035x259.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/2/92ca47ec7f5002341228b47fcbe2f494787fc662_2_1380x346.png 2x" data-dominant-color="40424B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">saving_2</span><span class="informations">1530×384 192 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>and I get the embedding for vocab 7:<br>
          <a href="https://i.stack.imgur.com/p3jqF.png" target="_blank" rel="noopener nofollow ugc" class="onebox">
            <img src="https://i.stack.imgur.com/p3jqF.png" width="690" height="173">
          </a>
</p>
<p>But after loading even though my weights are the same (titled Parameter containing: ), my embedding for vocab 7 is different!<br>
          <a href="https://i.stack.imgur.com/0S0vr.png" target="_blank" rel="noopener nofollow ugc" class="onebox">
            <img src="https://i.stack.imgur.com/0S0vr.png" width="690" height="272">
          </a>
</p>
<p>Is there something I’m not aware of or is this expected behavior (some precision issue?) and this is a red herring.</p>
Is this loss function for knowledge distillation correct?

username: Ritvik_Rawat
cooked: <p>I have two networks: student and teacher. Both are being trained for a recognition task (against ground truth values, “targets”). Also I want that the student network should learn from the teacher network.</p>
<p>loss1 : takes care of student learning from ground truth values<br>
loss2:  takes care of teacher learning from ground truth values<br>
dist_loss : Is this correct? I want to ensure that only the student learns from the teacher and not the opposite. Is detaching the output of teacher the correct way to prevent gradients of teacher network being updated from the dist_loss term?</p>
<pre><code class="lang-auto">def dist_loss(t, s):
    prob_t = F.softmax(t/T, dim=1)
    log_prob_s = F.log_softmax(s/T, dim=1)
    dist_loss = -(prob_t*log_prob_s).sum(dim=1).mean()
    return dist_loss

loss1 = F.cross_entropy(student_outputs, targets)
loss2 = F.cross_entropy(teacher_outputs, targets)
teacher_p = teacher_outputs.detach()
loss = loss1 + loss2 + dist_loss(teacher_p, student_outputs)
optimizer.zero_grad()
loss.backward()
optimizer.step()
</code></pre>
<p>Thank you</p>

username: Aryan_Asadian
cooked: <p>If you want to implement the regular offline knowledge distillation, there is no need to add the loss2, since the teacher should be trained already. The loss function for the student is the combination of teacher generated probabilities( high temperatured) and also the ground truth labels.</p>

username: Sagnik_Mukherjee
cooked: <p>Agreed with Aryan. Also, I believe that you should balance, i.e. normalize, your losses rather than simply adding them up.</p>
<p>The standard practice is to pre-train your teacher, save it. And while training the student you take a linear combination of two losses as your loss function.</p>
<ol>
<li>A loss between the gold labels and student’s prediction</li>
<li>A loss between the teacher’s prediction and the student’s prediction</li>
</ol>
<p>This is the pattern followed by the TinyBERT paper. I’ve done this in practice as well and it works pretty decently.</p>

username: achyut_mani_Tripathi
cooked: <p>Exclude loss 2 in your overall calculation</p>
Decompose Conv2d input unfold-gemm-fold

username: zeakey
cooked: <p>I implemented the decomposition of Conv2d into <code>im2col -&gt; gemm -&gt; col2im</code> as below, this is related to <a href="https://discuss.pytorch.org/t/how-to-use-custom-convolution-module-instead-of-torch-nn-conv2d/21458">https://discuss.pytorch.org/t/how-to-use-custom-convolution-module-instead-of-torch-nn-conv2d/21458</a>:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
from torch.nn.parameter import Parameter
import math

class MyConv2d(nn.Module):
    def __init__(self, in_channels, out_channels, kernel_size, stride=1, padding=0, dilation=1):
        super(MyConv2d, self).__init__()
        self.in_channels = in_channels
        self.out_channels = out_channels
        self.kernel_size = kernel_size
        self.stride = stride
        self.padding = padding
        self.dilation = dilation

        self.weight = Parameter(torch.Tensor(out_channels, in_channels, kernel_size, kernel_size))

    def forward(self, input):
        h_in, w_in = input.shape[2:]
        h_out = math.floor((h_in + 2*self.padding - self.dilation*(self.kernel_size-1)-1)/self.stride+1)
        w_out = math.floor((w_in + 2*self.padding - self.dilation*(self.kernel_size-1)-1)/self.stride+1)

        # x: [bs ksize num_sliding]
        x = torch.nn.functional.unfold(input, kernel_size=self.kernel_size, padding=self.padding)

        bs = input.shape[0]
        ksize = self.in_channels*self.kernel_size*self.kernel_size
        num_sliding = x.shape[2]

        assert x.shape[1] == ksize

        # x: [bs*num_sliding ksize]
        x = torch.transpose(x, 1, 2).reshape(-1, ksize)

        weight_flat = self.weight.view(self.out_channels, ksize)
        x = torch.mm(x, weight_flat.t())

        x = x.reshape(bs, num_sliding, self.out_channels)
        x = torch.transpose(x, 1, 2)
        x = torch.nn.functional.fold(x, output_size=[h_out, w_out], kernel_size=1, padding=0, dilation=self.dilation, stride=1)
        return x


if __name__ == '__main__':
    input = torch.rand(10, 32, 1, 1)
    weight = torch.rand(64, 32, 3, 3)

    conv2d = MyConv2d(32, 64, kernel_size=3, padding=1)
    conv2d.weight.data.copy_(weight)

    y = conv2d(input)

    print(y.mean())

    conv2d = nn.Conv2d(32, 64, kernel_size=3, padding=1, bias=False)
    conv2d.weight.data.copy_(weight)

    y = conv2d(input)

    print(y.mean())
</code></pre>
<p>It seems that the <code>stride</code> and <code>kernel_size</code> in <code>torch.nn.functional.fold</code> is DIFFERENT from that in <code>torch.nn.functional.unfold</code>, am I right ?</p>
<p>I’ve manually checked the results by standard <code>Conv2d</code> and <code>MyConv2d</code> and the results are the same.</p>
<p>I think it’s better to give some hints in the documentation to make it clear.</p>
IndexError in the validation dataloader obtained using Subset

username: rubenIng93
cooked: <p>Hi!<br>
I’m trying to perform the validation aver the alexnet trained with the dataset ‘Caltech101’ and I tried different solutions but none these seems to work.<br>
The problem is that I’ve an index error like the following each time I run the validation part:</p>
<p><code>IndexError: index 3457 is out of bounds for axis 0 with size 2892.</code></p>
<p>Here my dataset class’ implementation:<br>
<a href="https://github.com/rubenIng93/Homework2-Caltech101/blob/master/caltech_dataset.py" rel="noopener nofollow ugc">https://github.com/rubenIng93/Homework2-Caltech101/blob/master/caltech_dataset.py</a></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/7/6743a7e850bd72815321b88db2cae0068a8bdad2.png" alt="prep_data" data-base62-sha1="eJw89B3ITMlgOBJUFP86wgBz6mK" width="668" height="365"></p>
<p>Any help would be appreciated!</p>

username: Dreamer4EVA
cooked: <p>you have downsized  train_dataset in line 16</p>
<p>just flip line 16 and line 17</p>

username: rubenIng93
cooked: <p>Oh thanks a lot, I did not see that</p>
Is there anyway to process image tensors on GPU with OpenCV or any other library?

username: sxlllslgh
cooked: <p>I use a network to get images in tensor type on GPU, but there are some postprocesses to do.<br>
Now I’m using OpenCV with CUDA to conduct them.</p>
<pre><code class="lang-python">cpu_numpy = gpu_tensor.byte().cpu().numpy()
gpu_mat = cv2.cuda_GpuMat()
gpu_mat.upload(cpu_numpy)
</code></pre>
<p>But it means image data is copied from GPU memory to memory and back.</p>
<p>Is there any method to process image tensors on GPU directly?</p>
Converting std::vector<Example> into an input tensor in C++

username: pianoman8
cooked: <p><strong>TL;DR:</strong> What’s the best way in C++ to convert the batch-size-length list of std::vector returned by a dataloader into a tensor, where the first dimension is batch_size and the remaining dimensions are the inputs in the Example.data ?</p>
<p>I have a numerical dataset for learning sequence, where the input dataset and output is a Tensor of size N x SEQLEN x INPUT, and the output is N x SEQLEN x OUTPUT.  In Python, I use a TensorDataset, passing both 3D tensors, without issue. However, it appears TensorDataset in C++ doesn’t (yet?) support a labelled dataset (libtorch 1.5).</p>
<p>So I created as custom dataset class and tried to use it with a dataloader (for batching, random, shuffling examples, etc…):</p>
<pre><code class="lang-auto">class CustomDataset : public torch::data::datasets::Dataset&lt;CustomDataset&gt; {
	using Example = torch::data::Example&lt;torch::Tensor, torch::Tensor&gt;;

	torch::Tensor X, Y;

	public:
		CustomDataset(const torch::Tensor&amp; Xs, const torch::Tensor&amp; Ys) : X(Xs), Y(Ys) {}
		Example get(size_t index) {
			return {X[index], Y[index]};
		}
		torch::optional&lt;size_t&gt; size() const {
    			return X.size(0);
  		}
};
</code></pre>
<p>However, when I create a dataloader using this dataset, in my training loop I get a std::vector.  My network’s forward function expects a tensor of shape BATCH_SIZE x SEQLEN x INPUT.</p>
<p>What is the best way to convert this list of Examples into a tensor of the right shape?  Or is there a better way to do this?</p>

username: pianoman8
cooked: <p>I’m able to get past this issue using this construct to manually turn the “list of examples” back into a new 3D tensor… but this seems MASSIVELY inefficient. Surely there’s a better way.</p>
<pre><code class="lang-auto">    		for (auto&amp; batch : *data_loader) {
			    std::vector&lt;torch::Tensor&gt; Xs_list, Ys_list;
		        std::transform(batch.begin(), batch.end(), std::back_inserter(Xs_list), [](auto &amp;e) {return e.data;});
		        std::transform(batch.begin(), batch.end(), std::back_inserter(Ys_list), [](auto &amp;e) {return e.target;});
			    torch::Tensor X = torch::stack(Xs_list);
			    torch::Tensor Y = torch::stack(Ys_list);
      			optimizer.zero_grad();
      			torch::Tensor prediction = model-&gt;forward(X);	
      			torch::Tensor loss = torch::mse_loss(prediction, Y);
....
</code></pre>
<p>Edit: previous version of this post tried to use <code>torch::from_blob(list.data(), ...</code>, and was wrong. Changing that to <code>torch::stack()</code> seems to work a lot better.</p>

username: dambo
cooked: <p>How did you resolve this?<br>
I have a tensor that returns from a module without a batch dimension:</p>
<p><code>torch::Tensor out_tensor = module.forward({input_tensor}).toTensor();</code><br>
However its is 3 by 512 by 512 and this fails:<br>
<code>out_tensor = out_tensor.squeeze().detach().permute({1, 2, 0});</code></p>
<p>How can I add the batch dimension?</p>
<p>Thanks</p>
Looking for LSTM seq2seq example

username: Andi
cooked: <p>I’m experimenting with LSTM models, but can’t find a simple guide/example for a time series (values not text) model predicting n steps into the future (like horizon in fb prophet). Ideal case would be with another “regressor” (like temperature) as an additional dimension for value at timestep x.</p>
<p>My main problem is I’m not sure whether I’m keeping stateful “the right way” and would like to learn from a proper implementation.<br>
My series predicted on 8-&gt;1 and cyclically shifted with a new prediction (last7+newpred) - &gt;1 etc. cycles too fast or gets instable very fast.<br>
Any links / examples would be great.<br>
Thanks in advance, Andy</p>
Is it possible to use PyTorch as backend for Keras?

username: Brando_Miranda
cooked: <p>Is it possible to use PyTorch as backend for Keras?</p>

username: ptrblck
cooked: <p>No, currently only Tensorflow, Theano and CNTK are supported (<a href="https://keras.io/backend/">source</a>). Mxnet seems to be in a beta version.<br>
I doubt that Pytorch will be supported in the near future. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=5" title=":wink:" class="emoji" alt=":wink:"><br>
If you are looking for a high-level API, you could try <a href="https://github.com/pytorch/ignite">Ignite</a> or <a href="https://github.com/dnouri/skorch">Skorch</a>.</p>

username: Brando_Miranda
cooked: <p>what are the advantages and disadvantages of Ignite and Skorch? Which one would you recommend?</p>

username: ptrblck
cooked: <p>Skorch is a scikit-learn compatible wrapper. So if you will feel comfortable using it, if you’re using scikit-learn a lot or like it’s API.<br>
On the other hand, Ignite let’s you define your model and training procedure and just provides some callbacks etc. for convenience. It sounds interesting.</p>
<p>I cannot recommend any wrapper so far, since I’m using plain Pytorch.<br>
For me it’s the right abstraction, so that I can easily define my nets and try some crazy idea.<br>
Using learning rate schedulers etc. I have everything I need so far.<br>
The only point I would like to have is a smoother Visdom integration. At the moment I feel like creating a lot of boilerplate code to visualize the loss etc., but that’s a Visdom issue. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=5" title=":wink:" class="emoji" alt=":wink:"></p>
<p>Give both wrappers a shot and post your experience.</p>

username: Franco_Arda
cooked: <p>I agree. Both, PyTorch and Keras are high-level API’s.<br>
Accessing PyTorch through Keras would be meaningless.</p>

username: alexmath
cooked: <p>I do agree that the high-level abstractions of Keras would be meaningless, especially when the high-level API of PyTorch is soo convenient. However, I have experienced a few cases were it would’ve been convenient when dealing with legacy code.</p>
<p>Example. I needed to use open-source code from a few years back which was written in Keras. I needed to extend the code with some linear algebra functionality I’ve written in PyTorch. The backend of the Keras code was TensorFlow. This forced me to rewrite the PyTorch linear algebra code to TensorFlow. If I could’ve changed the backend I would’ve circumvented having to rewrite from PyTorch to TensorFlow.</p>
Custom quantized Linear or Conv2d layer

username: martinferianc
cooked: <p>Dear Users,</p>
<p>I would like to ask for pointers for how to extend <code>nn.Linear</code> and <code>nn.Conv2d</code> for post-training static quantization or quantization-aware training without rewriting a lot of stuff, such that it can still be used with operator fusion etc… An example change could be to apply an affine transformation to the <em>weights</em> prior to calling the linear operation. Could someone help please? Thanks!</p>

username: jerryzh168
cooked: <p>the change in eager mode quantization will require inheriting <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/quantized/modules/linear.py#L103" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/torch/nn/quantized/modules/linear.py#L103</a> and also related fusion modules under <a href="https://github.com/pytorch/pytorch/tree/master/torch/nn/intrinsic" rel="nofollow noopener">https://github.com/pytorch/pytorch/tree/master/torch/nn/intrinsic</a> folder, and pass a white_list(<a href="https://github.com/pytorch/pytorch/blob/master/torch/quantization/quantize.py#L178" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/torch/quantization/quantize.py#L178</a>) extended with the new module. It will require familiarity of the whole eager mode quantization flow.</p>

username: martinferianc
cooked: <aside class="quote no-group" data-username="jerryzh168" data-post="2" data-topic="93514">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jerryzh168/48/15217_2.png" class="avatar"> jerryzh168:</div>
<blockquote>
<p><a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/quantized/modules/linear.py#L103" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/master/torch/nn/quantized/modules/linear.py#L103</a></p>
</blockquote>
</aside>
<p>Thanks Jerry, this is what I initially thought, but I wanted to double-check if my assumption was right. Thank you!</p>
Comm.broadcast_coalesced not returning

username: ignacio-rocco
cooked: <p>Hi, I’m trying to use DataParallel on a build from master (0.4.0a0+361baa5) but the replication of the module which calls comm.broadcast_coalesced is not returning. The program just hangs there.</p>
<p>I wonder if it has anything to do with <a href="https://github.com/pytorch/pytorch/pull/4999" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/4999</a>. Any other ideas?</p>
<p>Maybe <a class="mention" href="/u/fmassa">@fmassa</a> has some ideas? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>EDIT: I think there is a problem with NCCL, I’m now trying with NO_SYSTEM_NCCL</p>
<p>EDIT2: Recompiling PyTorch with the included NCCL instead of the system NCCL solved the issue</p>

username: deJQK
cooked: <p>Hi Ignacio-Rocco, I just met the exactly same problem and in my case it seems happen only occasionally, and sometimes there is no such problem. I wonder if you could explain a little more on the point “recompiling pytorch with the ‘included’ NCCL”. What do you mean by ‘included NCCL’?</p>
<p>I am using torch 1.4.0 .</p>
<p>Thanks a lot.</p>
<p>EDIT: The reason is setting somewhere <code>os.environment['CUDA_LAUNCH_BLOCKING'] = "1"</code> .</p>
How do I get the rank from torch.lstsq?

username: Shisho_Sama
cooked: <p>in numpy the <code>numpy.linalg.lstsq(a,b)</code> returns 4 values which are solution, residuals, rank and singulare values of a .<a href="https://numpy.org/doc/stable/reference/generated/numpy.linalg.lstsq.html" rel="nofollow noopener">ref</a>. However Pytorch <a href="https://pytorch.org/docs/stable/generated/torch.lstsq.html" rel="nofollow noopener">version</a> retuns only a tuple of Solution and QR.<br>
How am I supposed to get the ranks here? I’m lost here!</p>
<p>Any help is greatly apprecaited.</p>
Strange loss/epoch plot

username: Jona
cooked: <p>Hi everyone,</p>
<p>I took the exercise 2 from Andrew NG’s ML course on Coursera, and now, I’m trying to do it in pytorch …</p>
<p>I did the first exercise, a simple univariate linear regresion, and it worked well. In the exercise 2, a multivariate linear regresion, the gradient descent seems to work</p>
<p>However, I can’t get why the y-axis of the loss/epoch plot is to high (1e+11), eventhoug the r2 scores are well for train and test sets (you can see the plots at the bottom)</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/e/ae0614d756e3f39150210e474194af28e77d4959.png" alt="image" data-base62-sha1="oPu6LQm6LoghPCHWC3Z4ZuHfObL" width="386" height="267"></p>
<p>Can you help me find the correct conceptual and technical answer to this question?</p>
<p>Here is the NB (<code>Pytorch NN) Linear Regression II.ipynb</code>)</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://console.paperspace.com/tegoxhoz0/notebook/pryj29evg">
  <header class="source">
      <img src="https://console.paperspace.com/tegoxhoz0/notebook/src/favicon.svg" class="site-icon" width="" height="">

      <a href="https://console.paperspace.com/tegoxhoz0/notebook/pryj29evg" target="_blank" rel="noopener nofollow ugc">console.paperspace.com</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/433;"><img src="https://blog.paperspace.com/content/images/size/w2000/2020/05/5eb8ce053cb50746e44069c0_cluster-bg--1-.jpg" class="thumbnail" width="690" height="433"></div>

<h3><a href="https://console.paperspace.com/tegoxhoz0/notebook/pryj29evg" target="_blank" rel="noopener nofollow ugc">Paperspace</a></h3>

  <p>Build and scale ML applications with a cloud platform focused on speed and simplicity.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Bes regards</p>
Is == operator defined for tensor elements?

username: Shisho_Sama
cooked: <p>hello everyone.<br>
I wanted to do a simple check  like :</p>
<pre><code class="lang-cpp">if (output_size.numel() &gt; 0 &amp;&amp; output_size[0] == tmp_crop_size[0] &amp;&amp; output_size[1] == tmp_crop_size[1])
    ....
</code></pre>
<p>but ultimately endedup with:</p>
<pre><code class="lang-cpp">if (output_size.numel() &gt; 0 &amp;&amp; output_size[0].equal(tmp_crop_size[0]) &amp;&amp; output_size[1].equal(tmp_crop_size[1]))
    ....
</code></pre>
<p>it seems the <code>==</code> operator is not defined/implemented for each tensor element!</p>
Weight Sharing - weights for the next step

username: Ali_Akgoz
cooked: <p>Hi, I will use same layer many times (count will change at each batch) in my model, I found that example:</p>
<pre><code class="lang-auto">    def forward(self, x):
        """
        For the forward pass of the model, we randomly choose either 0, 1, 2, or 3
        and reuse the middle_linear Module that many times to compute hidden layer
        representations.

        Since each forward pass builds a dynamic computation graph, we can use normal
        Python control-flow operators like loops or conditional statements when
        defining the forward pass of the model.

        Here we also see that it is perfectly safe to reuse the same Module many
        times when defining a computational graph. This is a big improvement from Lua
        Torch, where each Module could be used only once.
        """
        h_relu = self.input_linear(x).clamp(min=0)
        for _ in range(random.randint(0, 3)):
            h_relu = self.middle_linear(h_relu).clamp(min=0)
        y_pred = self.output_linear(h_relu)
        return y_pred

</code></pre>
<p>My question is if I use this code it will create lets say 3 middle layers and will use same weights for all, but what about gradients? I mean during training different gradients will be calculated for each middle layer. When backpropagation finishes at the next step their weights will be different from each other. How pytorch will calculate the weight of that layer for the next batch? Will it take average? or take the last layer’s weights?</p>
<p>Thanks,<br>
Ali</p>

username: ptrblck
cooked: <p>By default the gradients will be accumulated in the parameters.</p>
<aside class="quote no-group" data-username="Ali_Akgoz" data-post="1" data-topic="93449">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ali_akgoz/48/11824_2.png" class="avatar"> Ali_Akgoz:</div>
<blockquote>
<p>When backpropagation finishes at the next step their weights will be different from each other.</p>
</blockquote>
</aside>
<p>That shouldn’t be the case, since the parameters are not updated during the <code>backward()</code> call, but by the <code>optimizer.step()</code>.</p>
What is the best way to declare the flatten Layer in Pytorch

username: Harsh_Ris
cooked: <p>In general, we have to calculate the dimension of previous layer to declare a fully connected layer(flattened layer) after performing convolutions and pooling operations.</p>
<p>Some forums also displayed the use of functions for the purpose like</p>
<pre><code class="lang-auto">def count_input_neuron(model, image_dim):
    return model(torch.rand(1, *(image_dim))).data.view(1, -1).size(1)

</code></pre>
<p>which can be seen here on <a href="https://discuss.pytorch.org/t/linear-layer-input-neurons-number-calculation-after-conv2d/28659/3">Pytorch forum</a> and <a href="https://stackoverflow.com/questions/55875279/how-to-get-an-output-dimension-for-each-layer-of-the-neural-network-in-pytorch" rel="nofollow noopener">Stack overflow</a></p>
<p>whereas some used flatten function of the torch</p>

username: ptrblck
cooked: <p>You could either use the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Flatten.html#torch.nn.Flatten"><code>nn.Flatten</code></a> module directly or use the functional API via <code>torch.flatten</code> or <code>tensor.view</code> in the forward method.<br>
The approach depends on your use case (e.g. <code>nn.Flatten</code> should be used in an <code>nn.Sequential</code> container) and your coding style.</p>
Pure Python pip install and external CUDA installation

username: ptab
cooked: <p>Dear forum members,</p>
<p>On my new Windows 10 AMD Ryzen system I wanted to go for a pure python (non-Anaconda) based pytorch install, since I didn’t get numpy to work well with openblas on Anaconda.</p>
<p>So I created a fresh virtual environment, first installed numpy and then installed pytorch by running</p>
<pre><code class="lang-auto">pip3 install torch===1.3.0 torchvision===0.4.1 -f https://download.pytorch.org/whl/torch_stable.html
</code></pre>
<p>The installation finished but when importing torch I ran into the error</p>
<pre><code class="lang-auto">from torch._C import *
ImportError: DLL load failed: The specified module could not be found
</code></pre>
<p>I tried installing the cpu-only version of torch which had no issues.</p>
<p>I was finally able to fix the error on the GPU-version of pytorch by installing CUDA manually before installing pytorch. (again by running <code>pip3 install torch===1.3.0 torchvision===0.4.1 -f https://download.pytorch.org/whl/torch_stable.html</code>)</p>
<p>This confuses me as it was my understanding that pytorch brings it’s own CUDA and cudNN which are independent from the system’s CUDA installation.<br>
So I would like to understand how installing CUDA could have fixed my problem?</p>
<p>I would be grateful for some insight into this behaviour, as I would like to understand how my pytorch installation is configured.</p>
<p>Thank you,<br>
Paul</p>

username: albanD
cooked: <p>Hi,</p>
<p>pytorch does not bring cuda. It does bring cudnn though.<br>
In conda, we have cuda as a dependency because conda package can contain cuda. But it’s not the case in pip, so you need to install the matching cuda version to be able to use cuda with the pip installed package.</p>

username: adongyo
cooked: <p>Thank you! I am new to pytorch.<br>
Yesterday I buy a new PC,and  I did’t install the CUDA manually, but  when I download the<br>
pytorch cuda==10.1  with pip, I import it and test, it turns out the  cuda is available!  I don’t konw why!<br>
I input the “nvcc -V”  in CMD but  it does’t work.<br>
then, I installed the tensorflow-gpu in another pip venv, but it  does’t work<br>
<strong>I have no idea about  if I have the CUDA!</strong><br>
I want to know how to confirm the CUDA is exist?<br>
thanks!</p>

username: ptrblck
cooked: <p>Double post from <a href="https://discuss.pytorch.org/t/can-i-link-pytorch-to-already-installed-cuda/36279/6">here</a> with an answer.</p>
How to oversample the imbalanced classes in BoW

username: kit_m
cooked: <p>Suppose my task is to build a BoW predictor from image features. Inputs are image features and outputs are BoW vectors. Each element of the BoW are cat, dog, pig, panda and penguin. The BoW in the training data is imbalanced. For example, [1, 0, 0, 0, 0], [1, 0, 1, 0, 0],[1, 1, 1, 0, 0], [1, 0, 0, 0, 1],[1, 0, 1, 1, 0]…</p>
<p>My question is how I should oversample the data? In the example above, panda is the minority class. If I try to increase frequencies of panda by oversampling, cat and pig also increase as well.<br>
Should I use <code>torch.utils.data.distributed.</code> <code>DistributedSampler</code>?</p>

username: ptrblck
cooked: <p><a href="https://discuss.pytorch.org/t/unbalanced-data-in-multiple-object-detection/88104/2">This post</a> has some suggestions for different sampling implementation when dealing with an imbalanced multi-label dataset. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Curious about nn.TransformerDecoder

username: 111378
cooked: <p>I am new at Transformer</p>
<p>My curiosity is this.</p>
<p>I will explain it with example.</p>
<p>I know there are two type of masks</p>
<p>subsquent mask and padding mask( of course there are memory mask in nn.Transformer but it was not in original Transformer so I ommitted)</p>
<p>memory is the output of encoder right?</p>
<p>so is it right that memory key padding mask(in TransformerDecoder) is same as src key padding mask (in nn.TransformerEncoder)</p>
<p>or keep it simple, when I want to use original Transfomer(in the paper), is it right that I don’t have to use any of memory mask and memory key padding mask in TransformerDecoder?</p>
Model.cuda() vs. model.to(device)

username: tcsn_wty
cooked: <p>I suppose that <code>model.cuda()</code> and <code>model.to(device)</code> are the same, but they actually gave me different running time.</p>
<p>I have the follwoing:</p>
<pre><code class="lang-auto">device = torch.device("cuda")
model = model_name.from_pretrained("./my_module")  # load my saved model
tokenizer = tokenizer_name.from_pretrained("./my_module")  # load tokenizer
model.to(device)  # I think no assignment is needed since it's not a tensor
model.eval()  # I run my model for testing
</code></pre>
<p>However, later testing process takes 2 min 19 sec, which is different from if I do <code>model.cuda()</code> instead of <code>model.to(device)</code>, while the latter takes 1 min 08 sec. I know they both are fast, but I don’t understand why their running times are quite different while the two ways of coding should be the same thing.</p>
<p>I was wondering due to the data amount, time might fluctuate. I can understand that. But I just wanna make sure the the two ways of coding a the same.</p>

username: albanD
cooked: <p>Hi,</p>
<p>They do the same thing yes: send each param to the GPU one after the other.<br>
Are you sure that you don’t have something else on the machine that could be using either the GPU, the CPU or the disk and that would slow down your eval?</p>

username: tcsn_wty
cooked: <p>Hi,</p>
<p>Yes, I didn’t modify any line of code except changing the ways of utilizing GPU. If they actually do the same thing, then I guess it might due to the case that warm-up time varies.</p>

username: albanD
cooked: <p>Note that if it is a shared machine, this kind of thing can also be cause by other users <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: tcsn_wty
cooked: <p>That makes sense. Thank you!</p>