akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Getting Cuda OOM error while training coreference resolution model through spacy-experimental

username: Tanmay
cooked: <p>Hi everyone,<br>
I have been trying to train my own coreference resolution model on google colab but I keep on getting  <code>torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 23.32 GiB (GPU 0; 39.59 GiB total capacity; 23.95 GiB already allocated; 12.29 GiB free; 24.00 GiB reserved in total by PyTorch)</code></p>
<p>I tried solving it by setting the environment variable <code>os.environ["PYTORCH_CUDA_ALLOC_CONF"] = 'max_split_size_mb: 256'</code> as recommended by others but I am still getting this error.</p>
<p>Can anyone help me how can I fix it?</p>
How to create Traceable and Differentiable CUSTOM C++ OPERATORS with CUSTOM C++ CLASSES as argument in Python?

username: wpybtw
cooked: <p>I learnt CUSTOM C++ OPERATORS and CUSTOM C++ CLASSES [ <a href="https://pytorch.org/tutorials/advanced/torch_script_custom_classes.html" rel="noopener nofollow ugc">Extending TorchScript with Custom C++ Classes — PyTorch Tutorials 1.13.1+cu117 documentation</a>].<br>
I can create autograd::Function using custom OPs and custom classes and use it in C++.</p>
<p>But what if I want to reuse that written autograd::Function in Python.</p>
<ol>
<li>Are there any methods to do so?</li>
<li>Or I must implement Python-version autograd.Function with the custom ops? Would it be Traceable and Differentiable?</li>
</ol>
<p>I saw one blog <a href="https://lernapparat.de/pytorch-traceable-differentiable" rel="noopener nofollow ugc">Lernapparat - Machine Learning</a> using TraceableFunction. But it is rather old and I don’t know if it can support custom class. Are there any new recommended methods?</p>
Confusion about backward of clamp operation

username: Mirocos
cooked: <pre data-code-wrap="c++"><code class="lang-plaintext">    torch::Tensor t = torch::tensor(0.0 , torch::kFloat32);
    t.requires_grad_(true);
    std::vector&lt;float&gt; data{ -0.2, 0.3, 0.4, -0.1, -0.2, -0.2 };
    torch::Tensor a = torch::from_blob(data.data(), { 2, 3 }, torch::kFloat32);
    a.requires_grad_(true);
    auto mr = torch::clamp(a, t);
    mr.backward(torch::ones_like(mr));

    LOG(INFO) &lt;&lt; t.grad();
    LOG(INFO) &lt;&lt; a.grad();
</code></pre>
<p>I what to figure out the gradient of second parameter in clamp function.<br>
And I have  above code, dmrdt equals to 4 which looks like its gradient equals to the number of element that less than t(0.0).</p>
<p>I didn’t not found any souce code of <code>torch::clamp</code> in LibTorch, but only thing I cound found is something in Python style. It seems the blow code do not return the gradient of second parameter or third parameter.</p>
<pre><code class="lang-python">class Clamp(Function):

    @staticmethod
    def forward(ctx, i, min_val, max_val):
        ctx._mask = (i.ge(min_val) * i.le(max_val))
        return i.clamp(min_val, max_val)

    @staticmethod
    def backward(ctx, grad_output):
        mask = Variable(ctx._mask.type_as(grad_output.data))
        return grad_output * mask, None, None
</code></pre>
<p>So what confused me is that how LibTorch produce the gradient of <code>dmrdt</code>?</p>

username: ptrblck
cooked: <p>I think the derivative is defined <a href="https://github.com/pytorch/pytorch/blob/df4b3b13bc84e5016854670e61a8ff0c635e2d7c/tools/autograd/derivatives.yaml#L383-L414">here</a> and this comment might be relevant:</p>
<blockquote>
<p>For clamp, gradient is not defined at the boundaries. But empirically it’s helpful to be able to get gradient on min and max, so we return the subgradient 1 for these cases.</p>
</blockquote>
What's the point of an activation function for the output layer for a regression problem?

username: Dev2150
cooked: <p>For instance, the DQN will perform the decision with the highest Q-value.<br>
Is there a point in processing the output layer e.g. with a .softmax()?</p>

username: J_Johnson
cooked: <p>Sigmoid maps the values between 0 and 1. This is typically done for probabilities and works well with certain loss functions. Perhaps the person who made the DQN network just did so out of habit. I’ve also seen dropout commonly used in DQNs but have never seen anyone prove this is of any benefit in those types of networks.</p>
<p>Check your reward function. If it’s giving values outside of the range of 0 to 1, you might do well to try removing the final sigmoid function.</p>
How can I get broadcasting benefits?

username: reza_yazdi
cooked: <p>I have a simple custom NN as below. it works but is very slow. every epoch was accomplished in about 27 minutes !!! how can I vectorize and utilize broadcasting in my implementation?</p>
<pre><code class="lang-auto">class MaxSigNet(nn.Module):

    def __init__(self, in_channels, out_channels):
        super(MaxSigNet, self).__init__()
       
        mywheights=torch.tensor([[[[(0), (15/255.0), (3/255.0), (4/255.0), (0)],
               [(37/255.0), (1/255.0), (2/255.0), (4/255.0), (6/255.0)],
               [(65/255.0), (128/255.0), (0), (8/255.0), (9/255.0)],
               [(68/255.0), (64/255.0), (32/255.0), (16/255.0), (15/255.0)],
               [(0), (58/255.0), (37/255.0), (24/255.0), (0)]]]],dtype=torch.float32,requires_grad=True)
                
        mymaps=torch.tensor([[[[0,1,1,1,0],
                               [1,1,1,1,1],
                               [1,1,0,1,1],
                               [1,1,1,1,1],
                               [0,1,1,1,0]]]],dtype=torch.float32,requires_grad=False)
        
        self.weight=nn.Parameter(torch.mul(mywheights,mymaps))
        print("my initial weights are:",self.weight)
        self.map=nn.Parameter(mymaps)
        self.out_channels=out_channels
        self.in_channels=in_channels
        
    def Apply_MaxSig(self,img):  
        imgn=F.pad(input=img, pad=(2,2,2,2), mode='constant',value = 0.0)
        newimgn=torch.zeros_like(imgn)
        img_max=imgn.max()

        for i in range(2,imgn.shape[0]-2):
            for j in range(2,imgn.shape[1]-2):

                a=torch.flatten(torch.mul(self.weight,self.map))
                b=torch.flatten(imgn[i-2:i+3,j-2:j+3])
                
                siga=torch.sigmoid(a)
                
                sigb=torch.sigmoid(b/img_max)
               
                aa=torch.maximum(siga,sigb)
                max5=torch.sum(aa)        
                
                newimgn[i,j]=(24*sigb[12] - max5)
       
        return newimgn[2:-2,2:-2]#undo pad
 
    def forward(self, input_image):
        for i in range(self.out_channels):
            for j in range(self.in_channels):
                
                img=input_image[i,j]
                img=self.Apply_MaxSig(img)
                input_image[i,j]=img                
                
        return input_image

class build_simple_net(nn.Module):
    def __init__(self):
        super().__init__()
        self.s0 = MaxSigNet(1,1)

    def forward(self, inputs):
        return self.s0(inputs)
</code></pre>

username: reza_yazdi
cooked: <p>Hi guys. I solved this. maybe useful for somebody.</p>
<pre><code class="lang-auto">def forward(self, input_image):
        B=torch.nn.functional.unfold(torch.sigmoid(input_image), kernel_size=(5,5), padding=2, stride=1)
        B=B.reshape(input_image.shape[0],25,input_image.shape[2] , input_image.shape[3])
        C=torch.sigmoid(self.weight).reshape(1,25,1,1)
        D=torch.maximum(C,B) * self.map.reshape(1,25,1,1)
        E=torch.sum(D,axis=1)
        F=24*B[:,12,...] - E     
        return F.unsqueeze(dim=1)
</code></pre>
"Torch not compiled with CUDA enabled" error, anaconda environment

username: Pavils_Jurjans
cooked: <p>I am trying to set up and run Stable Diffusion 2.1 on my Ubuntu. It has NVIDIA GeForce RTX 2070 Super graphics card (2560 CUDA Cores, 8192 MB total memory). The VRAM size is barely enough to run the Stable Diffusion, but I know it is possible.</p>
<p>I am being haunted by this  just to keep returning to this <code>AssertionError: Torch not compiled with CUDA enabled</code> error. I have been trying to uninstall, reinstall PyTorch in all kinds of ways, just to keep returning to the same error.</p>
<p>I have installed it in a conda environment. Running <code>torch.cuda.is_available()</code> gives <code>False</code>.</p>
<p>What is the suggested way to completely remove PyTorch in order to start anew? I have tried this:</p>
<pre><code class="lang-auto">$ pip uninstall torch
$ pip cache purge
$ conda remove pytorch
$ conda clean --all
</code></pre>
<p>Then I do</p>
<pre><code class="lang-auto">$ conda install pytorch==1.12.1 -c pytorch
</code></pre>
<p>When I execute that command, I get the <code>The following packages will be downloaded</code> list, and I see there <code>pytorch-1.12.1 | cpu_py310he8d8e81_0 | 60.9 MB</code>. That “cpu” there in the build name doesn’t look promising. On what basis conda has decided that it’s the right build to download and install? Is there a way I can force specific build?</p>

username: ptrblck
cooked: <p>Please use these <a href="https://pytorch.org/get-started/locally/">install instructions</a> and select a valid CUDA runtime version (11.6 or 11.7) since you are installing the CPU-only conda binaries right now.</p>
Why does my custom GRU doesn't match pytorch GRU implementation?

username: Nelson_Sharma
cooked: <p>in the file <code>torch\nn\modules\rnn.py</code></p>
<p>I replaced the _VF with custom code  i.e.</p>
<pre><code class="lang-auto">ret = _VF.gru_cell(
    input, hx,
    self.weight_ih, self.weight_hh,
    self.bias_ih, self.bias_hh,
)
</code></pre>
<p>replaced by</p>
<pre><code class="lang-auto">import torch as tt
import torch.nn.functional as ff
</code></pre>
<pre><code class="lang-auto">r1, r2 = 0, 1
z1, z2 = 1, 2
n1, n2 = 2, 3
R = tt.sigmoid(
    ff.linear(input, self.weight_ih[r1*self.hidden_size:r2*self.hidden_size,:], self.bias_ih[r1*self.hidden_size:r2*self.hidden_size]) \
        + \
    ff.linear(hx, self.weight_hh[r1*self.hidden_size:r2*self.hidden_size,:], self.bias_hh[r1*self.hidden_size:r2*self.hidden_size]) 
)
Z = tt.sigmoid(
    ff.linear(input, self.weight_ih[z1*self.hidden_size:z2*self.hidden_size,:], self.bias_ih[z1*self.hidden_size:z2*self.hidden_size]) \
        + \
    ff.linear(hx, self.weight_hh[z1*self.hidden_size:z2*self.hidden_size,:], self.bias_hh[z1*self.hidden_size:z2*self.hidden_size]) 
)
N = tt.tanh(
    ff.linear(input, self.weight_ih[n1*self.hidden_size:n2*self.hidden_size,:], self.bias_ih[n1*self.hidden_size:n2*self.hidden_size]) \
        + \
    R * ff.linear(hx, self.weight_hh[n1*self.hidden_size:n2*self.hidden_size,:], self.bias_hh[n1*self.hidden_size:n2*self.hidden_size]) 
)
ret = (1-Z) * N + (Z * hx)
</code></pre>
<p>I tested with some random data but there is difference between pytorch’s original GRU Cell.</p>
<pre><code class="lang-auto"># h_gru_cell_torch is output of torch.GRUCell
# h_gru_cell_custom output of my GRUCell
d_h_gru_cell = tt.sum(tt.abs((h_gru_cell_torch-h_gru_cell_custom)))
d_h_gru_cell=tensor(2.0512e-07)
</code></pre>
<p>is this ok or am I doing something wrong?</p>
Does batch size matters on inference

username: avatar
cooked: <p>I’m trying to evaluate a semantic segmentation model on Pascal Voc 2012 dataset. I’m using the mIoU metric. I’m evaluating the model with different batch sizes. For example, I tried batch size = 1 and batch size = 16 and I got different mIoU. Especially, with batch size = 1, I take smaller mIoU than with batch size =16. Does batch size affects the performance on evaluation?</p>
<p>Here is the code for mIoU metric:</p>
<pre><code class="lang-auto">def mIoU(pred_mask, mask, smooth=1e-10, n_classes=21):
    with torch.no_grad():
        pred_mask = pred_mask.to("cpu").contiguous().view(-1)
        mask = mask.to("cpu").contiguous().view(-1)
     
        iou_per_class = []
        for clas in range(0, n_classes): #loop per pixel class
            true_class = pred_mask == clas
            true_label = mask == clas
  
            if true_label.long().sum().item() == 0: #no exist label in this loop
                iou_per_class.append(np.nan)
            else:
                intersect = torch.logical_and(true_class, true_label).sum().float().item()
                union = torch.logical_or(true_class, true_label).sum().float().item()

                iou = (intersect + smooth) / (union + smooth)
                iou_per_class.append(iou)
        return np.nanmean(iou_per_class)
</code></pre>

username: ptrblck
cooked: <p>Are you comparing the single sample metric vs. the mean of the batch or are you calculating the mean of multiple samples vs. the mean of a batch?<br>
In any case, I would probably start by comparing the results of two pre-defined samples where the <code>mIoU</code> is known and pre-computed and check which approach yields invalid results.</p>

username: avatar
cooked: <p>I actually compute the mean of the mIoUs of each sample  when I set batch size = 1. On the other hand, when I set batch size = 16, I compute the mIoU over the current batch and then the mean over the total batches. From these two, i’m taking different results. Here is the code.</p>
<pre><code class="lang-auto">def evaluation(dataloader):
  miou = []
  deeplab.load_state_dict(torch.load("/content/drive/MyDrive/best_model1"))
  deeplab.eval()
  with torch.no_grad():
    for image, label in dataloader:
      X_batch = image.to(device).float()
      y_batch = label.to(device)
      preds = deeplab(X_batch)
      _ , predictions = torch.max(preds, dim=1)
      iou_score = mIoU(predictions , y_batch)
      miou.append(iou_score)
  
  score = np.array(miou)
  

  return np.mean(score)
</code></pre>

username: carbocation
cooked: <p>If your batch sizes are not all the same (e.g., your final batch isn’t the same size as the rest), then this will not yield the same results as separately calculating the metric for each item.</p>
Why does .cpu() yield TensorBase

username: Matthias_Moller
cooked: <p>For certain I/O functionality I need to transfer tensors that potentially reside in GPU memory to host memory. The following code works</p>
<pre><code class="lang-auto">torch::tensor a;
auto a_cpu = a.cpu();
auto a_accessor = a_cpu.accessor&lt;float, 1&gt;();
</code></pre>
<p>However</p>
<pre><code class="lang-auto">auto a_accessor = a.cpu().accessor&lt;float, 1&gt;();
</code></pre>
<p>does not compile with the following compiler error</p>
<pre><code class="lang-auto">libtorch/1.13.1/include/ATen/core/TensorBase.h:562:23: note: candidate function [with T = double, N = 1] has been explicitly deleted
  TensorAccessor&lt;T,N&gt; accessor() &amp;&amp; = delete;
</code></pre>
<p>Is it possible to do the transfer to host memory inline?</p>
Class activation mapping for videos

username: kzhang3256
cooked: <p>Does anyone know how to do class activation mapping for a video (during classification for action detection)? I’m using resnet3d50 architecture as pre-trained model.<br>
Thanks!</p>

username: harish2006
cooked: <p>Hi there, I saw your question. I also want to do this . Did you find a method for this? Have you tried running gradcam on an existing video classification network, does that work. I was wondering if you ran framewise sliding windows of a video through pytorch video and were able to get a class activation map for even one representative frame in a window. We can then stack all the maps and get a spatiotemporal blob. Do you think this will work?</p>
How to see what is backward doing?

username: Martina_Ragulikova
cooked: <p>Hi, how to see here how backward executed the derivation? Is it possible to see computational graph? Or is it possible anything how he does the backward of CustomCrossEntropy?<br>
I want to reproduce it manually.</p>
<p>Thank you.</p>
<pre><code class="lang-auto">import torch
def CustomCrossEntropy(y_pred, y_target):
    # convert class ID to one hot encoding
    one_hot = torch.zeros_like(y_pred)
    one_hot[range(y_target.shape[0]), y_target] = 1.0

    # compute cross entropy without softmax
    ce = -one_hot * (y_pred - y_pred.exp().sum(1).log().unsqueeze(1))

    # reduction mean
    result = ce.sum(dim=1).mean()
    return result


batch_size = 3
classes_count = 10
target_class_id = torch.randint(0, classes_count, (batch_size,))

# initial logits value
x_initial = torch.randn((batch_size, classes_count))
xc = torch.nn.Parameter(x_initial.clone(), requires_grad=True)

loss_func_c = CustomCrossEntropy

loss_c = loss_func_c(xc, target_class_id)
loss_c.backward()

print(loss_c)
</code></pre>

username: AlphaBetaGamma96
cooked: <p>Hi <a class="mention" href="/u/martina_ragulikova">@Martina_Ragulikova</a>,</p>
<p>Have a look at the torchviz library to visualize the computational graph.</p>
<p>The github repo is here: <a href="https://github.com/szagoruyko/pytorchviz" class="inline-onebox" rel="noopener nofollow ugc">GitHub - szagoruyko/pytorchviz: A small package to create visualizations of PyTorch execution graphs</a></p>

username: Martina_Ragulikova
cooked: <p>Hi thanks for reply, I think that is not enough, isn’t it possible to see mathematical operations?</p>

username: AlphaBetaGamma96
cooked: <p>The comp. graph won’t show you an explicit formula, but it’ll show you the forward pass of your function. This will also show the relative backward formula via the associated backward function, the github repo has a figure showing this. If you want to find the explicit formula for the derivative this is pretty hard to do.</p>
<p>You’ll need to identify every function in your forward pass, and then find the associated backward formula during the backward pass. The backward formulae are defined here (in C++ for reference, not Python), I believe: <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/autograd/FunctionsManual.cpp" class="inline-onebox" rel="noopener nofollow ugc">pytorch/FunctionsManual.cpp at master · pytorch/pytorch · GitHub</a></p>
<p>If you want an explicit expression for the backward, you’ll need to manually define the backward pass and represent it as an <code>torch.autograd.Function</code> object.</p>

username: Martina_Ragulikova
cooked: <p>So you think that finding derivative or “backward” of this function is hard to do?</p>
<pre><code class="lang-auto">ce = -x * (y - y.exp().sum(1).log().unsqueeze(1))
result = ce.sum(dim=1).mean()
</code></pre>
<p>Here is similar topic - <a href="https://discuss.pytorch.org/t/backward-of-crossentropyloss/169740/5" class="inline-onebox">Backward of crossentropyloss - #5 by Martina_Ragulikova</a></p>
<p>And I showed there same thing with MSE function but can not figure out backward for this one…</p>
Different batch size causes accuracy drop

username: Yuxuan_Xue
cooked: <p>Hi all,</p>
<p>i trained my module (mainly conv1d + dropout + BN) with batch size of 8.</p>
<p>In the evaluation, i activated model.eval(). I met a massive accuracy drop and thus i follow the post here: <a href="https://discuss.pytorch.org/t/performance-highly-degraded-when-eval-is-activated-in-the-test-phase/3323/71" class="inline-onebox">Performance highly degraded when eval() is activated in the test phase - #71 by Yuxuan_Xue</a>, which solves the issue when i set the batch size to 1. More specifically, my model performs well when the running stats of batch norm are manually set in evaluation mode.</p>
<pre><code class="lang-auto">@staticmethod
  def set_batch_norm_running_stats(module):
      for m in module.modules():
          for child in m.children():
              if type(child) == nn.BatchNorm2d or type(child) == nn.BatchNorm1d:
                  child.track_running_stats = False
                  child.running_mean = None
                  child.running_var = None
</code></pre>
<p>currently, i meet an issue: i can only use batch size of 1. If i use a higher batch size (e.g. 2, 4, or 8), the model again has massive accuracy drop.</p>
<p>Also one weird thing is that i have two modules containing batch normalization. I did the ablation study (set each module with <code>set_batch_norm_running_stats</code> function separately and do evaluation). However, only one module is affected. Another module is not affected and doesn’t need to be set the running stats manually to perform normally.</p>
<p>I am really confused now and really appreciate for any suggestions and discussion.</p>
PCA Reduces Acc drastically

username: Tornike
cooked: <p>Hi, It might not be pytorch Question but it is About Machine Learning.</p>
<p>Doing Sentiment Analysis Using Traditional ML Algorithm(Either SVM or RF). So, Without PCA i either overfit or have a very low ACC.</p>
<p>To avoid overfitting i try to use PCA on a text data. This is the code :</p>
<pre><code class="lang-auto">vectorizer = CountVectorizer(token_pattern=r'[^\s]+',ngram_range=(1, 2))
vectors = vectorizer.fit_transform(df["Comments"])
svd = TruncatedSVD(n_components=10, random_state=42)
data = svd.fit_transform(vectors) 
for train_index, test_index in skf.split(data, y):
    X_train, X_test = data[train_index], data[test_index]
    y_train, y_test = y[train_index], y[test_index]

    text_classifier = SVC(C=100, decision_function_shape='ovo', gamma=0.01)
    text_classifier.fit(X_train, y_train)
    train_yhat = text_classifier.predict(X_train)
    train_acc = accuracy_score(y_train, train_yhat)
    train_scores.append(train_acc)
    test_yhat = text_classifier.predict(X_test)
    test_acc = accuracy_score(y_test, test_yhat)
    test_scores.append(test_acc)

</code></pre>
<p>In this case, model has low acc. Without PCa it overfits.</p>
<p>What Should i do?</p>

username: Tornike
cooked: <p>Or is there Any other way to avoid Overfitting?</p>

username: carbocation
cooked: <p>If you are running PCA on all of your data, you may be (inappropriately) contaminating the training data with information from the validation data. That would be consistent with the observations you have described. (To be clear, I mean with respect to your statement “Without PCA i either overfit or have a very low ACC.” That is the opposite of the title of the post, by the way.)</p>

username: Tornike
cooked: <p>To be More clear, PCA reduces ACC when i overfit the data without PCA.</p>
<p>i’m fitting different hyperparameters. some have very low acc. 1-2 overfits.</p>
<p>when it overfits i use PCA and it reduces the ACC drastically.</p>
<p>Now i’m trying to use Linear Discriminant Analysis(LDA) which is another Dimensionality reduction but i could not understand how to prepare data for that.  LDA of course can’t read string, it should be digits. but once i try to vectorize the data and then apply it to LDA i have a Memory Crash In Colab.</p>
Torch.autograd.functional.hessian

username: AUM
cooked: <p>I am trying to calculate the trace of Hessian of a sample neural network model.</p>
<p>I am using <code>torch.autograd.functional.hessian </code> which basically returns a tuple of tuple of Tensors</p>
<p>for simplicity, let’s assume we have a linear model with 2 inputs and 2 outputs this will be my Hessian in this case for a random input and chosen loss function.</p>
<pre><code class="lang-auto">H =
((tensor([[[[ 0.1600,  0.1540],
            [-0.1600, -0.1540]],
  
           [[ 0.1540,  0.1482],
            [-0.1540, -0.1482]]],
  
  
          [[[-0.1600, -0.1540],
            [ 0.1600,  0.1540]],
  
           [[-0.1540, -0.1482],
            [ 0.1540,  0.1482]]]], device='cuda:0', grad_fn=&lt;ViewBackward0&gt;),
  tensor([[[ 0.1972, -0.1972],
           [ 0.1898, -0.1898]],
  
          [[-0.1972,  0.1972],
           [-0.1898,  0.1898]]], device='cuda:0', grad_fn=&lt;ViewBackward0&gt;)),
 (tensor([[[ 0.1972,  0.1898],
           [-0.1972, -0.1898]],
  
          [[-0.1972, -0.1898],
           [ 0.1972,  0.1898]]], device='cuda:0', grad_fn=&lt;ViewBackward0&gt;),
  tensor([[ 0.2431, -0.2431],
          [-0.2431,  0.2431]], device='cuda:0', grad_fn=&lt;ViewBackward0&gt;)))
</code></pre>
<p>How can I calculate the trace of Hessian in this case? <code>torch.trace</code> didn’t work for me any ideas or tips?</p>
On backpropogating multiple losses

username: Xiaohao_Lin
cooked: <p>Dear all,</p>
<p>My data is shaped as (9, n_features, n_timesteps), and I would like to slice it into 9 parts, and feed each part into a simple CNN.</p>
<p>For normal usage,<br>
-Does it make a difference if I backpropagate 9 losses separately to each CNN, versus summing up into 1 loss and backprop?</p>
<p>For negative correlation learning:<br>
-<a>Negative</a> correlation learning encourages diversity among each network.  Would it make a difference if I backprop 9 losses vs just 1 loss?</p>
<p>Thank you!</p>
<p>Best</p>

username: srishti-git1110
cooked: <p>Hi,<br>
Theoretically, the gradients calculated using a summed loss = sum of gradients from the separate terms (the terms that go into the summed loss).</p>
<p>In pytorch too, the gradients are accumulated over several backward calls unless you call <code>zero_grad</code> so the result should be (theoretically) same, though you might see some differences due to precision etc.<br>
The difference shouldn’t be large enough according to me.</p>
<p>Backpropagating 9 losses vs 1 loss -<br>
I assumed you are essentially wanting to know which one between 9 optimizer steps (one after each backward call) and 1 optimizer step (after a single backward call) is better.</p>
<p>This depends on the task at hand. Generally, we prefer several optimizer steps to enable faster training.<br>
Hence, this isn’t a PyTorch specific question. You might want to read up more on negative correlation learning to get a better idea.</p>
<p>Feel free to post further queries.<br>
Best,<br>
S</p>
How to handle multiple inputs and outputs?

username: zero1
cooked: <p>What is the best practice for adding multiple inputs and outputs for <code>torch::jit::script::Module</code> ?</p>
<p>I use below method:</p>
<pre><code class="lang-auto">// model
torch::jit::script::Module net_;

// input
std::vector&lt;torch::jit::IValue&gt; torch_inputs;
torch::Tensor a,b,c;
torch_inputs.push_back(a);
torch_inputs.push_back(b);
torch_inputs.push_back(c);

// 
std::vector&lt;at::Tensor&gt; torch_outputs = 
   net_.forward(torch_inputs).toTensorVector();

// output
torch::Tensor o1 = torch_outputs[0];
torch::Tensor o2 = torch_outputs[1];
torch::Tensor o3 = torch_outputs[2];
</code></pre>
<p>My question is whether it is the best practice to do this, and how to be compatible if the input situation is not like this？</p>
PyTorch Last & Best Models have the exact weights!

username: Wael_AbuRezeq
cooked: <p>Hi everyone,</p>
<p>I am building a PyTorch training function where I am intending to save the best model and last model. However, both of the models result in the exact weight although I am addining a condition of validation loss. Below is my code:</p>
<blockquote>
<p>model = NeuralNet(in_dimension=2, out_dimension=1)<br>
num_epoch = 500<br>
optimizer = nn.optim.Adam(…)<br>
criterion = nn.torch.BCELoss(…)<br>
train_loader= DataLoader(…)<br>
val_loader = DataLoader(…)</p>
<p>def a_training(num_epoch, model, optimizer, criterion, train_loader, val_loader):<br>
best_model = NeuralNet(in_dimension=2, out_dimension=1) <span class="hashtag-raw">#initializing</span> best model<br>
best_val_loss = None<br>
for epoch in range(num_epoch):<br>
train_loss = train_epoch(epoch, optimizer, criterion, model, train_loader)<br>
val_loss = validate_epoch(epoch, criterion, model, val_loader)<br>
if epoch ==0:<br>
best_val_loss = val_loss<br>
torch.save(model, ‘/content/_first_model.pt’)<br>
else:<br>
if val_loss &lt; best_val_loss:<br>
best_val_loss = val_loss<br>
best_model = model<br>
torch.save(best_model, ‘/content/_best_model.pt’)<br>
torch.save(model, ‘/content/_last_model.pt’)<br>
return None</p>
</blockquote>
<p>then I am running the function a_training(…)<br>
and later I am loading the models, checking their weights but the last and best are the same !</p>
<p><span class="hashtag-raw">#Verifying</span> that last and best models have different weights in the final layer</p>
<blockquote>
<p>_first_model = torch.load(‘/content/_first_model.pt’)<br>
_best_model = torch.load(‘/content/_best_model.pt’)<br>
_last_model = torch.load(‘/content/_last_model.pt’)</p>
<p>print(“First model”,torch.sum(syn1_first_model.fc2.weight.data))<br>
print(“Best model”,torch.sum(syn1_best_model.fc2.weight.data))<br>
print(“last model”,torch.sum(syn1_last_model.fc2.weight.data))</p>
</blockquote>
<p>and I am getting <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<blockquote>
<p>First model tensor(-0.4032)<br>
Best  model tensor(-4.8789)<br>
last  model tensor(-4.8789)</p>
</blockquote>

username: ptrblck
cooked: <p>Couldn’t the last model also be the best model or did you verify that the condition is not met in the last training iteration?<br>
You might also want to calculate the <code>abs().max()</code> error between some parameters as a pure visual comparison might not be sufficient and lower decimals might have been changed.</p>

username: Wael_AbuRezeq
cooked: <p>I tried different epochs, from 100 to 500 and they always result in the same fc2 layer weight. Yes, you are right as I need to verify that the condition is working properly.</p>
Developer Documentation - wiki - how to get access to add a new page?

username: aksj
cooked: <p>Hi! I wanted to add to the Pytorch developer facing documentation wiki (<a href="https://github.com/pytorch/pytorch/wiki" class="inline-onebox" rel="noopener nofollow ugc">Home · pytorch/pytorch Wiki · GitHub</a>) - specifically, I wanted to add more information about what the specific sub-files and sub-folders purpose is as an overview page. How can I get access to create a page on the wiki to do so?</p>
<p>Thank you!</p>

username: ptrblck
cooked: <p>Based on <a href="https://github.com/pytorch/pytorch/wiki/Where-or-how-should-I-add-documentation#adding-a-new-page-to-the-developer-wiki">this</a> and <a href="https://github.com/pytorch/pytorch/wiki/Where-or-how-should-I-add-documentation#a-page-looks-out-of-date-what-do-i-do">this</a> entry I would suggest to start with a GitHub issue describing your proposal and discuss it there first.</p>
Input range for TorchVision with Batteries Included?

username: carbocation
cooked: <p>In the <a href="https://github.com/pytorch/vision/issues/3911" rel="noopener nofollow ugc">TorchVision with Batteries Included</a> effort, I didn’t notice the <a href="https://pytorch.org/blog/how-to-train-state-of-the-art-models-using-torchvision-latest-primitives/" rel="noopener nofollow ugc">blog post</a> or the <a href="https://pytorch.org/vision/main/models/convnext.html" rel="noopener nofollow ugc">torchvision ConvNext documentation</a> indicating the expected input pixel value range before or after transform.</p>
<p>Are the pixels expected to be in the 0-1 range? 0-255 range? Something else?</p>

username: ptrblck
cooked: <p>The <code>transforms</code> for <code>ConvNext</code> all reuse <code>ImageClassification</code> as seen <a href="https://github.com/pytorch/vision/blob/35f68a09f94b2d7afb3f6adc2ba850216413f28e/torchvision/models/convnext.py#L209-L286">here</a> which accepts a <code>PIL.Image</code> and will scale it to <code>[0, 1]</code> first and then normalize it as described <a href="https://github.com/pytorch/vision/blob/35f68a09f94b2d7afb3f6adc2ba850216413f28e/torchvision/transforms/_presets.py#L38-L80">here</a> so I assume the input can be a pure <code>uint8</code> <code>PIL.Image</code> in <code>[0, 255]</code> assuming you are using the predefined transformation.</p>

username: carbocation
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="169781">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>here</p>
</blockquote>
</aside>
<p>Fantastic, thank you. I had seen your first link, but didn’t mentally parse the <code>partial</code> call over to ImageClassification. Much appreciated!</p>

username: ptrblck
cooked: <p>Let me know if this works of of you are seeing unexpected results as I’ve just checked the source code without a verification.</p>

username: carbocation
cooked: <p>My use-case is applying the ConvNext encoder for segmentation. I have been providing input in the [-1, 1] range and it works quite well, but now that you pointed me to the normalization constants I’ll try the usual <code>scale to [0, 1] and then normalize</code> approach and let you know if I get better results.</p>

username: carbocation
cooked: <p>It will take quite awhile before I can comment about the final metrics, but the training and validation loss are now decreasing faster than they were before. So it seems that identifying the right normalization range has been useful. Thanks again.</p>
Multiply a 3d and a 2d tensor

username: ranvirsv
cooked: <p>I have a 3d tensor A = [5, 12, 12] and a 2d tensor B = [5, 12]</p>
<p>So basically 5 2d tensors and 5 1d tensors</p>
<p>And I want to multiply them to get a 2d tensor C = [5, 12]<br>
i.e. I want to multiply every 1d tensor with every 2d tensor to get a 1d tensor and then stack them to get a 2d tensor.</p>
<p>I did something that I thought would work, but I am not getting the desired result.</p>
<pre><code class="lang-auto">def mul(A: torch.tensor, B: torch.tensor) -&gt; torch.tensor:
    t_res = []
    for i in range(len(A):
        t_res.append(B[i].matmul(Ai]))
    
    return torch.stack(tuple(t_res), dim = 0)
</code></pre>
<p>Is there a better/easier way to do this?</p>

username: ptrblck
cooked: <p><code>unsqueze</code>ing <code>B</code> should work:</p>
<pre><code class="lang-python">A = torch.randn(5, 12, 12)
B = torch.randn(5, 12)

t_res = []
for i in range(len(A)):
    t_res.append(B[i].matmul(A[i]))
ref = torch.stack(tuple(t_res), dim = 0)

out = torch.matmul(B.unsqueeze(1), A)

print((out.squeeze(1) - ref).abs().max())
# tensor(0.)
</code></pre>
LibTorch (nightly) build error with a file from the submodule iDeep

username: FuzzyAtish
cooked: <p>Hello,</p>
<p>I’m not 100% certain that this is the best place to ask, but I’ve seen a coupe of other similar questions, so I thought why not.</p>
<p>Recently, when attempting to build the C++ libraries on Linux using CMake, I receive the following error:</p>
<pre data-code-wrap="shell"><code class="lang-plaintext">/home/user/Build/pytorch-nightly/third_party/ideep/include/ideep/attributes.hpp:23:10: error: unknown type name 'dnnl_fpmath_mode_t'
  attr_t(dnnl_fpmath_mode_t fpmath_mode,
         ^
/home/user/Build/pytorch-nightly/third_party/ideep/include/ideep/attributes.hpp:29:27: error: unknown type name 'dnnl_fpmath_mode_t'
  attr_t&amp; set_fpmath_mode(dnnl_fpmath_mode_t mode) {
</code></pre>
<p>The strange thing is, that the above mentioned type is in fact declared in another file that’s part of another submodule, more precisely in <code>third_party/ideep/mkl-dnn/third_party/oneDNN/include/oneapi/dnnl/dnnl_types.h</code>.</p>
<p>The reason I’d like to build it myself from source is, because CUDA 12 support as per this issue on GitHub <a href="https://github.com/pytorch/pytorch/issues/91122" rel="noopener nofollow ugc">[CUDA][CUDA 12] CUDA 12 Support Tracking Issue</a> has been somewhat finalized already, and I would like to experiment with that.</p>
<p>If anyone else came across this or happens to know of a solution, please do share.</p>
<p>Thanks!</p>
Pytorch: Editing and re-organising directories

username: pvd
cooked: <p>I am working with a dataset in the format: It is a series of images and their respective object segmentation masks from 15 long videos. I want to re-organise it from:<br>
Folder/<br>
class1/<br>
x/<br>
image1.png<br>
image2.png<br>
…<br>
y/<br>
mask_image1.png<br>
mask_image2.png<br>
…<br>
class2/<br>
x/<br>
image50.png<br>
y/<br>
mask_image50.png<br>
…</p>
<p>to:<br>
Folder/<br>
x/<br>
class1/<br>
image1.png<br>
…<br>
class2/<br>
image50.png<br>
…<br>
y/<br>
class1/<br>
mask_image1.png<br>
…<br>
class2/<br>
mask_image50.png<br>
…</p>
<p>Any help would be great!</p>
Predicting median with LSTM

username: Planet_Boost
cooked: <p>I’m new with deep learning networks and LSTM and I have some trouble. My target values are often skewed and I want to make an LSTM that predicts more extremes more accurately. Please excuse me for me newbiness.</p>
<p>In RStudio, R, there is the ranger() function that is able to predict quantiles (a distribution), rather than simple point prediction. From the quantiles, I take the median and this gets me better predictions for extreme values. I want to do something similar with LSTM, however as I said before, I am new and don’t fully know what I should be doing.</p>
<p>Let’s have a look at the following LSTM:</p>
<pre><code class="lang-auto">class LSTMModel(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super().__init__()
        self.lstm = nn.LSTM(input_size, hidden_size, batch_first=True)
        self.linear = nn.Linear(hidden_size, output_size) 
        
    def forward(self, x):
        x, _ = self.lstm(x)
        x = self.linear(x)
        return x
</code></pre>
<p>and:</p>
<pre><code class="lang-auto">class LSTMModel(nn.Module):
    def __init__(self, input_size, hidden_size, output_size, num_quantiles, num_layers=1):
        super().__init__()
        self.lstm = nn.LSTM(input_size, hidden_size, num_layers=num_layers, batch_first=True)
        self.linear = nn.Linear(hidden_size, num_quantiles * output_size) 
        
    def forward(self, x):
        x, _ = self.lstm(x)
        quantiles = self.linear(x)
        return quantiles
</code></pre>
<p>Does the second model actually predict quantiles, compared to the first model? Does the first model practically already estimate a median prediction?</p>
<p>Say I set <code>num_quantiles = 50</code>. Is the median then the 25th prediction? :</p>
<pre><code class="lang-auto">Y_pred = model(X)
median_pred = Y_pred[:, :, quantile_number // 2::quantile_number]
</code></pre>
<p>Technically, if I set <code>num_quantiles = 1</code> (which to my layman’s eyes makes the model similar to the first model), is then every prediction the median? So, is it similar to <code>median_pred</code> above?</p>
<p>Then regarding the loss function. What is an advised loss function in this case? I have tried:</p>
<pre><code class="lang-auto">def quantile_loss(predicted_values, target_values, quantile):
    median_pred =  predicted_values[:, :, quantile]
    error = median_pred - target_values
    loss = torch.abs(error)
    return loss.mean()
</code></pre>
<p>But if I look at the median (quantile = 0.5), I get what I assume is practically similar to nn.L1Loss()?</p>
Range of ImageNet 2012 training set labels

username: fourleafclover
cooked: <p>I downloaded the ImageNet 2012 training and validation datasets and use them for training a ResNet-50 model. At first, during training a <code>CUDA device-side assert error</code> was triggered and after some web-search, I found its cause: the targets that are provided by my training set dataloader are in the range of [1, 1000], while <code>nn.CrossEntropyLoss</code>, which I use for training, expects the targets to be in [0, 999]. Simply subtracting 1 from all the targets provided by the dataloader of the training set (as <a href="https://discuss.pytorch.org/t/imagenet-labels-incorrect/20246/2">suggested here</a>) solved the problem.</p>
<p>While this is nice, I would like to understand a couple of things that are still mysterious to me:</p>
<ul>
<li>
<em>why</em> does the training set data loader provide targets that are in the incorrect range?</li>
<li>why does only the dataloader for the <em>training set</em> provide targets that are in the incorrect range? For the targets provided by the validation set, subtracting one is not necessary.</li>
</ul>
<p>As far as I understood, the range of the targets is defined by how the <a href="https://pytorch.org/vision/main/_modules/torchvision/datasets/folder.html#DatasetFolder" rel="noopener nofollow ugc">DatasetFolder</a> “builds” the dataset. More specifically, this depends on the <code>make_dataset()</code> function, which in turn calls <code>find_classes()</code> to get a dictionary <code>class_to_idx</code> that maps WordNet IDs (e.g. <code>n01440764</code>) to indices.</p>
<p>When I use <code>find_classes()</code> and pass to it the directory where I unpacked the ImageNet training set, the returned <code>class_to_idx</code> dictionary has values in the range of [0, 999], which seems right.</p>
<p>Does someone know what I am missing?<br>
Thanks in advance</p>

username: ptrblck
cooked: <ol>
<li>
<p>I guess your <code>root</code> folder contains an unneeded and unexpected folder since the class index mapping will be performed based on the number of folders. <code>torchvision.datasets.ImageNet</code> will initialize its base class <a href="https://github.com/pytorch/vision/blob/1353df9e45aae6739c23db4a053c3d9a9fd17b3d/torchvision/datasets/imagenet.py#L49">here</a> which is <code>ImageFolder</code> and which will eventually call into <code>find_classes</code> <a href="https://github.com/pytorch/vision/blob/1353df9e45aae6739c23db4a053c3d9a9fd17b3d/torchvision/datasets/folder.py#L35-L45">here</a> enumerating the folders starting at class index 0.</p>
</li>
<li>
<p>Same as above: check the <code>root</code> folder and make sure it contains 1000 subfolders.</p>
</li>
</ol>

username: fourleafclover
cooked: <p>This very much makes sense, thanks a lot for your swift reply!</p>
Can't use `torch.nn.functional.dropout1d` of pytorch

username: crownz
cooked: <p>I want to use torch.nn.functional.dropout1d for my code. But when I try to use it, python prompts me:</p>
<pre><code class="lang-auto">AttributeError: module 'torch.nn.functional' has no attribute 'dropout1d'. 
</code></pre>
<p>Then, I go to the source code of functional.py. And I find there is no dropout1d function in the script. However, PyTorch has officially provided the source code of dropout1d. The link is <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/functional.py" rel="noopener nofollow ugc">pytorch/functional.py at master · pytorch/pytorch (github.com)</a> and <a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/dropout.html#Dropout1d" rel="noopener nofollow ugc">torch.nn.modules.dropout — PyTorch 1.13 documentation</a>. From the webpage, dropout1d is introduced in torch-1.12.</p>
<p>My torch is installed using <code>pip install torch</code> on ubuntu and the version is ‘1.10.1+cu102’. According to the <a href="https://pytorch.org/" rel="noopener nofollow ugc">PyTorch</a>’s installation instructions of Linux, the Stable (1.13.1) version should be installed. However, the source code on my computer is different from the official source code.</p>
<p>Now, I think the the question becomes how to install the correct version of torch. But when I want to install the specified version of torch with command-<code>pip install --upgrade torch==1.12</code>, pip  prompts me:</p>
<pre><code class="lang-auto">ERROR: Could not find a version that satisfies the requirement torch==1.12 (from versions: 1.0.0, 1.0.1, 1.0.1.post2, 1.1.0, 1.2.0, 1.3.0, 1.3.1, 1.4.0, 1.5.0, 1.5.1, 1.6.0, 1.7.0, 1.7.1, 1.8.0, 1.8.1, 1.9.0, 1.9.1, 1.10.0, 1.10.1, 1.10.2)
ERROR: No matching distribution found for torch==1.12
</code></pre>
<p>Can anyone help me?</p>

username: ptrblck
cooked: <p>I would recommend to either uninstall all PyTorch binaries in your current environment and then use the install instructions or to create a new and empty virtual environment and install the latest PyTorch release there.</p>

username: crownz
cooked: <p>Okay, I will try it now.</p>

username: crownz
cooked: <p>It works, thank you!</p>
CUDA not available on aarch64

username: ElkeAusBerlin
cooked: <p>Hey,<br>
just to verify. the aarch64 release are not compiled with cuda enabled? Is this true?</p>
<p>The installation:<br>
<code>python3 -m pip install torch --extra-index-url https://download.pytorch.org/whl/cu102</code></p>
<p>The output:</p>
<pre><code class="lang-auto">root@localhost:~# python3 -m torch.utils.collect_env
Collecting environment information...
PyTorch version: 1.10.2
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (aarch64)
GCC version: Could not collect
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.25

Python version: 3.6.9 (default, Mar 15 2022, 13:55:28)  [GCC 8.4.0] (64-bit runtime)
Python platform: Linux-4.9.140-aarch64-with-Ubuntu-18.04-bionic
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.5
[pip3] torch==1.10.2
[conda] Could not collect
</code></pre>

username: ptrblck
cooked: <p>Yes, I think that’s the case as the aarch64 builds might have been created for Mac.<br>
You could use the <a href="https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch">NGC containers</a> which provide ARM support.</p>
Modifying forward/backward pass

username: yoelshoshan
cooked: <p>Is it possible to do the following (in pytorch 1.x or 2) during training to modify what happens before/after each of the following:</p>
<ol>
<li>
<p>During the forward pass, after the activations are store (to be later used in the backward pass), to do operations on them (for example, like transfering them to another device and other things)</p>
</li>
<li>
<p>During the backward pass, before every layer gradients calculations to do some operations on the activations (for example, like transfering them to another device and other things)</p>
</li>
</ol>
<p>Any information will be highly appreciated <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>[EDIT: I’m exploring usage of functions like register_module_full_backward_pre_hook]</p>

username: ptrblck
cooked: <p>Your use case sounds similar to CPU offloading, which uses <a href="https://pytorch.org/docs/master/autograd.html#torch.autograd.graph.saved_tensors_hooks"><code>torch.autograd.graph.saved_tensors_hooks</code></a> or <a href="https://pytorch.org/docs/master/autograd.html#torch.autograd.graph.save_on_cpu"><code> torch.autograd.graph.save_on_cpu</code></a> if I’m not mistaken, so you could take a look at these context managers.</p>

username: yoelshoshan
cooked: <p>Thanks a lot!! Sounds like exactly what I need</p>
Is there an easy way to compute mean of every two tensors in a 2D tensor?

username: pytorch_user1
cooked: <p>Suppose I have a tensor that looks like this:</p>
<pre><code class="lang-auto">a = torch.tensor([
                  [1. 2.],
                  [3. 4.],
                  [5. 6.],
                  [7. 8.]
                 ])
</code></pre>
<p>Is there a way to compute the mean of every two row tensors (without overlap) in <code>a</code> without looping?</p>
<p>The expected output tensor would be:</p>
<pre><code class="lang-auto">torch.tensor([
              [2. 3.], # mean of 1st and 2nd row tensors in `a`
              [6. 7.]  # mean of 3rd and 4th row tensors in `a`
            ])
</code></pre>

username: ptrblck
cooked: <p>Something like this would work:</p>
<pre><code class="lang-python">b = torch.stack([a_.mean(0) for a_ in a.split(2, 0)], 0)
print(b)
# tensor([[2., 3.],
#         [6., 7.]])
</code></pre>
<p>or using <code>scatter_reduce_</code>:</p>
<pre><code class="lang-python">torch.zeros(2, 2).scatter_reduce_(0, torch.tensor([[0, 0], [0, 0], [1, 1], [1, 1]]), a, reduce="mean", include_self=False)
</code></pre>
<p>or <code>index_put_</code> with <code>accumulate=True</code> and a scaling:</p>
<pre><code class="lang-python">c = torch.zeros(2, 2)
c.index_put_((torch.tensor([0, 0, 1, 1]), ), a, accumulate=True)
c /= 2
</code></pre>
<p>Depending on your actual use case, how the indices are calculated, how large the tensor is etc. one approach might perform better than others.</p>