akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Trainable scalar every n batches

L-Reichardt: <p>Dear Forum,</p>
<p>Is it possible to create a scalar that is only trained during n-amount of batches? I would create my scalar alpha (init. at 1) through:</p>
<p><code>alpha = nn.Parameter(torch.ones(1, requires_grad=True))</code></p>
<p>Am I correct to assume that the value of alpha then changes during training, e.g. becomes 0.9? Or did I set this up incorrectly.</p>
<p>How can I avoid alpha being trained during <code>.step()</code>, and only train it every nth Batch? Intuitively I would <code>model.alpha.requires_grad = False</code> during other batches. Then, every nth Batch.</p>
<pre><code class="lang-auto">for param in model.parameters():
    param.requires_grad = False
model.alpha.required_grad = True
</code></pre>
<p>Thanks in advance for your help.</p>

KFrank: <p>Hi L!</p>
<aside class="quote no-group" data-username="L-Reichardt" data-post="1" data-topic="142513" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/59ef9b/48.png" class="avatar"> L-Reichardt:</div>
<blockquote>
<p>How can I avoid alpha being trained during <code>.step()</code>, and only train it every nth Batch?</p>
</blockquote>
</aside>
<p>There are a number of ways to do this.  I would probably use two<br>
separate optimizers:</p>
<pre data-code-wrap="python"><code class="lang-python">optModel = torch.optim.SGD (myModel.parameters(), lr = 0.1)
optAlpha = torch.optim.SGD ([alpha], lr = 0.1)
</code></pre>
<p>(This assumes that <code>alpha</code> is not one of <code>myModel</code>’s parameters.)</p>
<p>Now you can call <code>optModel.step()</code> for every batch and only call<br>
<code>optAlpha</code> after every <code>n</code> batches.</p>
<p>Note, that every call to <code>loss.backward()</code> will <em>accumulate</em> loss’s<br>
gradient with respect to <code>alpha</code> into <code>alpha.grad</code>.  If you don’t want<br>
this you might do something like:</p>
<pre data-code-wrap="python"><code class="lang-python">optAlpha.zero_grad()
# calculate loss for just one batch ...
loss.backward()   # alpha.grad is now just from one batch
optAlpha.step()
</code></pre>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">for param in model.parameters():
    param.requires_grad = False
model.alpha.required_grad = True
</code></pre>
</blockquote>
</aside>
<p>You can make such a scheme work, but note that if your optimizer is<br>
using momentum or weight decay, having a parameter’s <code>grad</code> be zero<br>
will not prevent that parameter’s value being changed by an <code>opt.step()</code><br>
update.</p>
<p>Best.</p>
<p>K. Frank</p>

L-Reichardt: <p>Thank you for your detailed reply. I think this solves my problem. It didn’t occur for me to think about the impact of weight decay. I am an AdamW fan, so I will have to consider this.</p>
Returning early if tensor contains a value above a certain threshold in a jitted function

nicoco: <p>Hi all,</p>
<p>I guess I could have titled my topic: “Can torch.jit be used for the same purpose as numba’s njit or is it something strictly used to optimize models for inference?”, but this sounded a bit too long.</p>
<p>I am trying to understand why this torch jitted function is so very slow. I think this code snippet is better than any attempt to explain it:</p>
<pre><code class="lang-auto">import timeit
import torch


def if_max_torch_base(tensor: torch.Tensor, threshold=torch.Tensor) -&gt; torch.Tensor:
    "Just the base pytorch way"
    return torch.any(tensor &gt; threshold)


@torch.jit.script
def if_max_torch_jit(tensor: torch.Tensor, threshold: torch.Tensor) -&gt; torch.Tensor:
    "Attempt at exploiting torch.jit"
    for x in tensor:
        if x &gt; threshold:
            return torch.tensor(True)  # no need to test further, let's just return early
    return torch.tensor(False)

if_max_torch_jit = torch.jit.trace(if_max_torch_jit, (torch.rand(5), torch.tensor(0.5)))


def main():
    tensor = torch.linspace(0, 1, 1_000_000, device="cuda")

    for name, func in (
        ["base", if_max_torch_base],
        ["jit", if_max_torch_jit],
    ):
        for threshold in (0.5, 1.0):
            print(name, threshold)
            t = torch.tensor(threshold, device="cuda")
            timer = timeit.Timer(lambda: func(tensor, t))
            print(timer.repeat(repeat=3, number=100))


main()
</code></pre>
<p>And this is the output:</p>
<pre><code class="lang-nohighlight">base 0.5
[0.0002641710452735424, 0.00016550999134778976]
base 1.0
[0.00017875991761684418, 0.00016467086970806122]
jit 0.5
[70.17099338211119, 71.27563373814337]
jit 1.0
[139.18801530217752, 139.25591901200823]
</code></pre>
<p>Why does this jitted function have abysmal performance? Am I just completely misusing torch.jit? Can torch.jit can event be used for this purpose?</p>
<p>This is what <code>if_max_torch_jit.code</code> returns:</p>
<pre><code class="lang-auto">def if_max_torch_jit(tensor: Tensor,
    threshold: Tensor) -&gt; Tensor:
  _0 = uninitialized(Tensor)
  _1 = torch.len(tensor)
  _2 = False
  _3 = _0
  _4 = 0
  _5 = torch.gt(_1, 0)
  while _5:
    x = torch.select(tensor, 0, _4)
    if bool(torch.gt(x, threshold)):
      _6, _7, _8 = False, True, torch.tensor(True)
    else:
      _6, _7, _8 = True, False, _0
    _9 = torch.add(_4, 1)
    _5, _2, _3, _4 = torch.__and__(torch.lt(_9, _1), _6), _7, _8, _9
  if _2:
    _10 = _3
  else:
    _10 = torch.tensor(False)
  return _10
</code></pre>
<p>This seems overly convoluted, but I am not familiar with low-level programming at all…</p>
<p>Thanks for reading me and thanks in advance for educating me. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Dense layer tensor shape question

White_Sirilo: <p>Greetings!</p>
<p>I’m am trying to implement an actor-critic algorithm (more specifically ddpg) and I was wondering if I feed the data the right way. I’ve seen previously that dense layers often expect <strong>B x C x F shapes</strong> where B is batch size, C is Channel size and F is feature size although in my scenario I’m having just B x F and it seems to work (!?) Is there something I’m missing? I’m providing 2 screenshots of the critic forward function and the output where I’m printing various shapes.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/f/0fe55eb57d3a5e4e51258d8e79deb718decfc4c3.png" data-download-href="https://discuss.pytorch.org/uploads/default/0fe55eb57d3a5e4e51258d8e79deb718decfc4c3" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/f/0fe55eb57d3a5e4e51258d8e79deb718decfc4c3_2_690x363.png" alt="image" data-base62-sha1="2gCAgrtq7iX2pE34GJ6VF5RSQtJ" width="690" height="363" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/f/0fe55eb57d3a5e4e51258d8e79deb718decfc4c3_2_690x363.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/f/0fe55eb57d3a5e4e51258d8e79deb718decfc4c3_2_1035x544.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/f/0fe55eb57d3a5e4e51258d8e79deb718decfc4c3.png 2x" data-dominant-color="2C2D26"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1139×600 65.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/b/db89b7e60248f0bd41ea755c42c3ebf540ffdba1.png" alt="image" data-base62-sha1="vk7CPgss0B7tK7rHsDKrBrlQFgd" width="599" height="240"></p>
Error when using torch.multiprocessing

Gwon: <p>Hi,<br>
I have a problem with use torch.multiprocessing modules.</p>
<p>In my code,</p>
<pre><code class="lang-auto">import torch.multiprocessing as mp

class MultiProcessing:
     def run_inference(self):
          # Load detection model
          model = MyModel()
          ...
          processes = []
          for process_idx in range(2):
               p = mp.Process(target=self.do_detection, args=(model, ))
               p.start()  # Error here
               processes.append(p)
          for p in processes:
               p.join()

     def do_detection(self, model):
          vid_cap = cv2.VideoCapture("&lt;VideoFile&gt;")
          while True:
               # Read video
               _, img = vid_cap.read()
               # Input video frame to detection model
               pred = model(img)
               # Visualize detection result
               ...
if __name__ == '__main__':
     inference = MultiProcessing()
     inference.run_inference()
</code></pre>
<p>Error like below</p>
<pre><code class="lang-auto">THCudaCheck FAIL file=..\torch/csrc/generic/StorageSharing.cpp line=249 error=801 : operation not supported
Traceback (most recent call last):
  File "D:\anaconda3\envs\conda\lib\multiprocessing\popen_spawn_win32.py", line 89, in __init__
    reduction.dump(process_obj, to_child)
  File "D:\anaconda3\envs\conda\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "D:\anaconda3\envs\conda\lib\site-packages\torch\multiprocessing\reductions.py", line 247, in reduce_tensor
    event_sync_required) = storage._share_cuda_()
RuntimeError: cuda runtime error (801) : operation not supported at ..\torch/csrc/generic/StorageSharing.cpp:249
Traceback (most recent call last):
  File "D:\anaconda3\envs\conda\lib\multiprocessing\spawn.py", line 115, in _main
    self = reduction.pickle.load(from_parent)
EOFError: Ran out of input
</code></pre>
<p>To solve this problem, I search many solutions.<br>
Most solutions say set the <code>num_worker=0</code> in dataloader, but i’m not using dataloader.</p>
<p>How can i solve this problem?</p>
train_loss.backward(retain_graph=True) AttributeError: 'float' object has no attribute 'backward'

Mona_Jalal: <p>I am doing binary classification with a batch size of 8 and the output of my model is:</p>
<pre><code class="lang-auto">out shape is:  torch.Size([8, 2])
out is:  tensor([[-1.2086,  0.8501],
        [-0.8705, -0.2427],
        [-1.0448,  0.5410],
        [-1.2318,  0.2670],
        [-1.0920,  0.0126],
        [-0.9867,  0.1056],
        [-1.2848,  0.9758],
        [-1.2415, -0.2882]], device='cuda:0', grad_fn=&lt;AddmmBackward0&gt;)
</code></pre>
<p>and I am getting this error:</p>
<pre><code class="lang-auto">train loss:  0.0
Traceback (most recent call last):
  File "main_classifier.py", line 253, in &lt;module&gt;
    train_loss.backward(retain_graph=True) 
AttributeError: 'float' object has no attribute 'backward'
</code></pre>
<p>How can I solve this error?</p>

ptrblck: <p>Your <code>train_loss</code> seems to be a plain <code>float</code> (i.e. not a <code>tensor</code>) so make sure you are calling <code>backward</code> on the actual loss tensor not the Python <code>float</code> scalar.</p>
TypeError: unhashable type: 'numpy.ndarray' for nested lists

danielmanu93: <p>Hi All, I have 2 numpy arrays <code>n</code> and <code>e</code> which I want to iterate through but I get <code>unhashable type: 'numpy.ndarray'</code>. The shape of <code>n</code> is (112, 55, 55) and <code>e</code> is (112, 55).</p>
<p>Below are how both arrays look like:</p>
<p><code>n</code> array<br>
[[[4 1 0 … 0 1 0]<br>
[2 0 2 … 4 1 1]<br>
[1 2 1 … 3 0 0]<br>
…<br>
[0 4 4 … 3 0 1]<br>
[3 1 0 … 2 2 4]<br>
[3 1 1 … 0 4 3]]</p>
<p>[[2 4 0 … 1 2 4]<br>
[0 0 0 … 0 0 0]<br>
[4 0 0 … 0 2 3]<br>
…<br>
[0 0 0 … 1 4 0]<br>
[0 2 2 … 4 2 4]<br>
[4 2 3 … 2 0 0]]</p>
<p>[[3 0 1 … 3 2 4]<br>
[3 0 2 … 3 2 3]<br>
[3 2 0 … 2 1 1]<br>
…<br>
[3 0 4 … 1 2 4]<br>
[2 2 1 … 2 1 0]<br>
[4 0 2 … 3 0 4]]</p>
<p>…</p>
<p>[[1 3 0 … 3 3 2]<br>
[3 0 4 … 3 0 0]<br>
[4 0 2 … 4 2 4]<br>
…<br>
[0 1 4 … 3 0 1]<br>
[3 2 2 … 2 0 0]<br>
[2 0 3 … 1 0 4]]</p>
<p>[[4 3 2 … 0 1 1]<br>
[1 1 4 … 0 0 4]<br>
[0 2 2 … 1 3 3]<br>
…<br>
[1 0 1 … 3 2 3]<br>
[4 1 4 … 2 4 2]<br>
[1 0 3 … 1 0 3]]</p>
<p>[[4 0 2 … 4 0 0]<br>
[4 0 0 … 1 0 0]<br>
[2 0 2 … 1 2 3]<br>
…<br>
[4 1 0 … 1 0 0]<br>
[0 0 2 … 2 0 1]<br>
[3 3 1 … 1 4 0]]]</p>
<p><code>e</code> array<br>
[[3 0 5 … 5 5 1]<br>
[4 8 6 … 4 8 7]<br>
[5 8 0 … 3 6 0]<br>
…<br>
[0 8 0 … 6 0 8]<br>
[3 9 0 … 0 8 1]<br>
[7 3 2 … 2 3 2]]</p>

ptrblck: <p>The issue doesn’t seem to be PyTorch related so you might get a better and faster answer in e.g. StackOverflow.<br>
In any case, if I understand your use case correctly, you would like to iterate he numpy arrays, which works fine:</p>
<pre><code class="lang-python">n = np.random.randn(112, 55, 55) 
e = np.random.randn(112, 55)

for n_ in n:
    print(n_)
    
for e_ in e:
    print(e_)
</code></pre>

danielmanu93: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, I am using the line below to iterate through but getting unhashable type error:</p>
<p><code>m1 = [data.matrices2mol(n_, e_, strict=True) for n_, e_ in zip(n, e)]</code>. Do you happen to know how to solve this?</p>

ptrblck: <p>I guess the error is raised in <code>data.matrices2mol</code> (which I don’t know how it’s implemented), as a plain list comprehension works:</p>
<pre><code class="lang-python">m1 = [(n_, e_) for n_, e_ in zip(n, e)]
</code></pre>
RuntimeError: One of the variables needed for gradient computation has been modified by an inplace, operation

Medbob: <p>Hello everyone, I encountered the Error previously mentioned while working on a neural network project with Pytorch, here is my Module class:</p>
<pre><code class="lang-auto">class MyNeural(torch.nn.Module):
    def __init__(self , columns):
        super(MyNeural, self).__init__()
        # input/hidden/output layers
        # Nous devons préciser le nombre des couches cachées que nous allons utiliser
        self.f1 = torch.nn.Linear(columns, 32)
        self.f2 = torch.nn.Linear(32, 16)
        self.f3 = torch.nn.Linear(16, 1)
    def forward(self, x):
        x = self.f1(x)
        x = Functional.relu(x)
        x = self.f2(x)
        x = Functional.relu(x)
        x = self.f3(x)
        x = Functional.relu(x)
        return x
</code></pre>
<p>And here is my training method:</p>
<pre><code class="lang-auto">def training(self) :
        self.net = MyNeural(self.data.shape[1])
        self.entropyloss = torch.nn.BCELoss()
        self.optim = torch.optim.Adam(self.net.parameters(), lr=0.001)
        self.epochs=5
        for i in range(self.epochs): 
            self.net.train(mode=True)
            train_loss = 0.0
            loss = 0
            j = 0
            for  data, targets in self.train_DL:
                cprint(f"iteration : {i}" , 'blue')
                pred_targets = self.net(data)
                print("pred_targ: ",pred_targets)
                loss += self.entropyloss(pred_targets, targets)
                self.optim.zero_grad()
                loss.backward(retain_graph=True)
                self.optim.step()
                train_loss += loss.item()
</code></pre>

ptrblck: <p>You are accumulating the <code>loss</code> via <code>loss += self.entropyloss(pred_targets, targets)</code> in combination with <code>retain_graph=True</code>, which will keep all computation graphs alive. Changing the parameters inplace via <code>self.optim.step()</code> would create stale forward activations as described in <a href="https://discuss.pytorch.org/t/struggling-with-a-runtime-error-related-to-in-place-operations/102222/6">this post</a>.<br>
Could you explain your use case a bit more and especially why you are retaining the graph?</p>
Combining two pretrained models for Training

deadwalker7539: <p>I am trying to use resnet18 and densenet121 as the pretrained models and have added 1 FC layer at the end of each network two change dimensions to 512 of both network and then have concatenated the outputs and passed to two final FC layers.</p>
<p>this can be seen here:</p>
<pre><code class="lang-auto">class classifier(nn.Module):
  def __init__(self,num_classes):
    super(classifier,self).__init__()

    self.resnet=models.resnet18(pretrained=True)

    self.rfc1=nn.Linear(512,512)
    
    self.densenet=models.densenet121(pretrained=True)
    self.dfc1=nn.Linear(1024,512)

    self.final_fc1=nn.Linear(1024,512)
    self.final_fc2=nn.Linear(512,num_classes)
    self.dropout=nn.Dropout(0.2)

  def forward(self,x):
    y=x.detach().clone()

    x=self.resnet.conv1(x)
    x=self.resnet.bn1(x)
    x=self.resnet.relu(x)
    x=self.resnet.maxpool(x)
    x=self.resnet.layer1(x)
    x=self.resnet.layer2(x)
    x=self.resnet.layer3(x)
    x=self.resnet.layer4(x)
    x=self.resnet.avgpool(x)
    x=x.view(x.size(0),-1)
    x=nn.functional.relu(self.rfc1(x))

    y=self.densenet.features(y)
    y=y.view(y.size(0),-1)
    y=nn.functional.relu(self.dfc1(y))

    x=torch.cat((x,y),0)
    x=nn.functional.relu(self.final_fc1(x))
    x=self.dropout(x)
    x=self.final_fc2(x)

    return x
</code></pre>
<p>Error:</p>
<pre><code class="lang-auto">size mismatch, m1: [1048576 x 1], m2: [1024 x 512] at /pytorch/aten/src/THC/generic/THCTensorMathBlas.cu:283
</code></pre>
<p>but as i could see the densenet outputs 1024 feature to the last layer</p>
<p>Questions:</p>
<ol>
<li>Is the implementation correct ?</li>
<li>Can this implementation work as a Ensemble</li>
</ol>

ptrblck: <p>Your current implementation misses the <code>relu</code> as well as adaptive pooling layer from the <code>DenseNet</code> implementation as used <a href="https://github.com/pytorch/vision/blob/03b1d38ba3c67703e648fb067570eeb1a1e61265/torchvision/models/densenet.py#L193-L194">here</a>, which would create a tensor of <code>[batch_size, 1024, 7, 7]</code> for the output of <code>self.densenet.features</code>.<br>
Also, you might want to concatenate the activation tensors in <code>dim1</code> (feature dimension).</p>
<p>These changes should work:</p>
<pre><code class="lang-python">...
    y = self.densenet.features(y)
    y = F.relu(y)
    y = F.adaptive_avg_pool2d(y, (1, 1))
    y = y.view(y.size(0), -1)
    # y = F.relu(self.dfc1(y)) remove this here, if you are using the first relu

    x = torch.cat((x,y), 1)
...
</code></pre>

deadwalker7539: <p>Thanks, This Worked.</p>
Updating loss function largely drags down the speed

Grayson3455: <p>Hi,</p>
<p>I am implementing a sequence to one LSTM model. At the training step, I try to build the loss function based on more predictions, i.e. we use the first prediction to predict the next, and use the new prediction for the next and so on. For each time applying the model, we have a loss added.</p>
<pre><code class="lang-auto">loss = 0
for i in range(loss_step+1):   # loss_step=0 means the common loss function
     backup = Train_batch[:,1:,:]   # backup the second to the final items of the sequence batch
     optimizer.zero_grad()
     out = model(Train_batch)      # apply the sequence to one model to get a single output
     loss = loss + criterion(out,Truth_batch[:,i,:]) # update the loss function based on the new prediction and the truth
     # use the current prediction to update the current batch
     Train_batch[:,:-1,:] = backup.clone() # "shift" the batch upward for one item
     Train_batch[:,-1,:]  = out                    # put the new prediction to the bottom 
loss.backward()
optimizer.step()
</code></pre>
<p>However, it turns out this approach is really slow. Is it reasonable because we need more back-propagation steps? Or is there any trick to speed it up? Thanks for all the help!</p>
Torch.save() error only in release mode

horia1997: <p>Hi everyone, I am using libtorch-1.8.2 with cuda 11.1 and visual studio 2022.<br>
When I try to save a simple tensor it works just fine in debug mode and it fails in release. I think I linked everything ok since it’s the only difference between the two and I can do inference/training on both just fine.</p>
<p>The error always takes me to serialize.h (line 71).<br>
If anyone has any idea/suggestion I would really appreciate it.</p>
<p>Thanks!</p>

horia1997: <p>Nevermind, the issue was a mismatch with debug/release dlls.</p>
Question about multiple losses

deepfailure: <p>I have currently the following network with two losses:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/8/782d193ea6ae6621ec26a1bff0463f40d3697d0a.png" alt="loss" data-base62-sha1="h97V1EvIxogzu2vPMv2xRuzexQK" width="511" height="212"></p>
<p>I have a single optimizer and found that multiplying by a scalar one of the losses leads to better results such that:</p>
<p>L_total = lambda L1 + L2</p>
<p>I have the hypothesis that the lambda hyperparameter should only affect the shared representation. Therefore, I would like to “apply” it after both separated networks backpropagation. My questions are:</p>
<ol>
<li>How to apply the lambda to the loss after each of the single networks’ backpropagation so that only the shared representation is affected by (lambda L1 + L2). Is multiplying the gradients that come from L1 by lambda enough?</li>
<li>Does that make any sense?</li>
</ol>
<p>Thank you.</p>

anantguptadbl: <p><a class="mention" href="/u/deepfailure">@deepfailure</a> Multiplying with a scalar will not affect the overall loss. The only way it could have affected is if you have a backpropagation on the total loss and the lambda scalar is a negative number</p>

deepfailure: <p>Perhaps you misunderstood me.</p>
<p>The scalar is multiplied only on one of the two losses. I think this is a standard practice used in a lot of papers so it should work.</p>
<p>For instance, in the UNREAL paper various losses are used and each loss is multiplied by a different scalar: <a href="https://arxiv.org/pdf/1611.05397.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1611.05397.pdf</a></p>

thecho7: <p>I don’t have any idea applying <code>lambda L1 + L2</code> only to the shared representation.<br>
I think the paper you referred (<a href="https://arxiv.org/pdf/1611.05397.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1611.05397.pdf</a>) does not seem to follow the method you mentioned.</p>
<p>The authors calculated a loss by multiplying lambda to L1, that’s all.<br>
Consequently, L_total = lambda L1 + L2 is enough.</p>
Casting C++ Tensor to Python Tensor

Jesse_9231: <p>Currently, I am trying to have a C++ application to make calls to a Python function, which has a Tensor as input. However, I can not cast the C++ Tensor, such that I can use it as input for the Python function. <a href="https://discuss.pytorch.org/t/pytorch-tensors-from-c-to-python/36918">A previous forum post</a> explains that I could use a function named <code>THPVariable_Wrap</code> for this. However, I tried to import this function, but every time I do so, I get the following error:</p>
<pre><code class="lang-auto">Severity	Code	Description	Project	File	Line	Suppression State
Error	LNK2019	unresolved external symbol "__declspec(dllimport) struct _object * __cdecl THPVariable_Wrap(class at::TensorBase)" (__imp_?THPVariable_Wrap@@YAPEAU_object@@VTensorBase@at@@@Z) referenced in function main	pythonCppTorchExp	C:\Users\MyName\source\repos\pythonCppTorchExp\pythonCppTorchExp\example-app.obj	1	

</code></pre>
<p>I also looked at the linked code in the previous forum post and compared it to the current main branch. I saw that the <a href="https://github.com/pytorch/pytorch/blob/63214b572b6a18f7328687ca69420f0be0eaaaa9/torch/csrc/autograd/python_variable.cpp#L64-L76" rel="noopener nofollow ugc">old version</a> is entirely different from the <a href="https://github.com/pytorch/pytorch/blob/1df4eca6d7ab0885562cafb5b69f1ce35d448396/torch/csrc/autograd/python_variable.cpp#L179-L237" rel="noopener nofollow ugc">current version</a>. Should I still use <code>THPVariable_Wrap</code> to cast my Tensor, and if so, how do I import this function, because I am quite new to C++.</p>
<p>p.s. I also asked the question on StackOverflow, which shows the code I use (<a href="https://stackoverflow.com/questions/70848137/cast-c-pytorch-tensor-to-python-pytorch-tensor" rel="noopener nofollow ugc">link to it</a>)</p>
DDP's trained loss is obviously not as good as DP's

ioyy: <p>Hello everyone, I have a question that has been puzzling me for a long time.<br>
The model I trained using DDP mode is much worse than the model trained by DP mode.<br>
The specific settings are as follows: I keep the total batch_size the same. Each model runs 4 GPUs separately. training both use Adam optimizer. The learning rate is 1e-3.<br>
I don’t know what’s wrong. Can anyone help me?</p>

anantguptadbl: <p><a class="mention" href="/u/ioyy">@ioyy</a> on the onset, the difference between <a href="https://pytorch.org/tutorials/intermediate/dist_tuto.html" rel="noopener nofollow ugc">Data Parallel</a> and <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" rel="noopener nofollow ugc">Distributed Data Parallel</a> is that one uses single process and the other multiple processes. Both implement the gradient synchronization process to assimilate gradients trained in different threads/machines.</p>
<p>Please share your code so that we can look further into it. Please ensure similar seeds in all machines in DDD</p>
Understanding gradient descent in pytorch

Shaswat: <p>Hi,<br>
I have a doubt regarding gradient descent. The below function tires to approximate the value of w and b. After backpropagating the value for w.grad(w is initialized with 1.0) should have been a negative number but I am receiving high positive number.<br>
A negative grad value indicates that w should be increased while a positive number number indicates that w should be reduced.<br>
Am I missing something? Please help me resolve this query. Thank you.</p>
<pre><code class="lang-python">import torch
x = torch.rand([20, 1], requires_grad=True)
y = 3*x - 2
w = torch.tensor([1.], requires_grad = True)
b = torch.tensor([1.], requires_grad = True)

y_hat = w*x + b

loss = torch.sum((y_hat - y)**2)
print(loss) # tensor(79.0468, grad_fn=&lt;SumBackward0&gt;)

loss.backward()
print(w.grad, b.grad) # tensor([35.0812]) tensor([76.0854])
</code></pre>

mMagmer: <p>If you drive mathematic for grad, you can see that nothing is wrong with the result.</p>
<pre><code class="lang-auto">grad_w = (2*(y_hat-y)*(x)).sum()
grad_b = (2*(y_hat-y)*(1)).sum()
</code></pre>
<pre><code class="lang-auto">import torch
x = torch.rand([20, 1], requires_grad=True)
y = 3*x - 2
w = torch.tensor([1.], requires_grad = True)
b = torch.tensor([1.], requires_grad = True)

y_hat = w*x + b

loss = torch.sum((y_hat - y)**2)
print(loss) 

loss.backward()
print(w.grad, b.grad) 

grad_w = (2*(y_hat-y)*(x)).sum()
grad_b = (2*(y_hat-y)*(1)).sum()

print(grad_w,grad_b)
</code></pre>
<blockquote>
<p>output:<br>
tensor(71.8559, grad_fn=)<br>
tensor([37.8953]) tensor([73.1675])<br>
tensor(37.8953, grad_fn=) tensor(73.1675, grad_fn=)</p>
</blockquote>
Error: RuntimeError: CUDA error: device-side assert triggered

maze09: <p>Hi,<br>
after several epochs, I encounter the “RuntimeError: CUDA error: device-side assert triggered” in the interpolation function. I already checked if the there is an error in the indexing but seems everything alright. I appreciate any useful hint.<br>
system:<br>
Versions of relevant libraries:<br>
[pip3] numpy==1.21.5<br>
[pip3] pytorch-lightning==1.5.2<br>
[pip3] torch==1.10.1<br>
[pip3] torch-cluster==1.5.9<br>
[pip3] torch-geometric==2.0.3<br>
[pip3] torch-points-kernels==0.7.0<br>
[pip3] torch-scatter==2.0.9<br>
[pip3] torch-sparse==0.6.12<br>
[pip3] torch-spline-conv==1.2.1<br>
[pip3] torchaudio==0.10.1<br>
[pip3] torchmetrics==0.6.0<br>
[pip3] torchvision==0.11.2</p>
<p>def interpolation(xyz, new_xyz, feat, offset, new_offset, k=3):<br>
“”"<br>
input: xyz: (m, 3), new_xyz: (n, 3), feat: (m, c), offset: (b), new_offset: (b)<br>
output: (n, c)<br>
“”"<br>
assert xyz.is_contiguous() and new_xyz.is_contiguous() and feat.is_contiguous()<br>
idx, dist = knnquery(k, xyz, new_xyz, offset, new_offset) # (n, 3), (n, 3)<br>
dist_recip = 1.0 / (dist + 1e-8) # (n, 3)<br>
norm = torch.sum(dist_recip, dim=1, keepdim=True)<br>
weight = dist_recip / norm # (n, 3)</p>
<pre><code>new_feat = torch.cuda.FloatTensor(new_xyz.shape[0], feat.shape[1]).zero_()
for i in range(k):

    new_feat += feat[idx[:, i].long(), :] * weight[:, i].unsqueeze(-1)
    #print(feat.size(),idx[:, i])
return new_feat</code></pre>

ptrblck: <p>I would guess that you are running into an indexing issue.<br>
Rerun the code via <code>CUDA_LAUNCH_BLOCKING=1 python script.pt args</code> or on the CPU to get a better stacktrace.</p>

maze09: <p>If I rerun the code on the CPU only I directly get the error " CUDA error: device-side assert triggered" in line torch.sqrt(dist2). There are no nan values or values smaller than 0 in dist2. The error could result from the function itself since the knnquery is a C++ and Cuda implementation.</p>

ptrblck: <p>Rerunning the code on the CPU shouldn’t raise a CUDA error, if the GPU is not used at all.<br>
I don’t know if you are using a Jupyter notebook or another REPL environment, but make sure to restart the environment once you ran into a CUDA assert as these errors might be sticky.</p>

maze09: <p>The knnQuery is always running on GPU since it is a compiled GPU library. I will implement the functions with pure pytorch functions and check if the error still appears. I do not use any REPL environment. Thank you for the recommendations.</p>
Tensor Slice Views?

tempraryworkaccount: <p>Making the swap from TF to pyTorch im really liking how view is implemented on the back end, but im having trouble scouring the source for the advanced indexing implementation.</p>
<p>Does pytorch make new tensors according to a tensor[slice] or is it also a like viewing mask? If the former, can you point me to the source files where i could change the standard to the latter?</p>

albanD: <p>Hi,</p>
<p>The <code>.view()</code> calls basically only change the metadata of the tensor (size and strides).<br>
This means that the data in memory are completely untouched, only the way you interpret (read) them is changed.<br>
Does that answer your question?</p>

tempraryworkaccount: <p>hey, yes i am aware of that. I am wondering whether tensor slices like tensorA[32:64] are implemented in a similar way to view(), or if slices do actually copy the data in tensorA.</p>
<p>I want “view slices” for my model, not real numpy copy slices as im doing a lot of tensor slicing.</p>

albanD: <p>So the advance indexing does it’s best to avoid copy, but sometimes it has to do.<br>
If you absolutely want to avoid copies, you can use <code>.select()</code> and <code>.narrow()</code> to make sure you will always have copy-free subsets of your tensor.<br>
I don’t remember where the advance indexing code is, but it’s a few hundred lines of cpp somewhere <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=6" title=":smiley:" class="emoji" alt=":smiley:"></p>

checkpt: <p>Hi, I have a similar issue than the author, 2 years later <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I wanted to know it it’s possible to index a tensor (on cuda device), without copy.</p>
<p>That is do something like <code>x2 = x[idx]</code> where idx is a sequence of int.</p>
<p>From my understanding, <code>narrow()</code> can’t handle this case if the sequence of int is something like <code>idx=[1, 5, 9]</code>.</p>

albanD: <p>Hi,</p>
<p>A Tensor can only be backed with a continuous chunk of memory and with a single stride per dimension.<br>
So unless your indices are spaced in a very specific way, you won’t be able to do that <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>

Sourabh: <p>Hi <a class="mention" href="/u/alband">@albanD</a> ,I have been trying to do weight clustering and have also posed a question for the same at <a href="https://discuss.pytorch.org/t/weight-clustering/142155">https://discuss.pytorch.org/t/weight-clustering/142155</a>. However,as Patrick rightly points out that, the model size won’t reduce (which is my ultimate goal) because each weight is saved individually. Is there a way of using <code>view</code> to utilize the same memory storage to achieve this? Could you possibly direct me to a way or an example that can help achieve this in reference to the question I posed? As this thread was somewhat relevant with views , I thought of asking it here.</p>
TypeError: empty(): argument 'size' must be tuple of ints, but found element of type Tensor at pos 1

Samah_Abu_saleem: <p>I try to implement GCN on my custom dataset, but I got error:</p>
<p>my code:</p>
<pre><code class="lang-auto"># Define our GCN class as a pytorch Module
class GCN(torch.nn.Module):
    def __init__(self, hidden_channels):
        super(GCN, self).__init__()
        # We inherit from pytorch geometric's GCN class, and we initialize three layers
        self.conv1 = GCNConv(data.num_features, hidden_channels)
        self.conv2 = GCNConv(hidden_channels, hidden_channels)
        self.conv3 = GCNConv(hidden_channels, hidden_channels)
        # Our final linear layer will define our output
        self.lin = Linear(hidden_channels, data.num_classes)
        
    def forward(self, x, edge_index, batch):
      # 1. Obtain node embeddings 
      x = self.conv1(x, edge_index)
      x = x.relu()
      x = self.conv2(x, edge_index)
      x = x.relu()
      x = self.conv3(x, edge_index)   
 
      # 2. Readout layer
      x = global_mean_pool(x, batch)  # [batch_size, hidden_channels]
 
      # 3. Apply a final classifier
      x = F.dropout(x, p=0.5, training=self.training)
      x = self.lin(x)
      return x
    
model = GCN(hidden_channels=16)
print(model)
</code></pre>
<p>The error:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_59/1705364933.py in &lt;module&gt;
     32       return x
     33 
---&gt; 34 model = GCN(hidden_channels=16)
     35 print(model)

/tmp/ipykernel_59/1705364933.py in __init__(self, hidden_channels)
     14         self.conv3 = GCNConv(hidden_channels, hidden_channels)
     15         # Our final linear layer will define our output
---&gt; 16         self.lin = Linear(hidden_channels, data.num_classes)
     17 
     18     def forward(self, x, edge_index, batch):

/usr/local/lib/python3.8/dist-packages/torch/nn/modules/linear.py in __init__(self, in_features, out_features, bias, device, dtype)
     83         self.in_features = in_features
     84         self.out_features = out_features
---&gt; 85         self.weight = Parameter(torch.empty((out_features, in_features), **factory_kwargs))
     86         if bias:
     87             self.bias = Parameter(torch.empty(out_features, **factory_kwargs))

TypeError: empty(): argument 'size' must be tuple of ints, but found element of type Tensor at pos 1
</code></pre>

thecho7: <p>Could you print out <code>data.num_classes</code>?</p>

Samah_Abu_saleem: <p><code>tensor([0, 1, 3, 2])</code></p>

thecho7: <p><code>nn.Linear</code> should be defined by two integers.</p>
<p>In short<br>
<code>data.num_classes = n</code> instead of <code>data.num_classes = torch.tensor([0, 1, 3, 2])</code></p>
If sample has more than one feature,how can I use Lstm to solve problems?

Eternal_Night: <p>After I read the Pytorch doc,I tried to solve a problem which is about price forcasting.But the doc only show creating a model with the sample that has only one feature.My sample has more than one feature,So how do I use Lstm to create model .I will appreciate if you can show me with detail codes.</p>

vdw: <p>You an simply represent all your features for a data sample as a single vector.</p>
<p>How do you features look like? Do you already have a working model with one feature?</p>

Eternal_Night: <p>Well,my sample is (1,13),label is (1,1),length is 2000.So can I predict label using sample based on torch.nn.LSTM? If I can,I will appreciate if u can tell how to write code.Thanks.</p>
How to confirm parameters of frozen part of network are not being updated?

Zafar_Aziz: <p>I am new to PyTorch. I set the requires_grade for the features extraction layers of vgg16 to false (as I want to freeze these layers for fine tuneing the model) using following code:</p>
<p>for name, param in model.named_parameters():<br>
if param.requires_grad and ‘features’ in name:<br>
param.requires_grad = False</p>
<p>Is it the correct way of freezing some of the network’s layers? Also is there any way to confirm that the weights and biases are not being updated in the frozen part of the network?</p>

ptrblck: <p>You could check the <code>.grad</code> attribute of the frozen parameters after the <code>backward()</code> call and make sure it’s set to <code>None</code>. Additionally, you could also print the parameters before and after the <code>optimizer.step()</code> call to make sure they are not updated.</p>

Zafar_Aziz: <p>Thanks for your response. I have no idea how to check .grad, please guide me.</p>

ptrblck: <p>You can directly access this attribute:</p>
<pre><code class="lang-python">print(model.layer.weight.grad) # should print None
loss.backward()
print(model.layer.weight.grad) # should still print None if the layer is frozen
</code></pre>

Zafar_Aziz: <p>Thank you again, I checked that the .grad attribute of frozen layers before and after loss.backward() are none, so its mean the parameters of frozen layers will not be updated during finetuning.</p>
Where is the actual implementation/source code for nn.PixelShuffle?

Anshumaan_Dash: <p>I can find just a docstring when I go to <code>torch.nn.functional.pixel_shuffle</code>. Where can I find the actual implementation?</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/pixelshuffle.py">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/pixelshuffle.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/pixelshuffle.py" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/master/torch/nn/modules/pixelshuffle.py</a></h4>


      <pre><code class="lang-py">from .module import Module
from .. import functional as F

from torch import Tensor


class PixelShuffle(Module):
    r"""Rearranges elements in a tensor of shape :math:`(*, C \times r^2, H, W)`
    to a tensor of shape :math:`(*, C, H \times r, W \times r)`, where r is an upscale factor.

    This is useful for implementing efficient sub-pixel convolution
    with a stride of :math:`1/r`.

    See the paper:
    `Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network`_
    by Shi et. al (2016) for more details.

    Args:
        upscale_factor (int): factor to increase spatial resolution by

</code></pre>



  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/pixelshuffle.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


ptrblck: <p>The implementation can be found <a href="https://github.com/pytorch/pytorch/blob/3a66a1cb99d7e1c1fe7abf8f390e177b9183a436/aten/src/ATen/native/PixelShuffle.cpp#L13-L58">here</a>.</p>

Anshumaan_Dash: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks a bunch!!</p>

Anshumaan_Dash: <p>Hey,<br>
I cannot find the cpp file in the installed packages anywhere. I want to change the implementation locally. Where can I do that?</p>

ptrblck: <p>You won’t be able to locally change a C++ source file and see the changes without rebuilding it.<br>
To apply changes to the <code>PixelShuffle</code> implementation, use <a href="https://github.com/pytorch/pytorch#from-source">this guide</a> to build PyTorch from source, manipulate the file, and rebuild it.<br>
<a href="https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md">This Contributing README</a> might also be useful for incremental builds.</p>

Anshumaan_Dash: <p>Thanks a lot for helping out. Really appreciate the effort.</p>
How to register `torch.empty(t, requires_grad = True)` to the model?

gebrahimi: <p>I have a model like this:</p>
<pre><code class="lang-auto">class Om_msg(nn.Module):
    def __init__(self):
        super(Om_msg, self).__init__()
        mode_clflen = [3, 128, 128]
        self.padding_parameters = [torch.empty(t,requires_grad=True) for t in mode_clflen]
        self.padding_parameters = [nn.init.normal_(w) for w in self.padding_parameters]
        
        lin = nn.Linear(3,4)
</code></pre>
<p>It seems like defining trainable parameters manually like above does not register them to the model and if i use the <code>model.state_dict</code>, the padding_parameters are not there and therefore i can not save them. The <code>nn.ModuleList</code>, also did not help and throws an error since torch.empty is not a module but a tensor. what is the best way to register the <code>padding_parameters</code> here?</p>
<p>Thanks.</p>

thecho7: <p>The easiest way to register the tensor is <code>nn.Parameter</code></p>
<p><code>torch.empty(t,requires_grad=True)</code> is to be<br>
<code>torch.nn.Parameter(torch.empty(t))</code></p>
<p><code>nn.Parameter</code> initially contains the gradient inside it.</p>
Finetuning Tutorial Missing Steps

awcchungster: <p>Hi there,</p>
<p>The Torchvision <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="noopener nofollow ugc">Object Detection Tutorial</a> discusses two paths, using Resnet50 or MobileNetV2. I tried copying the mobilnet code to replace the Resnet50 backbone and the image. Could someone either show me here or provide the updated colab notebook with the mobile net model working for the Fudan dataset fine-tuning?</p>
<p>Thanks.</p>
<blockquote>
<hr>
<p>TypeError                                 Traceback (most recent call last)<br>
 in ()<br>
7 )<br>
8 # For Training<br>
----&gt; 9 images,targets = next(iter(data_loader))<br>
10 images = list(image for image in images)<br>
11 targets = [{k: v for k, v in t.items()} for t in targets]</p>
<p>3 frames<br>
/usr/local/lib/python3.7/dist-packages/torch/_utils.py in reraise(self)<br>
432             # instantiate since we don’t know how to<br>
433             raise RuntimeError(msg) from None<br>
 → 434         raise exception<br>
435<br>
436</p>
<p>TypeError: Caught TypeError in DataLoader worker process 0.<br>
Original Traceback (most recent call last):<br>
File “/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/worker.py”, line 287, in _worker_loop<br>
data = fetcher.fetch(index)<br>
File “/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py”, line 49, in fetch<br>
data = [self.dataset[idx] for idx in possibly_batched_index]<br>
File “/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py”, line 49, in <br>
data = [self.dataset[idx] for idx in possibly_batched_index]<br>
File “”, line 67, in <strong>getitem</strong><br>
img, target = self.transforms(img, target)<br>
File “/content/transforms.py”, line 23, in <strong>call</strong><br>
image, target = t(image, target)<br>
File “/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py”, line 1102, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
TypeError: forward() takes 2 positional arguments but 3 were given</p>
</blockquote>
Error in self.dataset_iter = iter(dataset)

Oussama_Bouldjedri: <p>I am working on a json dataset, I manege to get the data into this shape :</p>
<p>[[[…],label 1],[[…].label2],…]</p>
<p>but I still cannot iterate tghough the data set this is my custum data set:</p>
<p>class JsonDataset(IterableDataset):<br>
def <strong>init</strong>(self, files):<br>
super(JsonDataset).<strong>init</strong>()<br>
self.files = files</p>
<pre><code>    self.data_full_list=[]
    for i in range(len(self.files)):
       
        liste_files_current_act=self.files[i]  ### list of files of this activity
       
        for j in range(len(liste_files_current_act)):
            
            with open(liste_files_current_act[j]) as file:
                 for line in file:
                   
                    line_tmp = file.readline()
                  
                    if line_tmp == None or line_tmp == '':
                        print('I got a null or empty string value for data in a file')
                        if line_tmp== None:
                            print('it is NONE')
                        if line_tmp== '':
                            print('it is a blank')
                        #input('none or blanck')
                    else:
                        res_line= json.loads(str(line_tmp))
                    
                    data_line=res_line['data']
      
                    self.data_full_list.append([data_line, i])
                
                     


    print('done')
    print(len(self.data_full_list))
 
def _len_(self):
    return(len(self.data_full_list))    

               
def __getitem__(self, idx):
    print(len(self.data_full_list))
    data,label=self.data_full_list[idx]
    
    
    
    data=torch.FloatTensor(data)
    print('type of data is ',type(data))
    print()
    print('label ',type(label))
    input()
    
   
    return(data,label)
</code></pre>
<p>I still get this error:</p>
<p>self.dataset_iter = iter(dataset)<br>
line 146, in <strong>iter</strong><br>
raise NotImplementedError</p>
<p>NotImplementedError</p>

ptrblck: <p><a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset"><code>IterableDatset</code></a> expects an <code>__iter__</code> method so if you want to use <code>__getitem__</code> use <code>torch.utils.data.Dataset</code> instead.</p>

Oussama_Bouldjedri: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="142433">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>torch.utils.data.Dataset</p>
</blockquote>
</aside>
<p>I am still having an error:</p>
<p>return iter(range(len(self.data_source)))</p>
<p>TypeError: object of type ‘JsonDataset’ has no len()</p>

ptrblck: <p>Your code snippet seems to have a typo in <code>__len__(self)</code> (note the two underscores while your code uses single underscores).</p>