akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to copy parameters from modelA to modelB?

RuoyuGuo: <pre><code class="lang-auto">NetA = Model()
NetB = Model()
optimizer = optimizer.Adam(list(NetA.parameters())+ list(NetB.parameters))
scheduler = scheduler.somescheduler(optimizer)

for epoch in range(epochs):
    NetA.forward()
    NetB.forward()
   
    #Calculate Loss...

    optimizer.zero_grad()
    Loss.backward()
    optimizer.step()
    
    scheduler.step()
    #Want to copy parameters from NetB to NetA
</code></pre>
<p>Hi all. I have two networks and I train them simultaneously. Both of them have the same structure. After every epoch, I want to copy the parameters from NetB to NetA and train the next epoch. I know <code>deepcopy()</code> and <code>torch.state_dict()</code> can create a copy but this may make the optimizer stop working. What is the correct way to only copy the value of parameters from NetB to NetA but doesn’t affect any others?</p>
<p>Thank you!</p>

ptrblck: <p>I would probably use <code>paramA.copy_(paramB)</code> in a <code>no_grad</code> context, but it seems also loading the <code>state_dict</code> works as the parameters are still updated independently:</p>
<pre><code class="lang-python">NetA = nn.Linear(1, 1, bias=False)
NetB = nn.Linear(1, 1, bias=False)
optimizer = torch.optim.Adam(list(NetA.parameters())+ list(NetB.parameters()), lr=1.)

for _ in range(5):
    out1 = NetA(torch.randn(1, 1))
    out2 = NetB(torch.randn(1, 1))
    loss = out1 + out2
    
    optimizer.zero_grad()
    loss.backward()
    
    optimizer.step()
    print("After step")
    print("NetA.weight {}".format(NetA.weight))
    print("NetA.weight {}".format(NetB.weight))
    
    a = True
    if a:
        with torch.no_grad():
            for paramA, paramB in zip(NetA.parameters(), NetB.parameters()):
                paramA.copy_(paramB)
    else:        
        NetA.load_state_dict(NetB.state_dict())
    print("After reassigning")
    print("NetA.weight {}".format(NetA.weight))
    print("NetA.weight {}".format(NetB.weight))
</code></pre>
<p>However, after checking <a href="https://github.com/pytorch/pytorch/blob/b2311192e6c4745aac3fdd774ac9d56a36b396d4/torch/nn/modules/module.py#L1566-L1574">the code</a> it seems the same approach is used internally in <code>load_state_dict</code>.</p>

RuoyuGuo: <p>Thanks your suggestions. So these two methods <code>paramA.copy_(paramB)</code> with <code>no_grad()</code> and <code>NetA.load_state_dict(NetB.state_dict)</code> are the same ?</p>

ptrblck: <p>Yes, based on my code snippet and the linked code the same approach would be used.<br>
However, it would be great if you could also verify it with your real model in case I’m missing something in my small example.</p>
Installing a production line version of Pytorch

zezimabig: <p>Dear to whom it may concern,</p>
<p>I have tried to install Pytorch on my instance but found that it is too big of a file (at ~700MB). Is there a minimal version of Pytorch (smaller size) that I could install for the main purpose of functioning in the production line (requires a few functions)?</p>
<p>Below are the only imports that were used:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
from torch.distributions import MultivariateNormal
from torch.distributions import Categorical
</code></pre>
<p>If there is, may I have a recommended approach to install the CPU and/or the GPU supported versions of Pytorch (with terminal lines)?</p>
<p>Thank you so much!</p>

ptrblck: <p>No, unfortunately there are no pre-built binaries with a subset of PyTorch (at least no official ones) and I’m also unaware of building a subset from source.</p>
<p>I don’t know which methods you are exactly using but <code>import torch</code> would import all modules (some modules might need an explicit <code>import</code> but are in the same namespace).</p>

zezimabig: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="162909">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>No, unfortunately there are no pre-built binaries with a subset of PyTorch (at least no official ones) and I’m also unaware of building a subset from source.</p>
<p>I don’t know which methods you are exactly using but <code>import torch</code> would import all modules (some modules might need an explicit <code>import</code> but are in the same namespace).</p>
</blockquote>
</aside>
<p>Oh dear. Could this be a feature that can be requested from Pytorch?</p>

ptrblck: <p>Sure, you can create this feature request on GitHub and depending on the interest, someone might want to work on it. However, note that I don’t see which subset should be built if you depend on <code>import torch</code> directly.</p>
Collate + Sampling the batch in particular order

lukasz_borecki: <p>Hi,</p>
<p>Problem statement: I got variety images to classify from 40x42 to 560x420. I want to train it dynamicaly:</p>
<p>Partial Solution:I got batches close to their image size (collate_fn), but batches are not shuffled. (Close to their image size to dont pad too much)</p>
<p>What i want: How to shuffle the batches like, take low size images batch then higher size image size sequentialy ? low, high , low , medium etc.</p>
<p>I dont want manualy label batches in pandas df <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>One approach would be to create a custom <code>sampler</code> implementing your logic of using the spatial sizes to batch “similar” images together. You could check the implementation of a few built-in <a href="https://github.com/pytorch/pytorch/blob/fd5085c445c3f1a4c90e55154cf26fe30f52a0ab/torch/utils/data/sampler.py">samplers</a>, implement your logic in a custom class, and pass it to the <code>DataLoader</code>. I don’t know if you have any information about the image shapes at each index, but it might be worth to grab this information once as loading the entire dataset just to check the image shapes might be too expensive.</p>

lukasz_borecki: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks for quick reply. Yes i got shape information and its no problem. I will check sampler implementations however during the night i found idea outside pytorch to sort df before putting into Dataset. Then divide //BATCH_SIZE to give batch_id. Then shuffle batch_ids. But i want to do it in pytorch mechanics so i am starting to read.</p>
<p>–<br>
EDIT:</p>
<p>If i understand correctly, this logic i wanted to make in df i can make in Sampler class</p>
<p>Best regards</p>
What does mean Top-1 error	and Top-5 error

israRe: <p>Hi, every one.<br>
I’m reading the docs in PyTorch vision but what is the meaning of <strong>Top-1 error</strong> and <strong>Top-5 error</strong> present in this <a href="https://pytorch.org/vision/0.8/models.html" rel="noopener nofollow ugc">page</a>.</p>

ptrblck: <p>You would increase the <code>Top-1 error</code> if the “top-1” prediction, i.e. the prediction of your model with the highest score, does not match the target.<br>
On the other hand, you would increase the <code>Top-5 error</code> if none of the “top-5” predictions, i.e. the 5 predictions with the highest score, matches the target.</p>
What are the t values in the Adamw documentation

JamesDickens: <p>I’m reading the documentation for the Adamw optimizer. In the updates for the m_t and v_t terms, the beta values have a superscript of t. I thought these beta values are fixed. Any insights what this superscript means much appreciated.</p>

ptrblck: <p>Check the <a href="https://arxiv.org/pdf/1711.05101.pdf">paper</a> referenced in the docs in Algorithm 2:</p>
<blockquote>
<p><code>beta_1^t</code> is taken to the power of <code>t</code><br>
<code>beta_2^t</code> is taken to the power of <code>t</code></p>
</blockquote>
Loss function recommendation for multi-labels, multi-targets

Saswat: <p>I’m working on a classification problem which can have a variable number of classes as the ground truth. And for each data point, I’d like to have k possible targets.</p>
<p>This problem can be modelled as a single-label, multi-class problem with probabilistic (“soft”) labels. The soft targets can be configured to have a probability of 0.7 for class 3, a probability of 0.2 for class 2, and a probability of 0.1 for class 5 (and zero for everything else). <a href="https://discuss.pytorch.org/t/multi-target-loss-recommendations/162768/5">Ref</a></p>
<p>But for certain examples, I want the model to predict more than 1 class for a given data point. Should I use <code>BCEWithLogitsLoss</code>, <code>CrossEntropyLoss</code> or some combination of both?</p>

srishti-git1110: <aside class="quote no-group" data-username="Saswat" data-post="1" data-topic="162833">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/saswat/48/49862_2.png" class="avatar"> Saswat:</div>
<blockquote>
<p>But for certain examples, I want the model to predict more than 1 class for a given data point.</p>
</blockquote>
</aside>
<p>This just means that one data point can fall in multiple categories (rather than just one) – a multi-label multi-class problem, essentially.<br>
You should be fine using <code>BCEWithLogitsLoss</code>. Please note from the docs -</p>
<blockquote>
<p>This loss combines a Sigmoid layer and the BCELoss in one single class.</p>
</blockquote>
<p>which means you do not have to explicitly include a sigmoid layer.</p>

Saswat: <p>Got it! Thanks a lot <a class="mention" href="/u/srishti-git1110">@srishti-git1110</a></p>

Saswat: <p><a class="mention" href="/u/srishti-git1110">@srishti-git1110</a><br>
I had a follow-up question on this. Say my ground truth classes were [3,4,5,6] and their probabilities were [0.7, 0.7, 0.3, 0.3] and I wanted my model to pick <strong>one</strong> of [3,4] and another of [5,6], can the loss function be modelled accordingly?</p>
<p>Because the way I say it, it’ll always try to converge into predicting [3,4] as it has the highest weight but turns out, [5,6] are the next best options to choose from.</p>
Help - vision build from sources in linux for ROCm failed

gmnidjan: <p>I build vision from sources in linux for and it fails<br>
ninja fail build because in build.ninja <strong>nvcc=bin/hipcc</strong> , not correct way <strong>nvcc=/opt/rocm/hip/bin/hipcc</strong></p>
<p>i try to change this and build fail again, because <strong>hip/hip_runtime.h not found</strong></p>
<p>I think this is because one of environment variables or torch parameters equals null<br>
I don’t know about this parameters and cannot understand this from python code of vision setup.py<br>
Help</p>
Pytorch lightning causes slurm nodes to drain

meshghi: <p>Hello! When I train with DDP strategy, any type of crashes like <code>Out Of Memory (OOM)</code> error or <code>scancel</code> slurm job results in slurm nodes to drain due to <code>Kill task failed</code> which means that the pytorch lightning process running on these nodes failed to clean up after termination. I was wondering how I could fix this?</p>
<p>This is with multi-node, 8 GPUs per node. Tried on 2,3,4,and 5 nodes and all cases cause the issue. Feels like the higher the nodes we try, the higher the probability that one or more will go into the drain state.</p>
<pre><code class="lang-auto">callbacks_list = [
        lr_monitor,
        checkpoint_callback,
        swa_ensemble,
        PrintCallback(),
        callbacks.ModelSummary(),
        callbacks.DeviceStatsMonitor(cpu_stats=True),
        ]

logger = [
        loggers.tensorboard.TensorBoardLogger(save_dir="./logs", version=slurm_job_id), 
        ]

trainer = pl.Trainer(
            gpus=_cfg.slurm_job.gpus_per_node,
            num_nodes=_cfg.slurm_job.number_of_nodes, 
            accelerator="gpu",
            strategy=DDPStrategy(find_unused_parameters=False),
            plugins=pl.plugins.environments.SLURMEnvironment(auto_requeue=False),
            max_epochs=_cfg.training.fit.epochs, 
            callbacks=callbacks_list,
            logger=logger,
            accumulate_grad_batches=_cfg.training.fit.accumulation_steps,
            profiler="simple",
            )
data_module = DataModule(_cfg)
model_module = LitModelModule(_cfg)

trainer.fit(
            model=model_module, 
            datamodule=data_module,
            ckpt_path=None
        )
</code></pre>
<p>Environment:</p>
<pre><code class="lang-auto">- PyTorch Lightning Version (e.g., 1.5.0): 1.7.7
- PyTorch Version (e.g., 1.10): 1.12.0
- Python version (e.g., 3.9): 3.9.12
- OS (e.g., Linux): Linux --RHEL7.4 
- CUDA/cuDNN version: 11.7
- GPU models and configuration: RTX 5000's
</code></pre>
Odd behavior in a custom triplet loss function

RelativeQuanta: <p>Hello,</p>
<p>I have a model that is comparing embeddings of its inputs to embeddings of a constant set of templates.  I’m using a custom triplet margin loss as my model calculates the L2 distance internally.  I’m getting an inconsistent error (it doesn’t happen every batch) where one of my dimensions gets off by 1.</p>
<pre><code class="lang-auto">def calc_batch_hard_triplet_loss(distances, gt_indices):
    """
    Given L2 distances between the embedded anchor in each batch and each of the N templates and the indices indicating the single template match, calculates the batch hard triplet loss
    Args:
        distances: float tensor shape (batch, N) containing the L2 embedding distances
        gt_indices: int tensor shape (batch) containing the index of the matching template for each anchor
    Returns:
        The mean of the hardest triplet loss for each anchor
    """
    batch_size = distances.shape[0]
    n_templates = distances.shape[1]
    template_indices = torch.arange(n_templates, dtype=torch.int64).repeat(batch_size, 1).cuda()
    non_matches = template_indices[torch.where(template_indices != gt_indices.view(batch_size,1))]
    non_matches = non_matches.view(batch_size, n_templates - 1) # error here after ~40 batches
    min_non_match_distances = torch.gather(distances, dim=1,index=non_matches).min(dim=1).values
    match_distances = torch.gather(distances, dim=1, index=gt_indices.view(batch_size,1))
    diff = match_distances - min_non_match_distances + 1.0
    loss = torch.where(diff &gt; 0, diff, torch.zeros_like(diff)).mean()
    return loss
</code></pre>
<p>This loss seems to works for most batches then crashes with</p>
<pre><code class="lang-auto"> File "/work/code/train.py", line 91, in calc_batch_hard_triplet_loss
    non_matches = non_matches.view(batch_size, n_templates - 1)
RuntimeError: shape '[18, 485]' is invalid for input of size 8731
</code></pre>
<p>This makes no sense to me.  How does the template index selection step end up off by 1?  A bug in torch.arange?</p>
<p>(I’m running torch under the <a href="http://nvcr.io/nvidia/pytorch:22.08-py3" rel="noopener nofollow ugc">nvcr.io/nvidia/pytorch:22.08-py3</a> docker container)</p>
<p>Thanks for any help!</p>

RelativeQuanta: <p>It appears there’s an occasional error in torch.arange().  I appear to have solved my error by calling arange once at the start of training then copying the array each time, rather than rebuilding it.</p>

RelativeQuanta: <p>The fix wasn’t permanent.  <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"> It just took longer to happen.  Ideas are still very welcome.</p>

RelativeQuanta: <p>Hey, looks like that function doesn’t have a problem.  I just have some bad data entries.  <img src="https://discuss.pytorch.org/images/emoji/apple/roll_eyes.png?v=12" title=":roll_eyes:" class="emoji" alt=":roll_eyes:" loading="lazy" width="20" height="20"></p>
Detectron2: DatasetMapper - How many augmented images are created?

oryx: <p>Hi All</p>
<p>In detectron2, say I create a trainer based on the DefaultTrainer with a mapper based on DatasetMapper (code below).  With how many images is my dataset augmented?  Does the mapper create an augmented image for each of the transforms, e.g. for <em>n</em> images and <em>T</em> transforms my final training set has <em>n(T+1)</em> images? Or is there somewhere a parameter to tune that specifies how many augmented images are created for each input image, with the augmentations randomly sampled from the specified transforms?</p>
<p>Thanks!</p>
<pre><code class="lang-auto">from detectron2.engine import DefaultTrainer
from detectron2.data import DatasetMapper, build_detection_train_loader
from detectron2.data import transforms as T

def build_sem_seg_train_aug(cfg):

    augs = [T.ResizeShortestEdge(
            cfg.INPUT.MIN_SIZE_TRAIN, cfg.INPUT.MAX_SIZE_TRAIN, cfg.INPUT.MIN_SIZE_TRAIN_SAMPLING
        ),
            T.RandomBrightness(0.5, 2),
            T.RandomContrast(0.5, 2),
            T.RandomSaturation(0.5, 2),
            T.RandomFlip(prob=0.5, horizontal=True, vertical=False),
            T.RandomFlip(prob=0.5, horizontal=False, vertical=True)]
    return augs

class Trainer(DefaultTrainer):
    @classmethod
    def build_train_loader(cls, cfg):
        mapper = DatasetMapper(cfg, is_train=True, augmentations=build_sem_seg_train_aug(cfg))
        return build_detection_train_loader(cfg, mapper=mapper)
</code></pre>
<p>The remainder of the model follows the Colab demo: <a href="https://colab.research.google.com/drive/16jcaJoc6bCFAQ96jDe2HwtXj7BMD_-m5" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a></p>

Luis-PP: <p>Hi <a class="mention" href="/u/oryx">@oryx</a> ! did you get an answer? I have the same question.</p>

Atotem: <p>Hi <a class="mention" href="/u/oryx">@oryx</a> <a class="mention" href="/u/luis-pp">@Luis-PP</a> , I have the same question. Did any of you get the answer? Thanks.</p>
GPU only using 150/350 watts while training

Soban_Khan: <p>GPU only uses 150/350 watts while training a model. Increasing the batch size only increases the use of VRAM. Is there a way to use full potential of my GPU?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/c/0c9da5e71b2feffebbaf45788e8e7b102448bc9a.png" data-download-href="https://discuss.pytorch.org/uploads/default/0c9da5e71b2feffebbaf45788e8e7b102448bc9a" title="nvidia_smi"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/c/0c9da5e71b2feffebbaf45788e8e7b102448bc9a.png" alt="nvidia_smi" data-base62-sha1="1NBujUAuBQKnSqDdEEZypboqabU" width="690" height="454" data-dominant-color="411B35"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">nvidia_smi</span><span class="informations">736×485 55.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Two forward passes during training yield lower accuracy

avatar: <p>I have the below code</p>
<pre><code class="lang-auto">            criterion = nn.CrossEntropyLoss(reduction='mean', ignore_index=-1)

            pred_weak = model(train_u_img)
            pred_str  = model(train_u_strong)
            pred, _ = model(train_l_img)

            sup_loss = criterion(pred, train_l_label)


            loss = sup_loss
            
            loss.backward()
            
                      

   
</code></pre>
<p>My question is why this 3 forward passes make the model to perform worse than 1 forward pass. Although I need to take the values pred_weak , pred_str. How I can deal with it?</p>
<p>Thanks</p>

ptrblck: <aside class="quote no-group" data-username="avatar" data-post="1" data-topic="162796">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/avatar/48/47791_2.png" class="avatar"> avatar:</div>
<blockquote>
<p>My question is why this 3 forward passes make the model to perform worse than 1 forward pass.</p>
</blockquote>
</aside>
<p>You are not sharing a lot of information how the performance is measured etc., but I would guess you are using e.g. batchnorm layers, which will update their running stats in each forward pass. If the data from the additional forward passes doesn’t fit into the data domain from your actual validation dataset, the performance might decrease.</p>

avatar: <p>Actually I’m training a semi-supervised model. The train_u_img are the unlabeled images and the train_u_strong are the same unlabeled images which are strongly augmented. So, I want to forward pass 2 times these values and 1 time the actual label images. Additionally, I measure the unsupervised loss as the MSE of the pred_weak and pred strong like this: <code>csst_loss = criterion_csst(softmax_pred_strong, softmax_pred_weak)</code></p>
Multiple parallel Models in One Model

J_Johnson: <p>I would like to create a model with Pytorch that has several identical models not connected by autograd, but would train in parallel. However, I’m running into issues with performing the backward pass on the current way I’ve structured it. How would you structure this to run concurrent models in a single model? I.e. the inputs flow through separate parallel paths and should have their loss and backward prop be kept separate.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn


class DummyModel(nn.Module):
    def __init__(self, in_size, hidden_size, out_size, num_advisors):
        super().__init__()
        self.n1 = in_size
        self.h1 = hidden_size
        self.n2 = out_size
        self.num_advisors = num_advisors

        self.modlist = nn.ModuleList()
        for _ in range(self.num_advisors):
            self.modlist.append(
                nn.Sequential(nn.Linear(self.n1, self.h1), nn.Linear(self.h1, self.h1), nn.Linear(self.h1, self.n2)))

        self.sigm = nn.Sigmoid()

    def forward(self, x):
        return self.sigm(torch.stack([_(x) for _ in self.modlist], dim=1))


in_size = 10
hidden_size = 64
out_size = 1
num_advisors = 20
batch_size = 100

model = DummyModel(in_size, hidden_size, out_size, num_advisors)
loss_func = nn.BCELoss()

dummy_inputs = torch.rand((batch_size, in_size))
dummy_targets = torch.rand((batch_size, num_advisors, 1))

dummy_outputs= model(dummy_inputs)
loss=torch.stack([loss_func(dummy_outputs[:,x,:], dummy_targets[:,x,:]) for x in range(dummy_outputs.size()[1])])
loss.backward()
</code></pre>
<p>The error is:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "scratch.py", line 38, in &lt;module&gt;
    loss.backward()
  File "pythonProject1\lib\site-packages\torch\_tensor.py", line 396, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "pythonProject1\lib\site-packages\torch\autograd\__init__.py", line 166, in backward
    grad_tensors_ = _make_grads(tensors, grad_tensors_, is_grads_batched=False)
  File "pythonProject1\lib\site-packages\torch\autograd\__init__.py", line 67, in _make_grads
    raise RuntimeError("grad can be implicitly created only for scalar outputs")
RuntimeError: grad can be implicitly created only for scalar outputs
</code></pre>

srishti-git1110: <p>According to the error message, the tensor you are trying to call backward on isn’t a scalar, but rather a vector.</p>
<p>Did you mean it to be scalar valued? If not, I can further help you with how to use backward with vectors - let me know.</p>
GPU memory leak in a function during training

zshn25: <p>GPU memory leaks and keeps increasing until I get <code>RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 31.75 GiB total capacity; 28.41 GiB already allocated; 4.00 MiB free; 30.51 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF</code></p>
<p>I have a function to normalize pointsets. The leak is happening due to this function</p>
<pre><code class="lang-auto">def normalize_pointset(x):
    """x has shape B,3or4,N"""
    if x.shape[1] == 4:
        points = x[:,:3,:] / x[:,3,:].unsqueeze(1)+1e-7
    else:
        points = x
    del(x)
    points -= points.mean(dim=-1).unsqueeze(-1)   # center to origin
    points /= (points[:,0,:]**2+points[:,1,:]**2+points[:,2,:]**2).max(dim=1).values.view(-1,1,1)   # max length=1
    return points
</code></pre>

ptrblck: <p>Could you post a minimal executable code snippet showing the leak, i.e. an increase in memory which cannot be released and is lost, please?</p>

zshn25: <p>I made a minimum working example but this one doesn’t leak memory as my original code does</p>
<pre><code class="lang-auto">import torch
from torch import nn, optim

def normalize_pointset(x):
    """x has shape B,3or4,N"""
    if x.shape[1] == 4:
        points = x[:,:3,:] / x[:,3,:].unsqueeze(1)+1e-7
    else:
        points = x
    del(x)
    points -= points.mean(dim=-1).unsqueeze(-1)   # center to origin
    points /= (points[:,0,:]**2+points[:,1,:]**2+points[:,2,:]**2).max(dim=1).values.view(-1,1,1)   # max length=1
    return points

class MyLinear(nn.Module):
  def __init__(self):
    super().__init__()
    self.id = nn.Identity()
    self.b = nn.Parameter(torch.randn(24,3,122880))

  def forward(self, input):
    return self.id(input) + self.b

model = MyLinear().cuda()
optimizer = optim.Adam(model.parameters(), 1e-4)
for i in range(10000):
    data1 = torch.randn(24,4,122880).cuda()
    data2 = torch.randn(24,4,122880).cuda()

    optimizer.zero_grad()
    loss = ((model(normalize_pointset(data1)) - normalize_pointset(data2))**2).mean()
    loss.backward()
    optimizer.step()

</code></pre>
<p>I run my code without the <code>normalize_pointset(x)</code> function and it works fine (without any memory leak).</p>

ptrblck: <p>OK, let me know once you are able to narrow provide a code snippet which shows the leak and ping me, please.</p>
What is the difference between torch.nn.PairwiseDistance and TORCH.CDIST

JuyiLin: <p>I tried two function. I want to calculate two embedding simliarity.</p>
<pre><code class="lang-auto">            pdist = torch.nn.PairwiseDistance(p=2)
            dist1 = pdist(out, subgraphout)
            distance = torch.cdist(out,subgraphout,p=2)
            print(dist1)
            print(distance)
</code></pre>
<p>tensor([3.1004, 0.8522, 0.6747, 5.0929, 7.9956, 2.2303])<br>
tensor([[3.1004, 3.2278, 1.9171, 4.4367, 3.4980, 4.4546],<br>
[0.5083, 0.8522, 1.6410, 1.1123, 0.7054, 1.0289],<br>
[0.5388, 0.7062, 0.6747, 2.0988, 1.4070, 1.9033],<br>
[3.6810, 3.7502, 2.4701, 5.0929, 4.1728, 5.0457],<br>
[6.9577, 6.6525, 6.1944, 8.5276, 7.9956, 7.9310],<br>
[1.2486, 0.8955, 1.2607, 2.7699, 2.3470, 2.2303]],)<br>
I don’t understand their differences. Which one is suitable for my goal?</p>
Vulkan Backend Training and Mixed/Half Precision?

Stanley_C: <p>I was pleasantly surprised to find out that Pytorch has a <a href="https://pytorch.org/tutorials/prototype/vulkan_workflow.html" rel="noopener nofollow ugc">Vulkan backend</a>. Unfortunately, it only does inferencing at fp32. Is this backend still being developed, and will it support training and half/mixed precision in the near(or distant) future?</p>
One-parameter repeat

mmoelle1: <p>When I want to grow a possibly multi-dimensional tensor along one dimension this can be elegantly achieved with the <code>repeat_interleave</code> function, e.g.</p>
<pre><code class="lang-cpp">torch::Tensor x = torch::ones({2,3});
x.repeat_interleave(3,0);
</code></pre>
<p>Is there an equivalent for the regular <code>repeat</code> function? So far, I always need to specify both dimensions explicitly, e.g.</p>
<pre><code class="lang-cpp">x.repeat({3,1})
</code></pre>
<p>which is impractical when writing generic functions that should work for tensors with arbitrary dimensionality. What I do right now is this</p>
<pre><code class="lang-cpp">switch (x.sizes().size()) {
  case 1:
    return x.repeat({3});
  case 2:
    return x.repeat({3,1});
  case 3:
    return x.repeat({3,1,1});
...
}
</code></pre>
<p>but that is of course not truly generic.</p>
Print Statement gives me the error | Process finished with exit code 134 (interrupted by signal 6: SIGABRT)

APats: <p>New to python and mac. using IntelliJ and  getting this error when i am trying to run a program.</p>
<p>dyld[24325]: Library not loaded: ‘<span class="mention">@executable_path</span>/…/.Python’<br>
Referenced from: ‘/Users/A/IdeaProjects/TAPI/venv/bin/python’<br>
Reason: tried: ‘/Users/A/IdeaProjects/TAPI/venv/bin/…/.Python’ (no such file), ‘/usr/local/lib/.Python’ (no such file), ‘/usr/lib/.Python’ (no such file)</p>
<p>Process finished with exit code 134 (interrupted by signal 6: SIGABRT)</p>

APats: <aside class="quote no-group" data-username="APats" data-post="1" data-topic="162814">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/e47774/48.png" class="avatar"> APats:</div>
<blockquote>
<p>dyld[24325]: Library not loaded: ‘<span class="mention">@executable_path</span>/…/.Python’</p>
</blockquote>
</aside>
<p>I was able to solve the problem. Issue was:</p>
<ol>
<li>I had moved my project from one location to another.</li>
<li>Hence, related project files within the project were missing.</li>
</ol>
<p>I created a new project and moved the .py files to the new project. ran the py file and boom, it worked.</p>
How to set a tensor's value by index?

KONG_Wei_Kun: <p>how to set a tensor’s value by index?<br>
For example, we have a tensor a=[[0,0,0], [0,0,0], [0,0,0]],<br>
and index=[[0,0], [1,1], [2,2]], values=[1,2,-1], how can a change a to [[1,0,0], [0,2,0], [0,0,-1]]?</p>
<p>I tied scatter but failed, could you also explain when we need scatter?</p>

KFrank: <p>Hi Weikun!</p>
<aside class="quote no-group" data-username="KONG_Wei_Kun" data-post="1" data-topic="162808" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kong_wei_kun/48/53744_2.png" class="avatar"> KONG_Wei_Kun:</div>
<blockquote>
<p>how can a change a to [[1,0,0], [0,2,0], [0,0,-1]]?</p>
</blockquote>
</aside>
<p>Use pytorch’s tensor indexing.</p>
<p>Because <code>values</code> has shape <code>[3]</code> you will want the two index tensors that<br>
you use to index into <code>a</code> to also have shape <code>[3]</code>.  Then you can assign<br>
<code>values</code> to the view into <code>a</code> obtained by indexing into <code>a</code>.  Specifically:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.12.0'
&gt;&gt;&gt; a = torch.zeros (3, 3, dtype = torch.long)
&gt;&gt;&gt; ind0 = torch.tensor ([0, 1, 2])
&gt;&gt;&gt; ind1 = torch.tensor ([0, 1, 2])
&gt;&gt;&gt; values = torch.tensor ([1, 2, -1])
&gt;&gt;&gt; a[ind0, ind1] = values
&gt;&gt;&gt; a
tensor([[ 1,  0,  0],
        [ 0,  2,  0],
        [ 0,  0, -1]])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

KONG_Wei_Kun: <p>Thank you very much! You really helped me a lot!! <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p>
Failed to import torchtext with undefined symbol AutogradMeta

jiapei100: <p>Details can be seen at <a href="https://github.com/pytorch/text/issues/1928" rel="noopener nofollow ugc">https://github.com/pytorch/text/issues/1928</a></p>
<p>I actually have <strong>torch</strong> built from source and successfully installed. However, even if I have <strong>torchtext</strong> also successfully built from source and installed, I couldn’t import it and got the following <strong>ERROR</strong>:</p>
<pre data-code-wrap="console"><code class="lang-nohighlight">➜  ~ python
Python 3.10.6 (main, Aug 10 2022, 11:40:04) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import torch
&gt;&gt;&gt; import torchtext as tt
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "~/.local/lib/python3.10/site-packages/torchtext/__init__.py", line 6, in &lt;module&gt;
    from torchtext import _extension  # noqa: F401
  File "~/.local/lib/python3.10/site-packages/torchtext/_extension.py", line 64, in &lt;module&gt;
    _init_extension()
  File "~/.local/lib/python3.10/site-packages/torchtext/_extension.py", line 58, in _init_extension
    _load_lib("libtorchtext")
  File "~/.local/lib/python3.10/site-packages/torchtext/_extension.py", line 50, in _load_lib
    torch.ops.load_library(path)
  File "~/.local/lib/python3.10/site-packages/torch/_ops.py", line 568, in load_library
    ctypes.CDLL(path)
  File "/usr/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: ~/.local/lib/python3.10/site-packages/torchtext/lib/libtorchtext.so: undefined symbol: _ZTVN5torch8autograd12AutogradMetaE
&gt;&gt;&gt; 
</code></pre>
<p>My <strong>torch</strong> and <strong>torchtext</strong> are respectively installed as:</p>
<pre data-code-wrap="console"><code class="lang-nohighlight">➜  ~ pip show torch
Name: torch
Version: 1.13.0a0+git71dddec
Summary: Tensors and Dynamic neural networks in Python with strong GPU acceleration
Home-page: https://pytorch.org/
Author: PyTorch Team
Author-email: packages@pytorch.org
License: BSD-3
Location: ~/.local/lib/python3.10/site-packages
Requires: typing-extensions
Required-by: functorch, lpips, thop, timm, torchdata, torchtext, torchvision
➜  ~ pip show torchtext
Name: torchtext
Version: 0.14.0a0+3f9c349
Summary: Text utilities and datasets for PyTorch
Home-page: https://github.com/pytorch/text
Author: PyTorch core devs and James Bradbury
Author-email: jekbradbury@gmail.com
License: BSD
Location: ~/.local/lib/python3.10/site-packages
Requires: numpy, requests, torch, tqdm
Required-by: 
</code></pre>
<p>Can anybody give me a hand please??</p>
Can i train a model to classify a person which was already trained to detect objects?

Manjuphoenix: <p>I’ve trained and saved a model to detect few diseases in tomato plant, but now i want to extend the capability of the model for other plants also. So can someone give a gist or roadmap to achieve this instead of training the model again for tomato and other plant all together at once.<br>
I wanted to train the model  plant by plant depending on the accuracies</p>
<p>Thanks in advance</p>
Move model to GPU but keep some attributes on CPU

qiminchen: <p>I have a model defined as</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__():
        super(model, self).__init__()
    
        self.conv_0 = nn.Conv2d()
        self.conv_1 = nn.Conv2d()
        self.conv_2 = nn.Conv2d()

        self.make_coordinates()

    def make_coordinates(self):
        xs = torch.arange(512, device="cpu").float() 
        ys = torch.arange(512, device="cpu").float()
        zs = torch.arange(512, device="cpu").float()
        if normalize:
            xs = xs / (H - 1) * 2 - 1 # (-1, 1)
            ys = ys / (W - 1) * 2 - 1 # (-1, 1)
            zs = zs / (D - 1) * 2 - 1 # (-1, 1)
        self.coords = torch.stack(torch.meshgrid(xs, ys, zs), dim=-1)

    def forward(self, x):
        pass


# move model to GPU
model = Model().cuda()
</code></pre>
<p>And I move the model to GPU, my question is that is <code>self.coords</code> still on CPU or is it already moved to GPU because I move the whole model to GPU? If latter, is there any way to move the trainable parameters on GPU but keep some attributes (<code>self.coords</code> in this case) on CPU?</p>
<p>I only want to move some batches of <code>self.coords</code> to GPU when using it because <code>self.coords</code> is extremely large and would take up almost 2GB if I initialize on GPU.</p>
<p>There is another solution in which I don’t initialize coordinates in the model but in dataloader and pass batches of coordinates to <code>forward</code>, but in this case, if I want to pass all coordinates to <code>forward</code>, then feed forward will be performed multiple times which might take a lot of time.</p>

ptrblck: <p><code>self.coords</code> will be on the CPU since you are explicitly creating its inputs via <code>device="cpu"</code> and since none of these tensors are a registered buffer or parameter and thus won’t be moved to the GPU via <code>model.to()</code>.</p>

qiminchen: <p>thank you so much, so only registered buffer or parameter defined in model will be moved to the GPU via <code>model.to()</code>?</p>

ptrblck: <p>Yes, as well as registered submodules (with their submodules, buffers, parameters, etc.).</p>
Why does torchrun change localhost to host IP for master_addr?

xuef: <p>Hi,</p>
<p>I specify rdzv_endpoint as localhost:29500 in torchrun, but it resulted to the IP address of the host, and also change the port number.<br>
I ran it with distributed.launch, it works as specified, i.e. the master_addr is not changed.</p>
<p>In my single node run, distributed.launch works, but torchrun doesn’t. How can I prevent torchrun to do this?</p>
<p>Below is the log using torchrun:</p>
<p>INFO:torch.distributed.launcher.api:Starting elastic_operator with launch configs:<br>
entrypoint       : train.py<br>
min_nodes        : 1<br>
max_nodes        : 1<br>
nproc_per_node   : 2<br>
run_id           : 1<br>
rdzv_backend     : c10d<br>
<strong>rdzv_endpoint    : localhost:29500</strong><br>
rdzv_configs     : {‘timeout’: 900}<br>
max_restarts     : 0<br>
monitor_interval : 5<br>
log_dir          : None<br>
metrics_cfg      : {}</p>
<p>INFO:torch.distributed.elastic.agent.server.local_elastic_agent:log directory set to: /var/tmp/pbs.1649498.scinfra2/torchelastic_y5dqd8_5/1_i8wso1of<br>
INFO:torch.distributed.elastic.agent.server.api:[default] starting workers for entrypoint: python<br>
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous’ing worker group<br>
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous complete for workers. Result:<br>
restart_count=0<br>
<strong>master_addr=xxx.xxx.xxx. com</strong><br>
<strong>master_port=59346</strong><br>
group_rank=0<br>
group_world_size=1<br>
local_ranks=[0, 1]<br>
role_ranks=[0, 1]<br>
global_ranks=[0, 1]<br>
role_world_sizes=[2, 2]<br>
global_world_sizes=[2, 2]</p>

agu: <p><a class="mention" href="/u/d4l3k">@d4l3k</a> Could you take a look?</p>
Purpose of eps parameter in TripletMarginLoss

aditya-hari: <p>I’m trying to understand how exactly TripletMarginLoss is implemented in PyTorch and while I get most of it, I am unable to figure where exactly epsilon fits in</p>

ptrblck: <p>The <code>eps</code> value is used in the <code>pairwise_distance</code> calls as seen <a href="https://github.com/pytorch/pytorch/blob/b106a04d766c21d15137318c506fc1ed823016b9/aten/src/ATen/native/Loss.cpp#L153-L175">here</a>.</p>

aditya-hari: <p>Ah, was pretty straightforward in hindsight. Thanks for quick response!</p>
torch.jit.enable_onednn_fusion(True)

Coder_Man: <p>I cannot use torch.jit.enable_onednn_fusion(True). It says AttributeError: module ‘torch._C’ has no attribute ‘_jit_set_llga_enabled’. How can use that?<br>
Pytorch: 1.12.1</p>

Hao_Hao_Tan: <p>Faced the same issue, checked thru and found this PR: <a href="https://github.com/pytorch/pytorch/pull/74596" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/pull/74596</a></p>
<p>According to “Limitations” description, oneDNN is only enabled for Linux at the moment.</p>
DDP: DataLoader and spawn child processes

Brasilino: <p>Hello!</p>
<p>Looking at torch.nn.parallel.DistributedDataParallel() documentation, it states that "This container parallelizes the application of the given module by splitting the input across the specified devices by chunking in the batch dimension. "</p>
<p>Does that mean I can call torch.utils.data.DataLoader() to get a trainset in the context of the main process (i.e. before calling mp.spawn() ), spawn child processes (ranks) and when I do:</p>
<pre><code class="lang-auto">for batch_idx, (inputs, targets) in enumerate(trainset):
        inputs, targets = inputs.to(rank), targets.to(rank)
</code></pre>
<p>DDP will automatically split the trainset to each rank ?<br>
Or should I call DataLoader() within the child process ?</p>
<p>Thanks!<br>
Lucas</p>

aazzolini: <p>Hi Brasilino, this seems like an imprecision in our documentation, good catch. I think the comment got copied over from DataParallel, but differently from DataParallel, DistributedDataParallel actually does not further split the input. You are expected to have an instance of the DatasetLoader on each of the ranks, and feed different batches into DistributedDataParallel.</p>

Brasilino: <p>Thanks <a class="mention" href="/u/aazzolini">@aazzolini</a> for clarifying it.</p>
Number of output channels for binary segmentation

Sudu: <p>Hello, I am attempting a binary segmentation problem. I am confused about what output dimensions are expected for loss functions. For <code>BCELoss()</code> should the network and labels both have a shape of <code>nBatchx1xhxw</code>? And for <code>CrossEntropyLoss()</code> should the network output be <code>nBatchx2xhxw</code> and label be <code>nBatchxhxw</code>? It would be great if someone can clarify.<br>
Thanks!</p>

ptrblck: <p>Yes, your explanation is correct and the difference would be:</p>
<ul>
<li>
<code>nn.BCEWithLogitsLoss</code> treats your use case as a binary segmentation use case directly where the labels indicate the negative (0) or positive (1) class. “Soft” labels are also allowed, but it depends on your actual use case how you want to interpret a target of e.g. <code>0.5</code>. The model output and target are expected to have the same shape as <code>[batch_size, 1, height, width]</code> and the target should contain values in <code>[0, 1]</code>. Use <code>nn.BCEWithLogitsLoss</code> and make sure your model outputs logits for a better numerical stability compared to <code>nn.BCELoss</code> + <code>sigmoid</code>.</li>
<li>
<code>nn.CrossEntropyLoss</code> can be used as a 2-class multi-class segmentation use case and your mentioned shapes are also correct. As you can see, the target tensor does not contain the class dimension, since it contains class labels now in the range <code>[0, nb_classes]</code>, so <code>0</code> and <code>1</code> in your case since you are only dealing with 2 classes.</li>
</ul>

Sudu: <p>Thanks for the clarification. Can I also use <code>output.view(-1)</code> and <code>target.view(-1)</code> before passing to <code>nn.BCEWithLogitsLoss</code>? I also have a data imbalance problem (negative:positive ratio is 100) for which I am using <code>pos_weight=torch.tensor(100)</code>.</p>

ptrblck: <p>It might work if you flatten the output and targets, but I would rather stick to the explicit shape of <code>[batch_size, 1, height, width]</code>.<br>
Yes, <code>pos_weight</code> is a proper way to try to reduce overfitting to a majority class.</p>

Sudu: <p>Ok, when I try with <code>pos_weight=None</code>, the training is smooth but when I use the <code>pos_weight=100</code>, the training becomes very noisy and the loss doesn’t seem to converge fast enough (it does reduce but its very gradual). Could there be any reason for this? Also, am I specifying <code>pos_weight</code> correctly or should be a list with a <code>length=2</code> or something?</p>
VGG16 input size

OuisYasser: <p>Hello everyone !<br>
I have a quick question regarding the input size of vgg16 for example. As we know the recommended size is (224,224) but when using global average pooling, any size is accepted, why that ?<br>
thank you</p>

ptrblck: <p>Adaptive pooling layers return a defined output size and accept variable input shapes. This allows you to use a fixed number of input features to the first <code>nn.Linear</code> layer in the classifier section of the model while the input (images) might have different spatial sizes.</p>

OuisYasser: <p>But if the model is trained on size of (224x224) and then we use transfer learning on other images of size (500,500) doesn’t that infect the weights ? will the mode behave the same even if we change the input size?</p>

ptrblck: <p>I’m unsure what “infect the weights” means, but yes the input resolution will have an effect on the model performance and I would expect to see a drop if the spatial size differs by a lot (but I haven’t seen any experiments checking this behavior).</p>
Avoiding for loop for image rotation

luk8r: <p>Hi,<br>
I am trying to use torchvision.transforms.functional.rotate to rotate a batch of images (e.g. a tensor with shape [1000, 96, 96], i.e. 1k images). This works well if they are to be rotated by the same angle. In my case I have a vector of angles, one for each image.<br>
Is there a way to do this without running a for loop over each image to individually perform the rotation?</p>
<p>Thanks a lot in advance!</p>

InnovArul: <p>I am not sure if a direct method exists in <code>torchvision.</code><br>
However, you could achieve it using <code>kornia</code> library by first getting the <a href="https://kornia.readthedocs.io/en/latest/geometry.transform.html#kornia.geometry.transform.get_rotation_matrix2d">rotation matrices from angles</a>, then using <a href="https://kornia.readthedocs.io/en/latest/geometry.transform.html#kornia.geometry.transform.warp_affine">warp_affine</a> to warp the images using these rotation matrices.</p>
List files with the use of dir

Berny: <p>Hi. I need to convert the following matlab code to python:</p>
<pre><code>DirList = dir(fullfile(Folder, '*.mat'));
Data = cell(1, length(DirList));
</code></pre>
<p>Is there a simple way to do it in python?</p>

InnovArul: <p>This question is related to general python and not related to <code>pytorch</code> as such.</p>
<p>As I understand, the code is trying to get the list of <code>.mat</code> files in the given path.<br>
You may want to look at <a href="https://docs.python.org/3/library/pathlib.html">pathlib</a> or <a href="https://docs.python.org/3/library/os.path.html">os.path</a> libraries to work with &amp; manipulate folder paths.</p>
<p>Below is one of the ways to achieve what you want.</p>
<pre><code class="lang-auto">from pathlib import Path
folder = './rootpath' # give the root path here
folderpath = Path(folder)
dirlist = [str(p.absolute()) for p in list(folderpath.glob('*.mat'))]
</code></pre>

Berny: <p>Thank you very much. Just an other question: in matlab dirlist is a struct with 6 fields (name, folder, date,…) and I could get its attributes with:</p>
<pre><code>DirList(k).name
</code></pre>
<p>How can I do the same in python since dirlist is now a list of strings?</p>

InnovArul: <p>Basically, <code>list(folderpath.glob('*.mat'))</code> gives a set of <code>Path</code> objects corresponding to <code>.mat</code> files in that directory.<br>
You can call the object’s methods and attributes to get the info you want.</p>
<p>You can refer to the available methods on this object here:</p>
<p><a href="https://docs.python.org/3/library/pathlib.html#methods" class="onebox" target="_blank" rel="noopener">https://docs.python.org/3/library/pathlib.html#methods</a></p>