akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
What is v vector in autograd and Jacobian matrix?

Mahdi_Amrollahi: <p>Here in this document we have a vector v which is multiplied by jacobian matrix.<br>
What is v vector?<br>
<a href="https://pytorch.org/tutorials/beginner/basics/autogradqs_tutorial.html" rel="noopener nofollow ugc">Pytorch doc</a></p>
Compute the gradients of all logins with respect to input

Yuancheng_Xu: <p>Given a neural network classifier with 10 classes (the final layer logits have dimension 10). Is there a way to compute the gradients of each of the logit w.r.t. the input in one backward propagation?</p>
<p>The only way I know to do this is by setting a for-loop like this</p>
<pre><code class="lang-auto">input_grad = torch.zeros(10,input_size)
logit = NN(x)
for i in range(10):
   x.grad.zero_()
   logit[i].backward(retain_graph=True)
   input_grad[i] = x.grad
</code></pre>
<p>However, this for loop is much slower than one backprop. Another minor question: is it correct that the above for-loop has 10 times computational cost as a single backprop, if the network is large enough?</p>

AlphaBetaGamma96: <p>Could you try the following code and see if you get the same values?</p>
<pre><code class="lang-auto">logit = NN(x)
input_grad, = torch.autograd.grad(logit, x, torch.ones_like(logit)) #note the comma after input_grad
</code></pre>
<p>Could try something like this to directly compare actually,</p>
<pre><code class="lang-auto">input_grad = torch.zeros(10,input_size)
logit = NN(x)
for i in range(10):
   x.grad.zero_()
   logit[i].backward(retain_graph=True)
   input_grad[i] = x.grad

logit_all = NN(x)
input_grad_all, = torch.autograd.grad(logit_all, x, torch.ones_like(logit_all))

print(torch.allclose(input_grad, input_grad_all))
</code></pre>

Yuancheng_Xu: <p>Thank you for the reply!</p>
<p>Let’s say x is of size 5 * 3 * 32 * 32 (5 is the batch size) and num_classes = 10.<br>
I think in your example, input_grad_all has the same size of the input x (both of them are 5 * 3 * 32 * 32). However, I would like to compute the gradients of each logit w.r.t. the input. That is, I am expecting the gradient to be of size 10 * 5 * 3 * 32 * 32.</p>

AlphaBetaGamma96: <p>Could you share a minimal reproducible example of <code>NN</code>? Just so there’s a complete example I can use to debug your problem.</p>
How to save intermediate results and load it later with a datalaoder again?

jS5t3r: <p>I want to find a data structure to save intermediate results. Numpy and Pickle is not an option because I cannot load the data in chunks.</p>
<p>The data is images, but saving <code>.jpeg</code> etc… is a loss of information.</p>
<p>How can I save these intermediate results and which enables me to load them memory efficient with a data loader?</p>

ptrblck: <aside class="quote no-group" data-username="jS5t3r" data-post="1" data-topic="152288">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/js5t3r/48/40545_2.png" class="avatar"> jS5t3r:</div>
<blockquote>
<p>Numpy and Pickle is not an option because I cannot load the data in chunks.</p>
</blockquote>
</aside>
<p>I don’t quite understand the “cannot load the data in chunks” part, but given that I guess <code>torch.save</code> is not an option.</p>
<aside class="quote no-group" data-username="jS5t3r" data-post="1" data-topic="152288">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/js5t3r/48/40545_2.png" class="avatar"> jS5t3r:</div>
<blockquote>
<p>The data is images, but saving <code>.jpeg</code> etc… is a loss of information.</p>
</blockquote>
</aside>
<p>You could use a lossless compression format such as PNG.</p>

jS5t3r: <p>I think that:  Using instead of pickle or <code>torch.save</code>/<code>torch.load</code>. It is better to use <code>tf.records</code>, e.g. from the PyTorch DALI plugin.</p>
<p>What do you think about that?</p>

ptrblck: <p>Yes, DALI is certainly an option and compatible with PyTorch. I don’t know enough about <code>tf.records</code> to say if it’s needed or not.</p>

jS5t3r: <p>Well, I think that <code>tf.records</code> are able to store the tuples (images, tuples). And load it batch-wise on the GPU. So that the RAM is not heavily used.</p>
<p>At the moment I use <code>torch.save</code> and <code>torch.load</code>, which has the drawback just loads files at once and if the files very huge, my RAM is too small.</p>
<p>I havnt seen anything comparable in the pytorch lib.</p>
Transfer learning-Preprocess Necessary?

Huzaifa_Capricorn: <p>I am trying to use Resnet 50 for CIFAR-10 classification. I was wondering that is the step below necessary? X_Cifar has dimensions [1,3,32,32] and I believe Resnet50 would only process [1,3,224,224]? However, I have seen implementation without the above step. Not sure what is the right way.</p>
<p>Step:<br>
weights = ResNet50_Weights.DEFAULT<br>
preprocess = weights.transforms()<br>
X_resnet = preprocess(X_Cifar10)</p>
Using nn.transformer for audio spectrograms?(speech recognition)

deadman: <p>Hi, I was wondering if anyone knows of any examples where people use  nn.transformer on spectrogram data?</p>
<p>I currently have a gru model but wanted to implement a transformer model as well, but most examples are for NLP and I end up getting confused when trying to convert the model to work with my data.</p>
<p>My data consists of spectrogram bands and the target labels are words that have been said in each spectrogram. In the form of (Samples, Sequence Length, Features/bands), label_example=‘blue’.</p>
<p>Thanks for any help!</p>
Debugging Memory usage & inplace modifications

LeanderK: <p>I am tightly memory bound and currently debugging pytorch memory usage. I am running my script with <code>"CUDA_LAUNCH_BLOCKING": "1"</code>env var.</p>
<p>I always get an error during a custom <code>torch.autograd.Function</code> in the forward-part. I can’t explain why it allocates, I think it shouldn’t:</p>
<pre><code class="lang-auto">CUDA out of memory. Tried to allocate 5.32 GiB (GPU 1; 31.75 GiB total capacity; 21.18 GiB already allocated; 1.09 GiB free; 29.22 GiB reserved in total by PyTorch)
</code></pre>
<p>while executing (jac is very big…):</p>
<pre><code class="lang-auto">jac[cond].mul_(negative_slope)
</code></pre>
<p>this should just multiply some values and not allocate! I thought it was just a synchronisation issue, but I run it with <code>"CUDA_LAUNCH_BLOCKING": "1"</code>.<br>
Another thing I am wondering…<code>jac.requires_grad</code> returns <code>True</code>. Is this expected inside the <code>forward</code> function? I am calculating the gradient myself, so it should not be needed…</p>
<p>And last, while trying to see the saved tensors I run:</p>
<pre><code class="lang-auto">def memReport():
    tensors = []
    for obj in gc.get_objects():
        try:
            if torch.is_tensor(obj):
                test = obj.element_size() * obj.nelement()
                tensors.append(obj)
        except:
            pass
    tensors = sorted(tensors, key=lambda a: a.element_size() * a.nelement())
    for t in tensors[-30:]:
        print(f"{(t.element_size() * t.nelement())/1e+9:0.3f}: {t.shape}")
    print(f"total found: {sum([(t.element_size() * t.nelement())/1e+9 for t in tensors]):0.3f}")
</code></pre>
<p>(got it from the forums)<br>
it returns <code>8.855</code>…which is obviously not <code>21.18 GiB</code>. Where are the additional tensors?</p>

eqy: <p>Is <code>jac</code> a parameter? In that case it would be expected to have a <code>.grad</code> field even if it were a parameter (e.g., used by a custom autograd layer), as the <code>.grad</code> field would be used by the optimizer to update the parameter values during <code>optimizer.step()</code>.</p>

LeanderK: <p><code>Jac</code> is a tensor needed in the computational graph, but not a pytorch-parameter. It’s a function-parameter for the autograd-forward function wrt I need to calculate gradients for.</p>

LeanderK: <p>For more experiments it crashed at the <code>jac[cond].mul_(self.negative_slope)</code> line. I really need to get the allocation down (if this line really is the culprit), I don’t see why it needs to allocate here really.</p>
<p>EDIT:</p>
<p>it looks like the following code performs better:</p>
<pre><code class="lang-auto">mul = torch.ones_like(input)
mul[cond] = self.negative_slope
mul = mul.unsqueeze(1).unsqueeze(1).unsqueeze(1)
jac.mul_(mul)
</code></pre>
<p>is this possible? It looks like it’s faster and doesn’t use as much memory</p>
Is there a libtorch.natvis for debugging in Visual Studio?

swilson314: <p>When debugging libtorch source in Visual Studio, it would be really nice to see a more readable inspector of tensor, etc. I figured VS supported this because at some point std::vector got significantly more readable in the debugger. It seems .natvis files can be installed to do this. Here is an example for the Eigen library: <a href="https://github.com/cdcseacave/Visual-Studio-Visualizers/blob/master/Eigen.natvis" class="inline-onebox" rel="noopener nofollow ugc">Visual-Studio-Visualizers/Eigen.natvis at master · cdcseacave/Visual-Studio-Visualizers · GitHub</a></p>
<p>I don’t have any expertise with this, but I’m willing to give it a try, at least to easily see the shape and strides of a tensor. However, I wanted to see if this already exists?</p>

anubane: <p>I believe my issue (<a href="https://discuss.pytorch.org/t/debug-libtorch-while-developing-c-extension-for-pytorch/146717">Python/C++ mixed mode debug problem</a>) may be related to this one.</p>

Arash_Babahaji: <p>I didn’t find any for libtorch so I made one for myself. This has tensor, data, requires_grad, grad_Fn and grad. You can change it and add anything yourself.</p>
<pre><code class="lang-auto">&lt;?xml version="1.0" encoding="utf-8"?&gt; 
&lt;AutoVisualizer xmlns="http://schemas.microsoft.com/vstudio/debugger/natvis/2010"&gt;
  &lt;Type Name="at::Tensor" Priority="High"&gt;
    &lt;DisplayString&gt;{{ size={impl_.target_-&gt;numel_} }}&lt;/DisplayString&gt;
    &lt;Expand&gt;
      &lt;Item Name="Raw" ExcludeView="simple"&gt;impl_.target_&lt;/Item&gt;
      &lt;Item Name="Data" ExcludeView="simple"&gt;impl_.target_-&gt;storage_.storage_impl_.target_-&gt;data_ptr_.ptr_&lt;/Item&gt;
      &lt;Item Name="requires_grad" ExcludeView="simple"&gt;(*((torch::autograd::AutogradMeta*)impl_.target_-&gt;autograd_meta_._Mypair._Myval2)).requires_grad_&lt;/Item&gt;
      &lt;Item Name="grad_fn" ExcludeView="simple"&gt;(*((torch::autograd::AutogradMeta*)(*((c10::TensorImpl*)impl_.target_)).autograd_meta_._Mypair._Myval2)).grad_fn_&lt;/Item&gt;
      &lt;Item Name="grad" ExcludeView="simple"&gt;(*((torch::autograd::AutogradMeta*)impl_.target_-&gt;autograd_meta_._Mypair._Myval2)).grad_&lt;/Item&gt;
      &lt;ArrayItems&gt;
        &lt;Size&gt;impl_.target_-&gt;numel_&lt;/Size&gt;
        &lt;ValuePointer&gt;(float *)impl_.target_-&gt;storage_.storage_impl_.target_-&gt;data_ptr_.ptr_.data_&lt;/ValuePointer&gt;
        &lt;/ArrayItems&gt;
    &lt;/Expand&gt;
  &lt;/Type&gt;
&lt;/AutoVisualizer&gt;
</code></pre>

ares89: <p>This does not work in vscode.</p>
Meta Learning - keep parameter dependence after gradient step

ChrisBerg: <p>Hi everyone,</p>
<p>I want to implement a meta-learning algorithm and struggle a bit. I have two sets of parameters phi and theta, that are basically the same. phi are the normal parameters that are supposed to minimize some loss and theta are meta-parameters that should regularize phi. Now I have</p>
<p>phi ← phi - gradient(loss(phi) - (phi - theta)^2 )</p>
<p>i.e. I minimize for the normal loss and for keeping phi close to theta.</p>
<p>Now the new phi has a dependence on theta.</p>
<p>I want to repeat that step a couple of times and then take gradient of loss(phi) with respect to theta (the dependence is due to the regularization)</p>
<p>Now the dependence is not retained in the pytorch code that i have below. How can i retain the dependence in order to calculate the gradient with respect to theta?</p>
<pre><code class="lang-auto">import torch
from torch import nn
import torch.optim as optim
import higher


class Model(nn.Module):

    def __init__(self, n_in, n_out):
        super().__init__()
        self.x = nn.Linear(n_in, n_out)

    def forward(self, input):
        return self.x(input)


x = Model(2, 1)
y = Model(2, 1)

in_ = torch.randn(2,)

inner_opt = optim.Adam(x.parameters(), lr=0.05)
optimizer_y = optim.Adam(y.parameters(), lr=0.05)

with higher.innerloop_ctx(x, inner_opt, copy_initial_weights=False) as (fnet, diffopt):
    # Optimize the likelihood of the support set by taking
    # gradient steps w.r.t. the model's parameters.
    # This adapts the model's meta-parameters to the task.
    # higher is able to automatically keep copies of
    # your network's parameters as they are being updated.
    for _ in range(2):
        spt_loss = fnet(in_)
        reg_loss = 0
        for p, q in zip(x.parameters(), y.parameters()):
            reg_loss += ((p-q)**2).sum()
        spt_loss += reg_loss
        diffopt.step(spt_loss)

    loss = fnet(in_)
    loss.backward()

optimizer_y.step()
# output here is None
for q in y.parameters():
    print(q.grad)
</code></pre>
After using torch.flip(tensor,0) twice, the data is the same, but the picture looks different

Fuzhiyuan: <p>I use torch.flip to flip my file(3d) twice, dim is 0, but the second result<code>s picture does not the same as the original</code>s.  It seems that the first flip is Flipping 180 degrees perpendicular to the screen, and the second flip is flipping 180 degrees horizontally on the screen.<br>
Code:</p>
<pre><code class="lang-auto">tensor = torch.from_numpy(sitk.GetArrayFromImage(img))

tensor_flip = torch.flip(tensor,(0,))
tensor_flip2 = torch.flip(tensor_flip,(0,))
</code></pre>
<p>The point is tensor_flip2<code>s data is the same as tensor, but the image it represents is different from tensor</code>s.</p>

ptrblck: <p>I don’t understand this claim:</p>
<blockquote>
<p>data is the same as tensor, but the image it represents is different from tensor`s</p>
</blockquote>
<p>Could you describe how the “image” the tensor represents can be different if all values, the shape, and strides match exactly?</p>
<p>Assuming you are trying to plot the image, could you share a code snippet which would reproduce the issue?</p>

Fuzhiyuan: <p>hi <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<pre><code class="lang-auto">img = sitk.ReadImage("D:\\Code\\Deploy\\resampleITK.nii.gz")
img_numpy = sitk.GetArrayFromImage(img)
img_tensor = torch.from_numpy(img_numpy)  

img_flip = torch.flip(img_tensor,(0,))
flip_numpy = img_flip.numpy()  
flip_img = sitk.GetImageFromArray(flip_numpy)
sitk.WriteImage(flip_img,"D:\\Code\\Deploy\\flipimage.nii.gz")

img_re = torch.flip(img_flip,(0,))
re_numpy = img_re.numpy()
re_img = sitk.GetImageFromArray(re_numpy)
sitk.WriteImage(re_img,"D:\\Code\\Deploy\\reimg.nii.gz")
</code></pre>
<p><strong>reimg.nii.gz</strong> does not look like <strong>resampleITK.nii.gz</strong><br>
I am sorry i can not provide the file.</p>
Is different modules of different number of inputs impossible with `nn.Sequential`?

Chanu_hcu: <p>Consider the following code</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
from collections import OrderedDict

class sum_t(nn.Module):
  def __init__(self):
    super().__init__()
  
  def forward(self, x, y):
    return x + y

model = nn.Sequential(OrderedDict([
          ('conv1', nn.Conv2d(3, 3, 1)),
          ('relu1', nn.ReLU()),
          ('conv2', nn.Conv2d(3, 3, 1)),
          ('relu2', nn.ReLU()),
          ('sum0', sum_t()),
          ('sum1', sum_t())
        ]))

x = torch.randn(3, 3, 64, 64)
y = torch.randn(3, 3, 64, 64)
</code></pre>
<p>I want to pass two inputs <code>x, y</code> to the last two modules only, but one input <code>x</code> to the first four layers. Is it possible to do with <code>nn.Sequential</code>? How can I do <code>model(x,y)</code>?</p>
<p>If possible, I want to do it so that I can use the code from PyTorch and hook my codes.</p>

ptrblck: <p><code>nn.Sequental</code> won’t accept multiple inputs by default, but a few workarounds are posted in <a href="https://discuss.pytorch.org/t/how-can-i-pass-multiple-inputs-to-nn-sequential-layers/50868">this topic</a> with links to GitHub issues with more suggestions.<br>
If these custom <code>nn.Sequential</code> modifications don’t fit into your use case, I would recommend to write a custom <code>nn.Module</code> and define the <code>forward</code> method explicitly using both inputs.</p>
Too many values to unpack CIFAR100 to ImageNet

KanZa: <p>Why I am facing this error only because of changing the Dataset from CIFAR100 to ImageNet</p>
<pre><code class="lang-auto">def train_epoch(models, loss_module, criterion, optimizers_backbone, optimizers_module, dataloaders, epoch, epoch_loss):
    global loss 
    loss_module.train()
    global iters
    
    for data in tqdm(dataloaders['train'], leave=False, total=len(dataloaders['train'])):
        with torch.cuda.device(CUDA_VISIBLE_DEVICES):
            inputs = data[0].cuda()
            labels = data[1].cuda()
        iters += 1
        optimizers_backbone.zero_grad()
        optimizers_module.zero_grad()

        scores, _, features = models(inputs)    
        target_loss = criterion(scores, labels)
        
        if epoch &gt; epoch_loss:
            features[0] = features[0].detach()
            features[1] = features[1].detach()
            features[2] = features[2].detach()
            features[3] = features[3].detach()

        pred_loss = loss_module(features)
        pred_loss = pred_loss.view(pred_loss.size(0))
        
        m_module_loss   = LossPredLoss(pred_loss, target_loss, margin=MARGIN)
        m_backbone_loss = torch.sum(target_loss) / target_loss.size(0)        
        
        loss = m_backbone_loss + WEIGHT * m_module_loss 
        
        loss.backward()
        optimizers_backbone.step()
        optimizers_module.step()  
    return loss
</code></pre>
<p>scores, _, features = models(inputs)<br>
ValueError: too many values to unpack (expected 3)</p>

knitemblazor: <p>Hi,<br>
could you check how many values the models is returning.</p>

KanZa: <p>The error is on this line, models was dictionary earlier, I removed the dictionary, and now it is giving error<br>
scores, _, features = models(inputs)</p>
Loss.requires_grad = True in validation?

Z_Huang: <p>Hi community,</p>
<p>I have a question regarding validation loss. I have train, validation, and test in my algorithms.</p>
<p>I put train and validation in one train function. When in train, I got model.train(). When I validation, I got model.eval().</p>
<p>However, I am not sure how should I deal with loss.rquires_grad and loss.backard in model.eval(). Should I remain loss.requires_grad = True and loss.backward() in validation?</p>
<pre><code class="lang-auto">            loss.requires_grad = True
            loss.backward()
</code></pre>

ptrblck: <aside class="quote no-group" data-username="Z_Huang" data-post="1" data-topic="157561">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/z_huang/48/8201_2.png" class="avatar"> Z_Huang:</div>
<blockquote>
<p>Should I remain loss.requires_grad = True and loss.backward() in validation?</p>
</blockquote>
</aside>
<p>No, you should not train the model during the validation phase and usually you would wrap it in a <code>with torch.no_grad()</code> block to disable Autograd and to save memory by avoiding to store intermediate forward activations needed for the gradient computation.</p>
Select Tensor Entries by Condition on indeces

Texfy: <p>I’m looking for the most pythonic way to select all entries of a Tensor whose indices full fill a certain condition. I am aware of <code>torch.where()</code>, but the condition here acts on the value of each entry, not its indices.</p>
<p>E.g. I would want to select all entries where the indices <code>i</code> and <code>j</code> along the first two dimensions fulfill <code>i // k &gt; j // k</code>. What is the best way to achieve this?</p>

thecho7: <p>This is not a PyTorch-like question but if I were you, I try to do that in this way</p>
<pre><code class="lang-auto">M, N = 10, 11
a = torch.rand(M, N)
mask = torch.tril(torch.ones(M, N), diagonal=-1)
output = torch.masked_select(a, mask)
</code></pre>

Texfy: <p>Mmmh, that doesn’t really answer my question. I’m aware that I can use <code>masked_select()</code> or <code>where()</code> to select entries based on some mask. And you’re right that for the special case of <code>k = 1</code> I could use <code>tril()</code> or <code>triu()</code> to get a mask in very few lines of code.</p>
<p>What I’m asking though if there is a more general way to either select entries directly, or at least construct a boolean mask elegantly for an arbitrary function of the indices <code>i, j, ...</code> along a tensor’s dimensions.</p>

thecho7: <p>Sry, I don’t know that magical thing…<br>
Wait for somebody who can help</p>
Radix sort for tensor

Lily_Liu: <p>I’m wondering if PyTorch exposes the radix sort API or is there an efficient way to implement radix sort?</p>
<p>For example, I have a 2-D tensor:</p>
<pre><code class="lang-auto">[[1, 3,   2,  2, 1, 1],  --&gt; row 1
 [6, 7,   8,  8, 8, 7],  --&gt; row 2
 [9, 10, 10,  9, 9, 9]]  
</code></pre>
<p>If I perform a radix sort on the tensor (row 1 is the most important dim, if the elements are the same, they are ordered by row 2), I can get</p>
<pre><code class="lang-auto">[[1, 1,   1,  2, 2, 3],  --&gt; row 1
 [6, 7,   8,  8, 8, 7],  --&gt; row 2
 [9, 9,  9,  9, 10, 10]]  
</code></pre>
<p>Any suggestions on implementing this efficiently in PyTorch? Thanks!</p>

ptrblck: <p>You could directly call <code>x.sort()</code> and allow PyTorch to select the right sorting algorithm. Right now, it would call <code>bitonicSort</code> or <code>radixSort</code> depending on the size of the inputs as seen <a href="https://github.com/pytorch/pytorch/blob/a85d1f0bcdd02cf18d3b0517337458cb51a18cdb/aten/src/ATen/native/cuda/Sort.cu#L269-L281">here</a>.<br>
If you want to explicitly only call into a radix sort algorithm, you could write a custom CUDA extension as sen <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html">here</a>.</p>
Weird batch size memory behavior

Tulipian: <p>I have two the same object detection datasets in coco annotation format with different image sizes: one is 233MB (images are of variable size) and another one is 13MB (max side is 300px).</p>
<p>For BOTH datasets I cannot fit more than 5 images per batch using fasterrcnn_resnet50_fpn model on 24GB GPU.</p>
<p>This is how memory looks (commas separate differences in memory consumption for different runs):</p>
<p>Before executing script = 2.3GB</p>
<p>small ds 13 MB:<br>
batch = 1 → 5.8GB<br>
batch = 2 → 8.8 GB<br>
batch = 3 → 12.4GB<br>
batch = 4 → 12.6GB<br>
batch = 5 → 22.6GB, 18.8GB, 23.4GB, 22.9<br>
batch = 8 → 23GB</p>
<p>large ds 233MB:<br>
batch = 1 → 6.5GB, 6.7GB<br>
batch = 2 → 9.6GB<br>
batch = 3 → 11.3GB, 13.2GB, 11.3GB<br>
batch = 4 → 12.7GB, 15.5GB<br>
batch = 5 → 22.7GB<br>
batch = 8 → 22.9GB</p>
<p>I wonder why there is no differences between both datasets (probably because they are both small enough) and why there is such weird behavior: increasing batch from 5 to 8 takes almost no extra GPU memory, while going from 4 to 5 takes many GBs of memory.<br>
In any case, adding one extra image per batch causes huge increase in memory consumption, why is it so? Is it because of specific model architecture? I remember packing hundreds of images on 24GB GPU for Imagenet on classification models. I am trying to understand why in this scenario I can use only tiny batches of 5…</p>
<p>Img transformations consist of just turning image to Tensor:</p>
<pre><code class="lang-auto">def tensor_transform() -&gt; torchvision.transforms.Compose:
    custom_transforms = []
    custom_transforms.append(torchvision.transforms.ToTensor())
    return torchvision.transforms.Compose(custom_transforms)
</code></pre>
<p>and dataset is created asL</p>
<pre><code class="lang-auto">    cppe5 = Cppe5(
        root=train_data_dir,
        annotation=train_annotation_file,
        transforms=tensor_transform(),
    )

    return torch.utils.data.DataLoader(
        cppe5,
        batch_size=train_batch_size,
        shuffle=shuffle,
        num_workers=num_workers,
        collate_fn=collate_fn,
    )
</code></pre>
<p>Entire dataloader object code is <a href="https://codeshare.io/Od8q0g" rel="noopener nofollow ugc">here</a>.</p>
<p>Thank you!</p>

eqy: <p>How is the memory consumption being measured? Is memory stats being used? <a href="https://pytorch.org/docs/stable/generated/torch.cuda.memory_stats.html#torch.cuda.memory_stats" rel="noopener nofollow ugc">torch.cuda.memory_stats — PyTorch 1.12 documentation</a></p>
Cross-Validation and Epoch

Z_Huang: <p>I am trying to understand the relationship between cross-validation and number of epoch.</p>
<p>Let’s say I am using 10 fold cross-validation, and now I set the number of epoch = 100.</p>
<p>Does that mean my model will run through the whole data by 1000 times? Then, the actual number epoch became 1000?</p>

knitemblazor: <p>In the basic approach, called <em>k</em>-fold CV, the training set is split into <em>k</em> smaller sets (other approaches are described below, but generally follow the same principles). The following procedure is followed for each of the <em>k</em> “folds”:</p>
<blockquote>
<ul>
<li>A model is trained using k−1 of the folds as training data;</li>
<li>the resulting model is validated on the remaining part of the data (i.e., it is used as a test set to compute a performance measure such as accuracy).</li>
</ul>
</blockquote>
<p>So my answer is that the whole model will see your 10 - 1 th fold data which gets shuffled 100 times while training<br>
and the last fold for validation (100 times) in hundred epochs.</p>

Z_Huang: <p>i am sure one dataset will be executed 10 times in one epoch coz of 10 fold.</p>

knitemblazor: <p>then it contradicts the actual meaning of an epoch.<br>
An epoch in machine learning means one complete pass of the training dataset through the algorithm.</p>

Z_Huang: <p>True, that’s why I am feeling confused. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Pytorch Object Detection model deploy in iOS

akawsar: <p>I want to deploy a hand gestures model in iOS.The model has mentioned bellow. Can anyone help me to find a procedure how can i deploy this model in iOS. Or how to deploy pytorch object detection model in iOS.<br>
I want to mention one thing, I tried to produce .pt or .ptl file and run this on xcode following the pytorch objection detection sample project that is provided by pytorch official github page. But this produced error ‘can’t found model’. I think there may some issue. I am new in pytorch, please help.</p>
<p>Model Link: <a href="https://github.com/hukenovs/hagrid" class="inline-onebox" rel="noopener nofollow ugc">GitHub - hukenovs/hagrid: HAnd Gesture Recognition Image Dataset</a></p>

fpozzi: <p>Hi <a class="mention" href="/u/akawsar">@akawsar</a>,</p>
<p>here some useful resources:</p>
<ul>
<li><a href="https://github.com/pytorch/ios-demo-app/tree/master/ObjectDetection" class="inline-onebox" rel="noopener nofollow ugc">ios-demo-app/ObjectDetection at master · pytorch/ios-demo-app · GitHub</a></li>
<li><a href="https://pytorch.org/mobile/ios/" class="inline-onebox" rel="noopener nofollow ugc">iOS | PyTorch</a></li>
<li><a href="https://pytorch.org/tutorials/recipes/model_preparation_ios.html" class="inline-onebox" rel="noopener nofollow ugc">Model Preparation for iOS Recipe — PyTorch Tutorials 1.12.0+cu102 documentation</a></li>
</ul>

Linbin: <p>This sounds like a resource issue. Make sure you add the model file to xcode project correctly.</p>
Swap weights during training

Erica_Zheng: <p>Dear All,</p>
<p>I am implementing a two-branch model. Each branch has the same architecture. During the training process, some layers swap weights.</p>
<p>That is,<br>
some layers in <code>branch_A</code> copy weights from <code>branch_B</code>;<br>
some layers in <code>branch_B</code> copy weights from <code>branch_A</code>.</p>
<p>Any solution for this?<br>
Thank you.</p>

AlphaBetaGamma96: <p>Could you share a minimal reproducible example?</p>
<p>If <code>branch_A</code> and <code>branch_B</code> are their own objects that have all layers in that specific branch you could just do a zip for-loop over each branch like,</p>
<pre><code class="lang-auto">for layer_A, layer_B in zip(branch_A, branch_B):`
  for param_A, param_B in zip(layer_A, layer_B):`
    if(branch_A_to_branch_B == True):
      #copy across
    elif(branch_B_to_branch_A == True):
      #copy across
    else:
      #no copy, just pass here...
</code></pre>
<p>But do be careful to not overwrite layers that may be updated from one branch but needed in another branch.</p>
M4 competition Forecasting MLP

mlkj: <p>Hi,</p>
<p>I was wondering if somebody already tried to “solve” the M4 competition for forecasting using a simple MLP model with the use of PyTorch. I have already found a code for it where TensorFlow is used (<a href="https://github.com/Mcompetitions/M4-methods/blob/master/ML_benchmarks.py" class="inline-onebox" rel="noopener nofollow ugc">M4-methods/ML_benchmarks.py at master · Mcompetitions/M4-methods · GitHub</a>), but I would be interested in how it would be done using PyTorch.</p>
Struggling with differentiable renderer

sneiman: <p>I am working on disentangled representations employing a final ‘decoder’ that is a differentiable renderer - a simple line drawing algorithm.</p>
<p>The final stage of the network is a line renderer that takes a 4 dimensional latent representation, trained on synthetically generated images of edges. This portion of the net works well, and when coupled with a typical auto-encoder ‘decoder’ stage produces strong results - even with just 4 values as the latent rep.</p>
<p>The line renderer/decoder is  bresenham’s algorithm. It has been tested and works fine.</p>
<p>My problem is that the network no longer learns with the line renderer.</p>
<p>I am uncertain of a few things:<br>
Are errors being propagated through the renderer? How can I tell?<br>
Are gradients NOT being collected on operations in the renderer? How can I be sure?</p>
<p>Here is the exact code for the final stage - it is separated into a separate function in preparation for moving it to warp. I build the image in a numpy array to avoid all the leaf and in place operation issues:</p>
<pre><code class="lang-auto">def line_draw(inp: Tensor, args):

    img     = np.zeros((args.batch_size, 3, args.patch_size, args.patch_size))
    for b, i in enumerate(inp[:]):
        x0                  = round(i[0].item())
        y0                  = round(i[1].item())
        x1                  = round(i[2].item())
        y1                  = round(i[3].item())
        dx                  = abs(x1-x0)
        sx                  = 1 if x0&lt;x1 else -1
        dy                  = -abs(y1-y0)
        sy                  = 1 if y0&lt;y1 else -1
        error               = dx+dy
        
        while True:
            img[b,:,y0,x0]  = 1
            if x0==x1 and y0==y1:
                break
            e2              = 2*error
            if e2&gt;=dy:
                if x0==x1:
                    break
                error       = error+dy
                x0          = x0+sx

            if e2&lt;=dx:
                if y0==y1:
                    break
                error       = error+dx
                y0          = y0+sy
    
    timg    = torch.from_numpy(img).to(torch.float32)
    timg.requires_grad=True
    return timg

class VAEDecoder(pl.LightningModule):

    def __init__(self, args):
        super(VAEDecoder, self).__init__()
        self.save_hyperparameters()
        self.args               = args

    def forward(self, inp):
        return line_draw(inp, self.args)

    def __call__(self, inp):
        return self.forward(inp)
</code></pre>
<p>Any insight and help appreciated,</p>
<p>s</p>
How can I get the statistics automatically for Normalize()

Omar_Zayed: <p>Since last year, I’ve been using means and standard deviations that are far from my dataset and would like to know how to compute them in pytorch.</p>

fpozzi: <p>Hi <a class="mention" href="/u/omar_zayed">@Omar_Zayed</a>,<br>
you can just iterate over your dataset, collect data and then compute statistics.</p>
<p>Here an example with <a href="https://pytorch.org/vision/stable/generated/torchvision.datasets.MNIST.html#torchvision.datasets.MNIST" rel="noopener nofollow ugc">MNIST</a> dataset:</p>
<pre data-code-wrap="pyhon"><code class="lang-nohighlight">from torchvision.datasets import MNIST
from torchvision.transforms import ToTensor

from torch.utils.data import DataLoader
import torch

dataset = MNIST(root=".", download=True, transform=ToTensor())
dt = DataLoader(dataset, batch_size=8)

data = []

for sample, label in dt:
  data.append(sample)

data_stacked = torch.stack(data, dim=0)

# stats per channel:
mean = data_stacked.mean(dim=[0, 1, 3, 4])
std = data_stacked.std(dim=[0, 1, 3, 4])

print(f"mean: {mean}")
print(f"std: {std}")
</code></pre>
Problem with DistributedSampler and 'for' loop in multi-card environment

KevinZhang: <p>The pytorch version is 1.12.<br>
I use ‘for xxx in f:’ to load data in my code. and N cards are used to train model, so technically torch will start up N processes to load data. but the file IO shares memory so the ‘f’ is called over and over again by these data-loading processes. All the data-loading processes get into dead loops and can’t get the data right.<br>
I wrote a small concept of my code like this:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/a/4ae19ea96792375074c95369cf0dc702d50db059.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/4ae19ea96792375074c95369cf0dc702d50db059" title="c23dc45e0fd86f4288da0e5e2"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4ae19ea96792375074c95369cf0dc702d50db059_2_408x500.jpeg" alt="c23dc45e0fd86f4288da0e5e2" data-base62-sha1="aGqIHjG4NHan6mIreGmnh7KLjYZ" width="408" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4ae19ea96792375074c95369cf0dc702d50db059_2_408x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4ae19ea96792375074c95369cf0dc702d50db059_2_612x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4ae19ea96792375074c95369cf0dc702d50db059_2_816x1000.jpeg 2x" data-dominant-color="323038"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">c23dc45e0fd86f4288da0e5e2</span><span class="informations">1244×1522 54.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Pytorch distributed init_process_group hangs when using a custom port

justanhduc: <p>Hi. I am trying to use Pytorch distributed but strangely when calling <code>init_process_group</code> using a custom port by setting <code>os.environ['MASTER_PORT']</code> in Pytorch 1.10.1, it hangs but not in Pytorch 1.8. The initialization works well if I don’t set a custom port. Do you have any idea about this?</p>

anj: <p>Can you provide a small code snippet for us to repro the issue?</p>
<p>cc <a class="mention" href="/u/h-huang">@H-Huang</a> <a class="mention" href="/u/kwen2501">@kwen2501</a></p>
Error in DistributedDataParallel when parameters are torch.cfloat

NavidK: <p>I’m trying to implement torch’s DistributedDataParallel for my model,</p>
<pre><code class="lang-auto">model = nn.parallel.DistributedDataParallel(model)
</code></pre>
<p>In one of the model layers I have a complex-valued nn.Parameter,</p>
<pre><code class="lang-auto">weights =  nn.Parameter(self.scale * torch.rand(dim1, dim2, dtype=torch.cfloat))
</code></pre>
<p>When I wrap it with nn.parallel.DistributedDataParallel, the following run-time error appears:</p>
<pre><code class="lang-auto">File "python-3.10.0/lib/python3.10/site-packages/torch/distributed/utils.py", line 131, in _sync_params_and_buffers
    dist._broadcast_coalesced(
RuntimeError: Invalid scalar type
</code></pre>
<p>However, when I make the weights tensor with torch.float:</p>
<pre><code class="lang-auto">weights =  nn.Parameter(self.scale * torch.rand(dim1, dim2, dtype=torch.float))
</code></pre>
<p>There are no errors, does anyone know how can I come up with this error?</p>

ptrblck: <p>The complex support in <code>DDP</code> is tracked <a href="https://github.com/pytorch/pytorch/issues/80080">here</a> and not implemented yet.</p>

NavidK: <p>Thank! Ok, I did a bit of mathematical manipulation and replaced the complex tensors with real ones. It’s working now</p>
Example/label slices batching using DataPipes

BlueskyFR: <p>Hi!</p>
<p>With TensorFlow’s <code>tf.data.Dataset</code>, a dataset can be created from <code>([a, b, c], [d, e, f])</code>, in which case the tuples <code>(a, d)</code>, <code>(b, e)</code> and <code>(c, f)</code> will be issued when reading it.<br>
The main interest is that when calling <code>.batch(X)</code>, each dimension is batched separately, which allows easy preprocessing vectorization.</p>
<p>Now, one may think <em>hey, what about doing this with 2 DataPipes?</em> - well, if the in-memory structure you are reading examples + labels from only returns both at a time, the only efficient (high performance) way of building the pipeline is to have a single one that does what TF do.</p>
<p>I am very new to TorchText but I felt like this was not possible, so I am asking here <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>
<p>Thanks in advance for the answers!</p>
Understanding garbage collection of computation graphs in pytorch

handesy: <p>Hi, I am interested in how unused computational graphs are automatically garbage collected in pytorch. I used to use dynet (another dynamic NN library), where I can use <code>renew_cg()</code> to remove all previously created nodes every time I started creating a new graph for the current training example. However in pytorch, everything seems to be handled automatically, regardless whether I call <code>__call__</code> of an <code>nn.Module</code> or directly calling the function that implements the computation. Is there any source code/documentation that I can refer to? Thanks!</p>

smth: <p>the graphs are freed automatically as soon as the output Variable holding onto the graph goes out of scope. Python implemented refcounting, so the freeing is immediate.</p>

smth: <p>for example:</p>
<pre><code class="lang-python">x = Variable(...)

# Example 1
try:
   y = x ** 2
   z = y * 3
except:
    pass

# graph is freed here
</code></pre>
<pre><code class="lang-auto"># Example 2
try:
   y = x ** 2
   z = y * 3
   z.backward(...) # graph is freed here
except:
    pass
</code></pre>
<pre><code class="lang-auto"># Example 3
try:
   y = x ** 2
   z = y * 3
   z.backward(..., retain_variables=True)
except:
    pass
# graph is freed here
</code></pre>

jekbradbury: <p>The fundamental difference is that DyNet’s graphs are global objects held by the singleton autograd engine. In PyTorch (and Chainer) the graphs are attached to the variables that are involved in them and (as Soumith demonstrated) go out of scope when those variables do.</p>
Will the future of Pytorch soon officially support JavaScript and Java window?

JonathanSum: <p>I know we have ONNX, but it does not truly support a lot of functionality of Pytorch.<br>
I heard Tensorflow officially does not officially support ONNX and officially Javascript by themselves.</p>
<p>In Pytorch, I don’t think using a Backend in C++ or backend in Python for web development always works.</p>
<p>I hope it will be true by the Facebook team or Pytorch for Javascript, Java in Window, and other languages( maybe using torch.jit.</p>

albanD: <p>Hi,</p>
<p>There is no current work that I am aware of to try to add a javascript or desktop java frontend.<br>
You can check this issue that talks about it: <a href="https://github.com/pytorch/pytorch/issues/25091">https://github.com/pytorch/pytorch/issues/25091</a><br>
But no core dev is working toward that afaik.</p>
<p>But as mentioned in the issue, there is already a third party rust binding so I’m sure you can create one for js/java if you want to. We would be happy to answer any question you might have if you try to do that.</p>

JonathanSum: <p>I just want to make sure. If my window is window 10, does it mean I can do Android development with Pythorch and Java backend(Servlet or Spring) in window 10 with Pytorch? I tried to install the PyTorch from the website, it does not have an option for Java.</p>

albanD: <p>I am not sure that it would help as we only have binaries for android/ios.<br>
You can find the instructions here: <a href="https://pytorch.org/mobile/home/">https://pytorch.org/mobile/home/</a></p>

saudet: <p>It’s possible to use the C++ API from Java using the JavaCPP Presets for PyTorch:</p><aside class="onebox githubfolder">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">
      <a href="https://github.com/bytedeco/javacpp-presets/tree/master/pytorch" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:400/400;"><img src="https://avatars.githubusercontent.com/u/5856852?s=400&amp;amp;v=4" class="thumbnail" width="400" height="400"></div>

<h3><a href="https://github.com/bytedeco/javacpp-presets/tree/master/pytorch" target="_blank" rel="noopener nofollow ugc">bytedeco/javacpp-presets</a></h3>

<p><a href="https://github.com/bytedeco/javacpp-presets/tree/master/pytorch" target="_blank" rel="noopener nofollow ugc">master/pytorch</a></p>

  <p><span class="label1">The missing Java distribution of native C++ libraries</span></p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>It’s not a high-level Java API, yet anyway, so a lot of it isn’t actually nice to use from Java, but it’s usable, and doesn’t require Python, so it’s appropriate for applications in Spring and what not.</p>

JonathanSum: <p>Thank you for answering.</p>
java.lang.ArrayIndexOutOfBoundsException: vector

Virgilio_Magalhaes: <p>Hello everyone,</p>
<p>I’ve been trying to load a model that accept 2 images as input source and passed as a list. I got this error:</p>
<pre><code class="lang-auto">Process: org.pytorch.demo.imagesegmentation, PID: 10778
    java.lang.ArrayIndexOutOfBoundsException: vector
        at org.pytorch.LiteNativePeer.forward(Native Method)
        at org.pytorch.Module.forward(Module.java:52)
        at org.pytorch.imagesegmentation.MainActivity.run(MainActivity.java:140)
</code></pre>
<p>Code to load model:</p>
<p>I tried this one;</p>
<pre><code class="lang-auto">Map&lt;String, IValue&gt; outTensors = mModule.forward(
                IValue.tupleFrom(
                        IValue.from(inputTensor),
                        IValue.from(inputTensor2)
                )
        ).toDictStringKey();
</code></pre>
<p>and this</p>
<pre><code class="lang-auto">IValue img = IValue.listFrom(inputTensor, inputTensor2);
final Tensor outputTensor = mModule.forward(img).toTensor();
</code></pre>
<p>Please, someone could help me?</p>

Linbin: <p>Separate “forward” and “toTensor/toDictStringKey” statements so you can better identify the issue. If still not resolved, please paste the complete inference code here.</p>

JoshGreifer: <p>As I’m facing a very similar problem, and have tried similar solutions to no avail,  I’d be interested to know if anybody can produce a minimal working example of forward() with more than one input,  including the ‘example_inputs’ used in the torch.jit.script() command used to export the model.<br>
I’m using pytorch 1.10.2 and pytorch_android_lite:1.10.0</p>

Virgilio_Magalhaes: <p>Hi <a class="mention" href="/u/joshgreifer">@JoshGreifer</a></p>
<p>I got run my code with success when I used like it:</p>
<p><code>final Tensor outputTuple = mModule.forward(IValue.from(inputTensor), IValue.from(inputTensor2)).toTensor();</code></p>

JonathanSum: <p>I guess the reason is the model accepts 2 input, but you originally use one input or one list with 2 items to the model.</p>
[SOLVED] Load_state_dict error

Petro_Key: <p>Hi, thank you always for your help.<br>
When I load my trained model using load_state_dict, it raises the following error:</p>
<p>File “/root/.pyenv/versions/anaconda3-5.3.1/lib/python3.7/site-packages/torch/optim/optimizer.py”, line 114, in load_state_dict<br>
raise ValueError("loaded state dict contains a parameter group "<br>
ValueError: loaded state dict contains a parameter group that doesn’t match the size of optimizer’s group</p>
<p>I could loaded trained models until yesterday, and have suddenly become unable to load them.<br>
I will looking forward to hearing any suggestion to correct this issue.</p>
<p>Thank you in advance:)</p>

tom: <p>Apparently the number parameters known to the optimizer changed somehow.<br>
Note that this is from loading the optimizer’s state dict, not the model.</p>
<p>Best regards</p>
<p>Thomas</p>

Petro_Key: <p>Hi tom,<br>
Thank you for your kind reply.<br>
As you suggested, I used wrong numbers of parameters.<br>
I tried to load parameters for different layer depth by mistake.<br>
It now works by setting the model size and parameter size.</p>
<p>Thank you for your help!</p>

SU_HE: <p>Hi Tom,<br>
I met an issue that I want to add an additional layer on the original model but still using a pretrained checkpoint. Is there a possible solution?</p>

tom: <p>Yes, you can either modify the state dict or make <code>load_state_dict</code> less strict.<br>
Personally, I tend to favor the former variant (having a translation function for keys and/or adding the model.state_dict() values for things not in the saved state dict) because it seems less likely that I forget things, but the latter would probably be faster.</p>
<p>Best regards</p>
<p>Thomas</p>

SU_HE: <p>Thanks for your information. It is very helpful!<br>
Best regards!<br>
Su</p>

loner: <p>Hi Tom,</p>
<p>in train<br>
last_step = saver.restore(model_dir, map_location=self.device)</p>
<p>raise ValueError("loaded state dict contains a parameter group "<br>
ValueError: loaded state dict contains a parameter group that doesn’t match the size of optimizer’s group</p>
<p>Is there a possible solution?</p>

tom: <p>The problem you have there likely is that the number of parameters has changed between saving the optimizer and restoring it. I am afraid that this is a tough problem as the optimizer is completely unaware of which parameter is what (the parameters are an just an ordered list to the optimizer).</p>
<p>Best regards</p>
<p>Thomas</p>