akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Problem errror in Resnet

randinoo: <pre><code class="lang-auto">class ResNetUNet(nn.Module):
  def __init__(self, n_class):
    super().__init__()

    self.base_model = torchvision.models.resnet18(pretrained=True)
    self.base_layers = list(self.base_model.children())

    self.layer0 = nn.Sequential(*self.base_layers[:3]) # size=(N, 64, x.H/2, x.W/2)
    self.layer0_1x1 = convrelu(64, 64, 1, 0)
    self.layer1 = nn.Sequential(*self.base_layers[3:5]) # size=(N, 64, x.H/4, x.W/4)
    self.layer1_1x1 = convrelu(64, 64, 1, 0)
    self.layer2 = self.base_layers[5]  # size=(N, 128, x.H/8, x.W/8)
    self.layer2_1x1 = convrelu(128, 128, 1, 0)
    self.layer3 = self.base_layers[6]  # size=(N, 256, x.H/16, x.W/16)
    self.layer3_1x1 = convrelu(256, 256, 1, 0)
    self.layer4 = self.base_layers[7]  # size=(N, 512, x.H/32, x.W/32)
    self.layer4_1x1 = convrelu(512, 512, 1, 0)

    self.upsample = nn.Upsample(scale_factor=2, mode='bilinear', align_corners=True)

    self.conv_up3 = convrelu(256 + 512, 512, 3, 1)
    self.conv_up2 = convrelu(128 + 512, 256, 3, 1)
    self.conv_up1 = convrelu(64 + 256, 256, 3, 1)
    self.conv_up0 = convrelu(64 + 256, 128, 3, 1)

    self.conv_original_size0 = convrelu(1, 64, 3, 1)
    self.conv_original_size1 = convrelu(64, 64, 3, 1)
    self.conv_original_size2 = convrelu(64 + 128, 64, 3, 1)

    self.conv_last = nn.Conv2d(64, n_class, 1)

  def forward(self, input):
    x_original = self.conv_original_size0(input)
    x_original = self.conv_original_size1(x_original)

    layer0 = self.layer0(input)
    layer1 = self.layer1(layer0)
    layer2 = self.layer2(layer1)
    layer3 = self.layer3(layer2)
    layer4 = self.layer4(layer3)

    layer4 = self.layer4_1x1(layer4)
    x = self.upsample(layer4)
    layer3 = self.layer3_1x1(layer3)
    x = torch.cat([x, layer3], dim=1)
    x = self.conv_up3(x)

    x = self.upsample(x)
    layer2 = self.layer2_1x1(layer2)
    x = torch.cat([x, layer2], dim=1)
    x = self.conv_up2(x)

    x = self.upsample(x)
    layer1 = self.layer1_1x1(layer1)
    x = torch.cat([x, layer1], dim=1)
    x = self.conv_up1(x)

    x = self.upsample(x)
    layer0 = self.layer0_1x1(layer0)
    x = torch.cat([x, layer0], dim=1)
    x = self.conv_up0(x)

    x = self.upsample(x)
    x = torch.cat([x, x_original], dim=1)
    x = self.conv_original_size2(x)

    out = self.conv_last(x)

    return out

Given groups=1, weight of size [64, 3, 3, 3], expected input[6451, 1, 5, 5] to have 3 channels, but got 1 channels instead
</code></pre>

eqy: <p>It looks like something is strange with your input. Standard torchvision models expect 3-channel color images in NCHW format (usually something like [64, 3, 224, 224] for ImageNet]). An odd-numbered batch size with 5x5 spatial resolution [6451, 1, 5, 5] is odd here.</p>
How cross entropy loss is calculated?

showbit: <p>output with 4 classes 0,1,2,3,-&gt;&gt;&gt;&gt;tensor([[-0.0820,  0.0076, -0.0952,  0.0285]]<br>
real output -&gt;&gt;&gt; tensor([1])<br>
Loss-&gt;&gt;tensor(1.3449, dtype=torch.float64, grad_fn=)</p>
<p>1*Log 0.0076 ???</p>

pascal_notsawo: <p>No, it is log(softmax(.)).<br>
Since what you have is the logits / unnormalized log probability. The softmax transforms it into a probability.</p>
<pre><code class="lang-auto">import torch
from torch import tensor
import torch.nn.functional as F

logits = tensor([[-0.0820, 0.0076, -0.0952, 0.0285]])
y = tensor([1])

#F.cross_entropy(logits, y)
-torch.log(F.softmax(logits, dim=1))
"""
tensor([[1.4345, 1.3449, 1.4477, 1.3240]]), ==&gt; 1.3449 for y=1
"""

</code></pre>
<p>The advantage with pytorch cross_entropy loss function is that it effectively combines the log(softmax()) for stability reasons.</p>
<pre><code class="lang-auto">-F.log_softmax(logits, dim=1)
</code></pre>

showbit: <p>Thank u so much pascal</p>
Compare two segmentation maps predictions

AKASH_KUMAR: <p>Hi,</p>
<p>I am using consistency between two predicted segmentation maps on unlabeled data. For labeled data, I’m using nn.BCEwithLogitsLoss and Dice Loss.</p>
<p>I’m working on videos that’s why 5 dimensions output.<br>
(batch_size, channels, frames, height, width)</p>
<p>I want to know how can we compare two predicted segmentation maps.</p>
<pre><code class="lang-auto"># gt_seg - Ground truth segmentation map. - (8, 1, 8, 112, 112)
# aug_gt_seg - Augmented ground truth segmentation map - (8, 1, 8, 112, 112)

predicted_seg_1 = model(data, targets)       # (8, 1, 8, 112, 112)
predicted_seg_2 = model(augmented_data, augmented_targets) #(8, 1, 8, 112, 112)

# define criterion
seg_criterion_1 = nn.BCEwithLogitsLoss(size_average=True)
seg_criterion_2 = nn.DiceLoss()

# labeled losses
supervised_loss_1 = seg_criterion_1(predicted_seg_1, gt_seg)
supervised_loss_2 = seg_criterion_2(predicted_seg_1, gt_seg)

# Consistency loss
if consistency_loss == "l2":
      consistency_criterion = nn.MSELoss()
      cons_loss = consistency_criterion(predicted_gt_seg_1, predicted_gt_seg_2)

elif consistency_loss == "l1":
      consistency_criterion = nn.L1Loss()
      cons_loss = consistency_criterion(predicted_gt_seg_1, predicted_gt_seg_2)

total_supervised_loss = supervised_loss_1 + supervised_loss_2
total_consistency_loss = cons_loss

</code></pre>
<p>Is this the right way to apply consistency between two predicted segmentation maps?</p>
<p>I’m mainly confused due to the definition on the torch website. It’s a comparison with input x with target y. I thought it looks correct since I want both predicted segmentation maps similar. But, 2nd segmentation map is not a target. That’s why I’m confused. Because if this could be valid, then every loss function can be applied in some or another way. That doesn’t look appealing to me. If it’s the correct way to compare, can it be extended to other segmentation-based losses such as Dice Loss, IoU Loss, etc.?</p>
<p>One more query regarding loss computation on labeled data:</p>
<pre><code class="lang-auto"># gt_seg - Ground truth segmentation map
# aug_gt_seg - Augmented ground truth segmentation map

predicted_seg_1 = model(data, targets)
predicted_seg_2 = model(augmented_data, augmented_targets)

# define criterion
seg_criterion_1 = nn.BCEwithLogitsLoss(size_average=True)
seg_criterion_2 = nn.DiceLoss()

# labeled losses
supervised_loss_1 = seg_criterion_1(predicted_seg_1, gt_seg)
supervised_loss_2 = seg_criterion_2(predicted_seg_1, gt_seg)

# augmented labeled losses
aug_supervised_loss_1 = seg_criterion_1(predicted_seg_2, aug_gt_seg)
aug_supervised_loss_2 = seg_criterion_2(predicted_seg_2, aug_gt_seg)

total_supervised_loss = supervised_loss_1 + supervised_loss_2 + aug_supervised_loss_1 + aug_supervised_loss_2

</code></pre>
<p>Is the calculation of total_supervised_loss correct? Can I apply loss.backward() on this?</p>
Error while loading t7 model

Sheethal00: <p>Hi all,</p>
<p>I am new to pytorch, i am trying load a model which trained using PyTorch v0.1.10</p>
<p>I am attaching the error screenshot for reference.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/e/4eff7020344571a54f2f431dceec663ca9e82882.png" data-download-href="https://discuss.pytorch.org/uploads/default/4eff7020344571a54f2f431dceec663ca9e82882" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/e/4eff7020344571a54f2f431dceec663ca9e82882_2_690x388.png" alt="image" data-base62-sha1="bgQvZLmcoMXCtUPHzZInLcGZvHA" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/e/4eff7020344571a54f2f431dceec663ca9e82882_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/e/4eff7020344571a54f2f431dceec663ca9e82882_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/e/4eff7020344571a54f2f431dceec663ca9e82882_2_1380x776.png 2x" data-dominant-color="191A1A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1920×1080 175 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Please help me here</p>
<p>Thanks</p>

ptrblck: <p>Based on the error message it seems you are trying to load a Torch7 model file, which is corrupted.<br>
This could happen, e.g. if the download failed, so you could try to re-download the file (or recreate it).</p>
Cif-10 dataset is normalized by dataloader?

jS5t3r: <p>Helllo,</p>
<p>I ve found this code: <a href="https://github.com/fra31/auto-attack/blob/master/autoattack/examples/eval.py" class="inline-onebox" rel="noopener nofollow ugc">auto-attack/eval.py at master · fra31/auto-attack · GitHub</a></p>
<pre><code class="lang-auto">    transform_list = [transforms.ToTensor()]
    transform_chain = transforms.Compose(transform_list)
    item = datasets.CIFAR10(root=args.data_dir, train=False, transform=transform_chain, download=True)
    test_loader = data.DataLoader(item, batch_size=1000, shuffle=False, num_workers=0)
</code></pre>
<p>Is the dataset already normalized by std mean and std var?</p>

AKASH_KUMAR: <p>No, you need to add it to the transforms_list.</p>
<p><code>transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])</code></p>
<p>You can look at the source code for CIFAR10 here: <a href="https://pytorch.org/vision/stable/_modules/torchvision/datasets/cifar.html#CIFAR10" class="inline-onebox" rel="noopener nofollow ugc">torchvision.datasets.cifar — Torchvision master documentation</a></p>
Question in pytorch result

randinoo: <p>can we  find the result of the accuracy is 93,40% and F1-score is 69.78%?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/a/8a1daeb48ad1db1f07500e3422893fccb724a8f8.png" alt="image" data-base62-sha1="jHPu5lnQreCZUuQLeieJQY6XYLu" width="221" height="105"></p>
Load fine-tuned model

lory: <p>Hello, I have a file where i trained a BERT model and saved the state_dict of the model. How can i load the model in another .py file and use it to predict the polarity some keyboard input sentences? I have the below code :</p>
<pre><code class="lang-auto"> import torch
from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-multilingual-uncased', do_lower_case=True)
model.load_state_dict(torch.load('model.pt'))
model.eval()
inputs = tokenizer("I do not feel good today.", return_tensors="pt")
outputs = model(**inputs)
print(outputs)
</code></pre>
<p>Assuming that <code>I do not feel good today.</code> is the sentence entered from keyboard, i would like the model to return 0 if it is negative or 1 if it is positive. How can I define the model so it will not be linked to the py file where I trained the model? I tried the below but that will start the training again( BertClassifier is the class where i trained the model) :</p>
<pre><code class="lang-auto">from bert import BertClassifier 
model = BertClassifier(freeze_bert=False)
</code></pre>
Positional Embedding in Bert

jyothiraditya: <p>Can someone explain how these positional embedding code work in BERT?</p>
<pre><code class="lang-auto">class PositionalEmbedding(nn.Module):

    def __init__(self, d_model, max_len=512):
        super().__init__()

        # Compute the positional encodings once in log space.
        pe = torch.zeros(max_len, d_model).float()
        pe.require_grad = False

        position = torch.arange(0, max_len).float().unsqueeze(1)
        div_term = (torch.arange(0, d_model, 2).float() * -(math.log(10000.0) / d_model)).exp()

        pe[:, 0::2] = torch.sin(position * div_term)
        pe[:, 1::2] = torch.cos(position * div_term)

        pe = pe.unsqueeze(0)
        self.register_buffer('pe', pe)

    def forward(self, x):
        return self.pe[:, :x.size(1)]
</code></pre>
Parallel Computation of Models in PyTorch

erenovic: <p>I am quite new to PyTorch, I have checked the discussion a lot but I could not find about the issue. I have 4 models which require around 6 GB memory on GPU but we have limited number of GPUs. So I want to parallelize all 4 models and I wonder if it is possible. I have thought about using all modules within a new module but I don’t know if the calculations would be parallel or sequential,</p>
<pre><code class="lang-auto">class combinedModel(nn.Module):
    def __init__(self):
        super(Model, self).__init__()

        self.model1 = Model1()
        self.model2 = Model2()
        self.model3 = Model3()

    def forward(self, x):
        out1 = self.model1(x)
        out2 = self.model2(x)
        out3 = self.model3(x)
        return out1, out2, out3

</code></pre>
<p>Given models don’t have any shared parameters and I am going to perform optimization for each of them seperately. How can I utilize my GPU memory instead of employing each model on seperate GPUs?</p>
Effective Batching with images of different shapes in PyTorch

husnainali21: <p>Say that from an image folder with 9k images I have 4k images of size (100,400) , 2k images of size(150 ,350) and the rest have a size of (200 , 500) …How can I train with such data like how would I define <code>collate_fn</code> to handle such images and effectively create batches of same sized images ? If I preprocess and save 3 hdf5 files for three different sized images how would I use all three in <code>Dateset</code> and <code>collate_fn</code> or is there a way that a single hdf5 file can store all these three subsets of data without using padding ?</p>
Predicting Digital Surface Models (DSM) with one channel

Caipi: <p>Hello community,</p>
<p>I recently build a Unet for predicting Digital Surface Models (DSMs or heightmaps) with a Unet. For this my input has a shape of 3x512x512 for the RGB channels of my satellite image while my ground truth is a DSM with 1x512x512 respectivley. Everything works fine and the losses looking good. But I am wondering of some metrics to measure something like accuracy or dice loss.</p>
<p>The task of course is not really a classic semantic segmentation because the resulting DSM image contains not 1 for segmented and 0 for the rest, it contains the corresponding height of each pixel. Thus it is useless to have an accuracy where you like compare the prediction DSM with the traget one pixel by pixel, because it will almost never have the EXACT height (based of float) isn’t it?</p>
<p>My idea was to have like a 10 percent range for the element wise comparison, but I dont if this can be done very efficiently. But maybe this is a known problem…?</p>
<p>Small example:<br>
Target-heights: [20.5435, 23.6946<br>
19.59    , 22.646 ]</p>
<p>Prediction-heights: [18.9545, 23.5007<br>
20.0153, 26.5432 ]<br>
This is just a made up one, I do not know how close the values actually will get… Also is there a way to write nice matricies in a question?</p>
How to save distributedSampler state for resuming later?

kevin_zhang: <p>Hi,</p>
<p>Let’s say I am using a DistributedSampler for multi-gpu training in the following fashion:</p>
<pre><code class="lang-auto">train_sampler = data.distributed.DistributedSampler(train_dataset,
                                                        num_replicas=hvd.size(),
                                                        rank=hvd.rank())

    train_loader = data.DataLoader(dataset=train_dataset,
                                   batch_size=config.dist_train.batch_size,
                                   shuffle=False,
                                   num_workers=args.data_workers,
                                   pin_memory=True,
                                   drop_last=True,
                                   sampler=train_sampler)
</code></pre>
<p>During training loop, I set epoch for sampler to ensure reproducibility:</p>
<pre><code class="lang-auto">for epoch in range ()...
      train_sampler.set_epoch(epoch - 1)
      for i, _data in enumerate(train_loader):
       ...
</code></pre>
<p>However, if in the middle of epoch 3 I checkpoint the model:</p>
<pre><code class="lang-auto">        torch.save({
            "epoch": epoch,
            "step": i,
            "total_iter": total_iter,
            "model_state_dict": model.state_dict(),
            "optimizer_state_dict": optimizer.state_dict(),
            "scheduler_state_dict": scheduler.state_dict()
        }, CHECKPOINT_PATH)
</code></pre>
<p>Once I resume it, the sampler will iterate over the same data it has previously seen for epoch 3, instead of remember what it has already iterated and starting from the next unseen data. Is there anyway to do this?</p>
<p>Thanks for your help!</p>

carloalbertobarbano: <p>I think that if you fix a seed, and you also restore the last epoch value (and call sampler.set_epoch) you should be good</p>
How to broad value into every GPU?

kellenf: <p>I have broad model in to  every gpu in multi_gpu inference, and I set something in rank=0, as bellow:<br>
if rank==0:<br>
model.cfg = ‘my setting’<br>
Dosomething<br>
(in Dosomething, I get model.cfg but I found only rank=0  is ‘my_setting’, so  how to broad this to all gpus???</p>

carloalbertobarbano: <p>You can use <code>dist.broadcast_object_list</code> which allows you to broadcast pickable objects across all of your workers: <a href="https://pytorch.org/docs/stable/_modules/torch/distributed/distributed_c10d.html#broadcast_object_list" class="inline-onebox" rel="noopener nofollow ugc">torch.distributed.distributed_c10d — PyTorch 1.8.1 documentation</a></p>
Problem in GAN implementation

randinoo: <p>when i tested this code</p>
<pre><code class="lang-auto">

    def forward(self, x):
        x = self.hidden0(x)
        x = self.hidden1(x)
        
        x = self.hidden2(x)
        x = self.out(x)
        return x</code></pre>

ptrblck: <p>Unfortunately you are not explaining any issues in the post, so please try to describe the error as well as the expected behavior, and, if possible, post a minimal code snippet to reproduce the issue.</p>

randinoo: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/b/eb3b9b99f3c77125e0fc523689760054bbad6ac1.png" data-download-href="https://discuss.pytorch.org/uploads/default/eb3b9b99f3c77125e0fc523689760054bbad6ac1" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/b/eb3b9b99f3c77125e0fc523689760054bbad6ac1.png" alt="image" data-base62-sha1="xyXUIavVOpIeHQBvbLMG6SJeJZD" width="690" height="286" data-dominant-color="F6F7FA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">825×343 13.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">with torch.no_grad():
        output1 = model(((train_x.float(),train_x1.float())))     

        softmax1 = torch.exp(output1).cpu()    
        prob1 = list(softmax1.numpy())
        predictions1 = np.argmax(prob1, axis=1)
        print('Validation accuracy train: {:.4f}%'.format(float(accuracy_score(train_y, predictions1)) * 100))
</code></pre>
<p>help me <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

ptrblck: <p>Based on the error message the model expects (at least) two inputs, while you are passing a single <code>tuple</code> to it, so you would need to remove the <code>()</code> around the tensors.</p>
Gradient explode during backward pass

cbd: <p>Data(ranges between -1 to 1) after normalization(ranges between 0 to 1)  passed through customize function which used the “exponential function” with some mathematical operation. During the backward pass, the gradients are exploding which change the weights value such that input values to the customize function goes out of bound(less than 0 and greater than 1) in next iteration. Backward pass is normal but can not figure out the issue.</p>

Kushaj: <p>You can reduce the size of gradients using <a href="https://pytorch.org/docs/stable/generated/torch.nn.utils.clip_grad_norm_.html#torch.nn.utils.clip_grad_norm_" rel="noopener nofollow ugc">`torch.nn.utils.clip_grad_norm</a>.</p>
Benchmarking optimizer timing while training DNNs

Jamesswiz: <p>Hello,<br>
I have a custom Pytorch optimizer for which I would like to compare the average execution time (with SGD/Adam) during each call to optimizer.step() and also for a specific operation (say matrix-matrix multiplication) within my custom optimizer.</p>
<p>What is the best and correct way to do this without the impact of other background processes?<br>
I want to do this while the network trains to see scalability performance of optimizers, where number of trainable network weights (among all initialized weights) increases over epochs.<br>
This needs be done both for CPU and GPU training.</p>
<p>Any suggestions to do a fair comparison would help. Thanks</p>

eqy: <p>I think the <a href="https://pytorch.org/tutorials/recipes/recipes/benchmark.html" rel="noopener nofollow ugc">benchmarking module</a> could be useful for this.</p>

Jamesswiz: <p><a class="mention" href="/u/eqy">@eqy</a> Can you be more specific on how to use it.? I think every call to benchmark.Timer( optimizer.step() ) will do say multiple runs over optimizer.step() for each batch and I don’t want this behaviour because I am training the network too.</p>

eqy: <p>Yes, typical benchmarking would use multiple runs to avoid measuring any startup costs. If you need to run this live, I would see if simply instrumenting your training loop with timing measurements (e.g, even just a crude <code>t1 = time.time() ... t2 = time.tim2()</code> and appropriate <code>torch.cuda.synchronize</code> calls provides enough precision.</p>

Jamesswiz: <p>Thanks for the pointers. Where should I use torch.cuda.synchronize ? Onetime at top while importing packages or within training loop for each batch or epoch?</p>

eqy: <p>It might extra overhead end to end, but the goal is to avoid leaving GPU operations in flight when timing. For example, you might synchronize to ensure that all GPU operations before your operations of interest have finished, and again to ensure that your timing finishes after all GPU operations of interest have completed.</p>
torch.nn.Embedding/torch.nn.LSTMCell and torch.jit

Rafael_Valle: <p>Looks like torch.jit is interpreting nn.Embedding and nn.LSTMCell as python functions. The code and error are below.</p>
<pre><code class="lang-auto">import torch
class Model(torch.jit.ScriptModule):
    def __init__(self):
        super(Model, self).__init__()

        self.embedding = torch.nn.Embedding(128, 512)

    @torch.jit.script_method
    def forward(self, x):
        return self.embedding(x)

m = Model()
m.save('temp.pt')
</code></pre>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "test.py", line 16, in &lt;module&gt;
    m.save('temp.pt')
RuntimeError: Couldn't export Python operator &lt;python_value&gt;

Defined at:
@torch.jit.script_method
def forward(self, x):
    return self.embedding(x)
           ~~~~~~~~~~~~~~ &lt;--- HERE
</code></pre>

jotline: <p>Torch Script supports a subset of the builtin tensor and neural network functions that PyTorch provides. Most methods on Tensor as well as functions in the  <code>torch</code>  namespace, all functions in  <code>torch.nn.functional</code>  and all modules from  <code>torch.nn</code>  are supported in Torch Script, excluding those in the table below. For unsupported modules, we suggest using <a href="https://pytorch.org/docs/master/jit.html#torch.jit.trace" rel="noopener nofollow ugc"> <code>torch.jit.trace()</code> </a>.</p>
<p>Unsupported  <code>torch.nn</code>  Modules</p>
<blockquote>
<p>torch.nn.modules.adaptive.AdaptiveLogSoftmaxWithLoss torch.nn.modules.normalization.CrossMapLRN2d torch.nn.modules.fold.Fold torch.nn.modules.fold.Unfold torch.nn.modules.rnn.GRU torch.nn.modules.rnn.LSTM torch.nn.modules.rnn.RNN torch.nn.modules.rnn.GRUCell torch.nn.modules.rnn.LSTMCell torch.nn.modules.rnn.RNNCell</p>
</blockquote>

luvwinnie: <p><a class="mention" href="/u/jotline">@jotline</a> hello i face the same problem that said that the LSTMCell is not supported.</p>
<p>That means I can’t convert to torchscript model if I have this LSTMCell module?</p>
<p>I’m trying to create a variable length OCR model.</p>
Reset the model parameters but the model stop to train

kavka: <p>Lets take the example from tutorial for example, suppose I train the model as following</p>
<pre><code class="lang-auto">import torch
import math
x = torch.linspace(-math.pi, math.pi, 2000)
y = torch.sin(x)
p = torch.tensor([1, 2, 3])
xx = x.unsqueeze(-1).pow(p)

model = torch.nn.Sequential(
    torch.nn.Linear(3, 1),
    torch.nn.Flatten(0, 1)
)

loss_fn = torch.nn.MSELoss(reduction='sum')

optimizer = torch.optim.RMSprop(model.parameters(), lr=1e-3)
for t in range(2000):
    y_pred = model(xx)
    loss = loss_fn(y_pred, y)
    if t % 100 == 99:
        print(t, loss.item())
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
....
99 18966.865234375
199 7549.88232421875
299 2734.0283203125
399 1074.5540771484375
499 667.1275024414062
599 529.754638671875
</code></pre>
<p>Now if I reset the model parameters like below, and retrain the model, the model seems stop trainning itself.</p>
<pre><code class="lang-auto">def weight_init(m):
    if isinstance(m, nn.Linear):
        init.xavier_normal_(m.weight.data, gain=1.0)
        if m.bias is not None:
            init.zeros_(m.bias)
model.apply(weight_init)

optimizer = torch.optim.RMSprop(model.parameters(), lr=1e-3)

for t in range(2000):

    y_pred = model(xx)

    loss = loss_fn(y_pred, y)

    if t % 100 == 99:

        print(t, loss.item())

    optimizer.zero_grad()

    loss.backward()

...
99 46380.5
199 46380.5
299 46380.5
399 46380.5
499 46380.5
599 46380.5
699 46380.5
799 46380.5
899 46380.5
999 46380.5
1099 46380.5
...
</code></pre>
<p>Can anyone help me to explain these?</p>

eqy: <p>Just to double check, did you simply omit the <code>optimizer.step()</code> from the second code snippet when it is in fact in the actual code?</p>
Parallelize along samples of batch

limes1: <p>I need to parallelize a model along the samples of a batch on CPU. The reason for that is that my model is a recursive network so I don’t have tensors which I can stack into a batch but tree structures I have to unfold during training.<br>
Basically, the code I want to achieve looks like this.</p>
<pre><code class="lang-auto">for b in batches:
  losses = 0  # needs to be synchronized among processes
  for sample in batch:  # sample is a recursive structure, not a tensor
     # do this in parallel on batch_size-many CPU cores
     loss += model(sample)
  # sum/average all losses from the subprocesses
  loss /= num_processes
  loss.backward()
  optimizer.step()
</code></pre>
<p>Basically, it should work like a multiprocessing.Pool. However, I cannot use this as gradients cannot be shared along processes. I know that DistributedDataParallel exists, but I don’t see how I can use this to achieve my goal.</p>
<p>Thanks!</p>

carloalbertobarbano: <p>I’m not sure I understand, but couldn’t you just parallelize at the batch-level? (every process would get a subset of batches).</p>

limes1: <p>How can I parallelize at the batch-level? Maybe I could set the batch size to 1 and do a gradient update every 16 batches, or so…</p>

carloalbertobarbano: <p>If <i>batch</i> is not a tensor and you are forced to iterate over it anyways, then yes I guess that setting the batch size to 1 with DDP could do the trick</p>

limes1: <p>Could you please show some code here? I don’t see how to do it.</p>

carloalbertobarbano: <p>You would probably need to make your own custom data loader to handle data structures other than tensor, but the general scheme could be something like this?</p>
<pre><code class="lang-python">model = DDP(model, ...)
sampler = DistributedSampler(dataset)
dataloader = CustomDataLoader(dataset, shuffle=False, sampler=train_sampler, batch_size=1)

for sample in dataloader:
  loss = model(sample)
  optimizer.zero_grad()
  loss.backward() #this is synchronized automatically
  optimizer.step()
</code></pre>

limes1: <p>Thank you. I implemented this approach. But when I set batch_size=1, it performs a gradient update after every sample, right? That is not what I want, I want larger batches. Is there a way to achieve this?<br>
When I set batch_size=16, every process gets all samples of this batch. Maybe I misunderstood DDP, but shouldn’t it split the batch into subbatches &lt; 16 and distribute these to the workers? The batches are python lists in my case.</p>

carloalbertobarbano: <p>When using DDP you are “simulating” a total batch size equal to batch_size*world_size. If you set batch size to 1 and spawn 16 workers (while using the DistributedSampler) you are synchronizing gradients from 16 different samples at the same time.</p>
<p>For more detail about DDP refer to <a href="https://pytorch.org/docs/master/notes/ddp.html" class="inline-onebox" rel="noopener nofollow ugc">Distributed Data Parallel — PyTorch master documentation</a></p>
How to unfold 3D tensor?

melike: <p>Hi all, I have an input tensor of shape <code>12x3x3</code> which corresponds to 12 patches of size <code>3x3</code>, can be explained as {Patch1, Patch2, …, Patch12}. I want to convert it to <code>1x12x9</code>, keeping the patches as described below,</p>
<p>Patch1 Patch2 Patch3<br>
Patch4 Patch5 Patch6<br>
Patch7 Patch8 Patch9<br>
Patch10 Patch11 Patch12</p>
<p>I tried <code>torch.unfold()</code> with different parameters, but couldn’t find the right way. Any ideas?</p>

tom: <p>How about <code>inp.view(4, 3, 3, 3).permute(0, 2, 1, 3).reshape(1, 12, 9)</code>?</p>
<p>(If anything, unfold would be used when getting into the other direction with overlapping patches.)</p>

melike: <p>Yes, that’s it, thank you. Much better than my desperate attempts. And for a different patch size <code>ps</code>, it should be,</p>
<pre><code class="lang-auto">inp.view(4, 3, ps, ps).permute(0, 2, 1, 3).reshape(1, 4 * ps, 3 * ps)
</code></pre>
N choose k function?

mhamilton723: <p>Hey, looking for the n choose k function similar to scipy.special.comb any ideas where I can find it? Thanks!</p>

tom: <p>While there is torch.special in PyTorch 1.9, it is not yet very complete.<br>
I would suggest to use the scipy function or implement it directly. Note that PyTorch integer tensors are 64 bits by default, things can get large quickly with these factorials.</p>

KFrank: <p>Hi Mark!</p>
<aside class="quote no-group" data-username="mhamilton723" data-post="1" data-topic="121974" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mhamilton723/48/38578_2.png" class="avatar"> mhamilton723:</div>
<blockquote>
<p>looking for the n choose k function similar to scipy.special.comb any ideas where I can find it?</p>
</blockquote>
</aside>
<p>Assuming that the scipy function computes the conventional binomial<br>
coefficient, you may use:</p>
<pre data-code-wrap="python"><code class="lang-python">((n + 1).lgamma() - (k + 1).lgamma() - ((n - k) + 1).lgamma()).exp()
</code></pre>
<p>Binomial coefficients may be computed in terms of factorials, the<br>
gamma function is the factorial function with its argument shifted<br>
by one, and pytorch implements an autograd-aware log-gamma<br>
function as <code>torch.lgamma()</code>.</p>
<p>So you can use <code>lgamma()</code> to build a binomial-coefficient function<br>
(that is differentiable and autograd aware):</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.7.1'
&gt;&gt;&gt; def combo (n, k):
...     return ((n + 1).lgamma() - (k + 1).lgamma() - ((n - k) + 1).lgamma()).exp()
...
&gt;&gt;&gt; tfive = torch.tensor ([5.0], requires_grad = True)
&gt;&gt;&gt; tthree = torch.tensor ([3.0], requires_grad = True)
&gt;&gt;&gt; c = combo (tfive, tthree)
&gt;&gt;&gt; c
tensor([10.], grad_fn=&lt;ExpBackward&gt;)
&gt;&gt;&gt; c.backward()
&gt;&gt;&gt; tfive.grad
tensor([7.8333])
&gt;&gt;&gt; tthree.grad
tensor([-3.3333])
</code></pre>
<p>Note that the binomial-coefficient function grows exponentially (as<br>
does the factorial function), so in many situations is better to work<br>
with logarithms.  If that fits your use case, just drop the trailing <code>exp()</code><br>
in the definition of <code>combo()</code>.</p>
<p>(The beta function is more or less the binomial-coefficient function<br>
so it would be nice if pytorch implemented a <code>lbeta()</code> function, but<br>
it’s hardly necessary.)</p>
<p>Best.</p>
<p>K. Frank</p>
Train() and eval() for BatchNorm and Dropout

jzy95310: <p>Hi all,</p>
<p>For some purpose, I want to use the eval() mode for BatchNorm layers and train() mode for Dropout layers during training. Does anyone know how to do this? If I simply call model.train() or model.eval(), both of them will be in train mode or evaluation mode.</p>

tom: <p>You can call train/eval on individual (sub-) modules, e.g.</p>
<pre><code class="lang-auto">model.train()
for m in model.modules():
    if isinstance(m, torch.nn.BatchNorm2d):
      m.eval()
</code></pre>
<p>should do the trick.<br>
Note that the weight and bias of batch norm will still require gradients and be trained. You have to set them to not require gradients if you want to avoid that.</p>
<p>Best regards</p>
<p>Thomas</p>

jzy95310: <p>Hi Tom,</p>
<p>Thank you so much for helping! I really appreciate it!</p>
<p>Best,<br>
Ziyang</p>
Reshape torch.distribution.Distribution batch_shape

tjoseph: <p>Hello everyone,<br>
is there a general way to reshape the <code>batch_shape</code> of a <code>torch.distributions.Distribution</code> (<a href="https://pytorch.org/docs/stable/distributions.html#torch.distributions.distribution.Distribution.batch_shape" class="inline-onebox" rel="noopener nofollow ugc">Probability distributions - torch.distributions — PyTorch 1.8.1 documentation</a>)?</p>
<p>Similar to <code>expand</code> (<a href="https://pytorch.org/docs/stable/distributions.html#torch.distributions.distribution.Distribution.expand" class="inline-onebox" rel="noopener nofollow ugc">Probability distributions - torch.distributions — PyTorch 1.8.1 documentation</a>) I would expect a <code>reshape</code> method, but I cannot find one.</p>
<p>Best,<br>
Tim</p>
Result of torch.profiler: inconsistent result in Self CUDA % and inability to remove ProfilerStep* row

Oktai15: <p>Let’s try to use <code>torch.profiler</code> for ResNet18:</p>
<pre><code class="lang-auto">import torch
import torch.profiler
import torchvision

rn18 = torchvision.models.resnet18().to("cuda:0")
rn18_params = {"x": torch.randn(16, 3, 224, 224).to("cuda:0")}

def run_model_profiler(callback, dict_params, wait=1, warmup=2, active=5, row_limit=15):
    with torch.profiler.profile(
            schedule=torch.profiler.schedule(wait=wait, warmup=warmup, active=active),
            on_trace_ready=None,
            profile_memory=False,
            record_shapes=False,
            with_stack=False,
            with_flops=True,
    ) as prof:
        for _ in range(wait + warmup + active):
            _ = callback(**dict_params)
            prof.step()

    print(prof.key_averages().table(sort_by="self_cuda_time_total", row_limit=row_limit))

run_model_profiler(rn18.forward, rn18_params)
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/5/85aa423418fe3ed2683dd75b66d455eec86dddd3.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/85aa423418fe3ed2683dd75b66d455eec86dddd3" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/5/85aa423418fe3ed2683dd75b66d455eec86dddd3_2_690x270.jpeg" alt="image" data-base62-sha1="j4shpZaoeTFAhP825RJibE2B1l1" width="690" height="270" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/5/85aa423418fe3ed2683dd75b66d455eec86dddd3_2_690x270.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/8/5/85aa423418fe3ed2683dd75b66d455eec86dddd3_2_1035x405.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/5/85aa423418fe3ed2683dd75b66d455eec86dddd3_2_1380x540.jpeg 2x" data-dominant-color="272727"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2868×1126 399 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<ol>
<li>Why sum of all values in <code>Self CUDA %</code> doesn’t equal to 100? It’s much more than 100.</li>
<li>How to remove <code>ProfilerStep*</code> from result? I don’t want so that result contains it and affects percent stats.</li>
</ol>