akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Calculating evaluation metrics!

username: Bar
cooked: <p>am I missing something? Because even using the ResNet50 model the accuracy is not improving.</p>
<pre><code class="lang-auto">def evaluate(model, iterator, criterion, device):
    epoch_loss = 0
    epoch_acc_1 = 0
    total = 0
    model.eval()
    with torch.no_grad():
        for (x, y) in iterator:
            x = x.to(device)
            y = y.to(device)
            y_pred = model(x)
            loss = criterion(y_pred, y)
            _, predicted = torch.max(y_pred.data, 1)
            total += y.size(0)
            epoch_acc_1 += (predicted == y).float().sum()
            epoch_loss += loss.item()       
    epoch_loss /= len(iterator)
    epoch_acc_1 /= total   
    return epoch_loss, epoch_acc_1

def epoch_time(start_time, end_time):
    elapsed_time = end_time - start_time
    elapsed_mins = int(elapsed_time / 60)
    elapsed_secs = int(elapsed_time - (elapsed_mins * 60))
    return elapsed_mins, elapsed_secs
</code></pre>
Can I make this tensor sliding faster?

username: seyeeet
cooked: <p>Hello<br>
I am doing something like this in my code but it is very slow.<br>
I was wondering if there is a faster way to do it.<br>
in my example <code>shift</code> and <code>ndivides </code> are subject to change from case to case.</p>
<pre><code class="lang-auto">x = torch.arange(0,32*400).reshape(1,32,400)
cat = []
shift = 4
ndivides = 99
for i in range(ndivides):
    cat.append(x[:,:, i*shift:(i+1)*shift +(shift //2)*2])
results = torch.stack(cat,dim=1)
</code></pre>
<p>sometimes if I change shift or ndivides, it wont work, but that is not an issue, the main issue is that the loop is too slow, unfortunately.</p>

username: ptrblck
cooked: <p><code>tensor.unfold</code> should work:</p>
<pre><code class="lang-python">x = torch.arange(0,32*400).reshape(1,32,400)
cat = []
shift = 4
ndivides = 99
for i in range(ndivides):
    cat.append(x[:,:, i*shift:(i+1)*shift +(shift //2)*2])
results = torch.stack(cat,dim=1)


tmp = x.unfold(2, 8, 4).permute(0, 2, 1, 3)
print((tmp == results).all())
&gt; tensor(True)
</code></pre>

username: seyeeet
cooked: <p>Thank you ptrblk.<br>
in the unfold version, if I want to make it general with respect to shift, how can I do it?<br>
should it be <code>x.unfold(2, 8, shift ).permute(0, 2, 1, 3)</code>?</p>

username: ptrblck
cooked: <p>Yes, this should yield the same results as your previous code once you adapt the <code>ndivides</code> value for the new <code>shift</code>:</p>
<pre><code class="lang-python">tmp = x.unfold(2, shift+(shift//2)*2, shift).permute(0, 2, 1, 3)
</code></pre>
"phantom" process with zero GPU usage

username: Jiaji_Huang
cooked: <p>I am running a Distributed Data Parallel (DDP) job on a single node. The node has 10 GPUs, but I’m using only 4 of them. The cluster is powered by slurm, and I use <code>sbatch exp.sh</code> to submit the job, where the exp.sh is as follows:</p>
<pre><code class="lang-auto">#!/bin/bash
#SBATCH --partition=$MY_PARTITION
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=4
#SBATCH --gres=gpu:4
#SBATCH --cpus-per-task=2

srun python main.py
</code></pre>
<p>In the main.py (and other .py files it calls), I have set necessary environmental variables and created communication between workers:</p>
<pre><code class="lang-auto">os.environ['LOCAL_RANK'] = os.environ['SLURM_LOCALID']
os.environ['RANK'] = os.environ['SLURM_PROCID']
os.environ['WORLD_SIZE'] = os.environ['SLURM_NTASKS']
os.environ['MASTER_ADDR'] = my_host_name
os.environ['MASTER_PORT'] = '32767'
torch.distributed.init_process_group(backend='nccl')
</code></pre>
<p>The job seems to run smoothly. Then I login to that node, and type nvidia-smi. I get<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/e/ee10230a690e9e0b9f4ece0f836ae957b519a21e.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/ee10230a690e9e0b9f4ece0f836ae957b519a21e" title="Screen Shot 2021-11-17 at 2.07.56 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee10230a690e9e0b9f4ece0f836ae957b519a21e_2_271x500.jpeg" alt="Screen Shot 2021-11-17 at 2.07.56 PM" data-base62-sha1="xY0d5lkZSzJXt2Gg4EvdBIWvPyK" width="271" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee10230a690e9e0b9f4ece0f836ae957b519a21e_2_271x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee10230a690e9e0b9f4ece0f836ae957b519a21e_2_406x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee10230a690e9e0b9f4ece0f836ae957b519a21e_2_542x1000.jpeg 2x" data-dominant-color="191919"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-11-17 at 2.07.56 PM</span><span class="informations">962×1770 195 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Please ignore GPU 4-9, as the job is not using them at all. However, on GPU 0-3, there are 16 processes, with 4 for each GPU. 3 out of these 4 are 0 usage! Instead, I think there should be 1 process for each GPU, so a total of 4 processes.</p>
<p>I also tried on other nodes, some with better GPUs and network connections. The above observation is not reproducible. Sometimes, I do get 1 process for each GPU instead.</p>
<p>Can anyone give some insights on what’s happenning?</p>
Repeat_interleave without cloning

username: ZimoNitrome
cooked: <p>Is there an equivalent method to <code>torch.repeat_interleave()</code> that does not copy the tensor?</p>
<p>Or operations that will achieve the same result?</p>

username: ZimoNitrome
cooked: <p>Bumping this topic because I still haven’t found any answer. Performing this operation is by far the most time consuming step in my training script.</p>

username: ZimoNitrome
cooked: <p>I figured it out:</p>
<pre><code class="lang-python">h, w = 4, 10
repeats = 2

my_tensor = torch.randint(0, 10, [h, w])
print("my_tensor:", my_tensor)

repeated_1 = my_tensor.repeat_interleave(repeats , -2)
print("repeated_1:", repeated_1)

repeated_2 = my_tensor.view(h, 1, w).expand(h, repeats, w).contiguous().view(-1, w)
print("repeated_2:", repeated_2)
</code></pre>
<pre><code class="lang-python">my_tensor:
tensor([[7, 6, 8, 5, 5, 2, 2, 1, 2, 8],
        [4, 0, 3, 2, 8, 4, 5, 9, 3, 1],
        [3, 3, 3, 1, 3, 3, 2, 4, 6, 5],
        [0, 4, 8, 7, 7, 7, 3, 4, 6, 9]])

repeated_1:
tensor([[7, 6, 8, 5, 5, 2, 2, 1, 2, 8],
        [7, 6, 8, 5, 5, 2, 2, 1, 2, 8],
        [4, 0, 3, 2, 8, 4, 5, 9, 3, 1],
        [4, 0, 3, 2, 8, 4, 5, 9, 3, 1],
        [3, 3, 3, 1, 3, 3, 2, 4, 6, 5],
        [3, 3, 3, 1, 3, 3, 2, 4, 6, 5],
        [0, 4, 8, 7, 7, 7, 3, 4, 6, 9],
        [0, 4, 8, 7, 7, 7, 3, 4, 6, 9]])

repeated_2:
tensor([[7, 6, 8, 5, 5, 2, 2, 1, 2, 8],
        [7, 6, 8, 5, 5, 2, 2, 1, 2, 8],
        [4, 0, 3, 2, 8, 4, 5, 9, 3, 1],
        [4, 0, 3, 2, 8, 4, 5, 9, 3, 1],
        [3, 3, 3, 1, 3, 3, 2, 4, 6, 5],
        [3, 3, 3, 1, 3, 3, 2, 4, 6, 5],
        [0, 4, 8, 7, 7, 7, 3, 4, 6, 9],
        [0, 4, 8, 7, 7, 7, 3, 4, 6, 9]])
</code></pre>
<p>I had to use <code>contiguous()</code>. Having not used it before, I expected the time to be similar to just using <code>repeat_interleave()</code>.</p>
<p>And… it is weird… timing these two operations gives me similar times but in my training script, the latter is multiple times faster… maybe due to <code>torch.backends.cudnn.benchmark = True</code>?</p>
<p>Anyway, I am happy with the results <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Multiprocessing total loss and best practices

username: Amit_Gupta
cooked: <p>I am trying to run simple regression example with pytorch multiprocessing. I am following the example here: <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html" class="inline-onebox" rel="noopener nofollow ugc">Multiprocessing best practices — PyTorch 1.10.0 documentation</a></p>
<p>However few things are unclear to me. In the example page it is written:</p>
<pre><code class="lang-auto">    for data, labels in data_loader:
        optimizer.zero_grad()
        loss_fn(model(data), labels).backward()
        optimizer.step()  # This will update the shared parameters
</code></pre>
<ol>
<li>What is meant by update the shared parameters?</li>
<li>Do they update on individual processes? or as the model is shared (<code>model.shared_memory()</code> achieves that right?) it updates the shared copy of parameters?</li>
<li>By default model parameters must be shared right? as it says in the note <code>If torch.Tensor.grad is not None, it is also shared.</code>
</li>
<li>If it updates the shared copy, then is the backwards call calculating loss over single thread or sum over losses from all threads? If it is total loss then how can I print it?</li>
<li>If the model contain a layers which has conditional branches, how will optimizer update parameters?</li>
<li>Each process saves its own backward tree? If yes then how are parameters updated?</li>
</ol>
<p>Sorry for the question dump but  couldn’t find proper answers as tutorials and documentation on this was rather scarce.</p>

username: rvarm1
cooked: <p>Regarding 1 &amp; 2, yes the shared parameters will be updated. With the call to <code>model.share_memory</code> parameter tensors are put into SHM, so each process is actually operating on the same model.</p>
<ol start="3">
<li>The wording is probably not the best, but that part means the <code>grad</code> tensor is also shared as long as <code>grad</code> is not none. If you’ve called <code>model.shared_memory</code> as example does, model params will always be shared.</li>
<li>There aren’t any application-level threads here, <code>loss</code> is computed per-process so it can be different across each process. So it is not total loss.</li>
<li>If the model has conditional branches, autograd automatically handles this by constructing the backward graph on the fly during forward pass.</li>
<li>Backward is run locally on each process, there is no inter-process coordination. The .grad field is updated in a shared way across each process, this can lead to some processes stepping on each other’s updates, but this is a result of hogwild training.</li>
</ol>
How are values nromalized during .train using a batchnorm layer

username: JanoschMenke
cooked: <p>So to be more precise, I would like to know how during training the normalized values are calculated.<br>
Below I have a code snippet which I will be referring to from now on.</p>
<p>If I run the following code, it will always return for the first column:<br>
<code>[-1.2247, 0.0000,  1.2247]</code> which I do not understand. The “right” normalization should be [-1.,0.,1.].<br>
When I use <code>bn.eval()</code>, then the output is correct.</p>
<p>What I would like to know is, what term am I missing during <code>.train()</code> that makes the values so different?</p>
<pre><code class="lang-auto">x = torch.tensor([[1.,2.,3.,4.,5.],
                           [2.,2.,2.,2.,2.],
                         [3.,3.,3.,3.,3.]])
bn = nn.BatchNorm1d(5)

[bn(x) for i in range(1000)]
</code></pre>

username: my3bikaht
cooked: <p>Track these values through the loop</p>
<pre><code class="lang-auto">torch.mean(x, axis=0)
bn.running_mean
bn.momentum
torch.var(x, axis=0)
bn.running_var
</code></pre>

username: JanoschMenke
cooked: <p>Maybe I did not explain my issue not well enough.<br>
In the following, I am normalizing the values as its explained in the documentation.<br>
However, I am unable to recreate the actual values obtained when applying <code>bn</code> to <code>beispiel_x</code></p>
<pre><code class="lang-auto">beispiel_x = torch.tensor([[1.,2.,3.,4.,5.],
                           [2.,2.,2.,2.,2.],
                         [3.,3.,3.,3.,3.]])
bn = nn.BatchNorm1d(5)

print((beispiel_x -torch.mean(beispiel_x, axis=0))/torch.sqrt(torch.var(beispiel_x, axis=0)+bn.eps))
print((beispiel_x -bn.running_mean)/torch.sqrt(bn.running_var+bn.eps))
bn(beispiel_x)
</code></pre>

username: mMagmer
cooked: <p><a href="https://discuss.pytorch.org/u/janoschmenke">@JanoschMenke</a> torch BatchNorm1d doesn’t use unbiased estimator for variance.<br>
And running_mean and running_var only apply when model is in eval mode.<br>
Also, <a href="https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm1d.html" rel="noopener nofollow ugc">BatchNorm1d</a> module has learnable affine parameters. That you should consider when computing output. But it initializes with 0 and 1 for bias and gamma.<br>
Running averages are changing as you forward the input through the model.</p>
<pre><code class="lang-auto">print((beispiel_x -torch.mean(beispiel_x, axis=0))/torch.sqrt(torch.var(beispiel_x, axis=0,unbiased=False)+bn.eps))
</code></pre>
<pre><code class="lang-auto">&gt;&gt;&gt; 
tensor([[-1.2247, -0.7071,  0.7071,  1.2247,  1.3363],
        [ 0.0000, -0.7071, -1.4142, -1.2247, -1.0690],
        [ 1.2247,  1.4142,  0.7071,  0.0000, -0.2673]])
</code></pre>

username: JanoschMenke
cooked: <p>Oh yes you are right I overread this part</p>
Initialising nn.Module from tensor keeping gradient flow

username: Mlxa
cooked: <p>I want to initialize weights in, for example, nn.Linear from given tensor. Then I will get gradients in nn.Linear and I want them to copy automatically to the first tensor. Code example:</p>
<pre><code class="lang-auto">module = nn.Linear(10, 1)
W = torch.rand((1, 10), dtype=float, requires_grad=True)
# here something like module.weight = W
out = module(torch.rand(10))
out.backward()
assert W.grad is not None
</code></pre>
<p>d)</p>
TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found object, but the output is torch.Tensor

username: luiejunior
cooked: <p>Hi everyone, I’m trying to do some classification for the CelebA dataset, but I’m encountering an error with trying to load my data.</p>
<p>The code is:</p>
<pre><code class="lang-auto">class MultiClassCelebA(Dataset):
    
    def __init__(self, dataframe, folder_dir, transform=None, target_transform=None):
        
        self.dataframe = dataframe
        self.folder_dir = folder_dir
        self.transform = transform
        self.target_transform = target_transform
        self.file_names = dataframe.index
        self.labels = dataframe.labels.values.tolist()
        
    def __len__(self):
        return len(self.dataframe)
    
    def __getitem__(self, index):
        
        image = Image.open(os.path.join(self.folder_dir, self.file_names[index]))
        label = self.labels[index][0]
        if self.transform is not None:
            image = self.transform(image)
        if self.target_transform is not None:
            label = self.target_transform(label)
        return image, label

tfms = transforms.Compose([transforms.Resize((256, 256)),
                           transforms.PILToTensor()])

train_dl = MultiClassCelebA(train_df, celeb_path + '/train/', transform=tfms)
val_dl = MultiClassCelebA(val_df, celeb_path + '/val/', transform=tfms)

train_dataloader = DataLoader(train_dl, shuffle = True, batch_size = 32)
val_dataloader = DataLoader(val_dl, shuffle = True, batch_size = 32)
next(iter(train_dataloader))
</code></pre>
<p>but I’m getting the above TypeError from the last line. I’ve checked the output of the dataset, e.g. for index 2</p>
<pre><code class="lang-auto">type(train_dl[2][0])
</code></pre>
<p>and it confirms that it is a torch.Tensor. Why is the dataloader seeing it as an Object and how can I fix this? Thanks</p>

username: ptrblck
cooked: <p>I’m unsure what might be causing the issue. Creating a random <code>PIL.Image</code> inside the <code>Dataset</code> works fine:</p>
<pre><code class="lang-python">class MultiClassCelebA(Dataset):
    def __init__(self, dataframe, folder_dir, transform=None, target_transform=None):
        self.transform = transform
        self.target_transform = target_transform
        
    def __len__(self):
        return 10
    
    def __getitem__(self, index):
        image = transforms.ToPILImage()(torch.randn(3, 224, 224))
        label = torch.randint(0, 10, (1,))
        if self.transform is not None:
            image = self.transform(image)
        if self.target_transform is not None:
            label = self.target_transform(label)
        return image, label

tfms = transforms.Compose([transforms.Resize((256, 256)),
                           transforms.PILToTensor()])

train_dl = MultiClassCelebA(None, None, transform=tfms)

train_dataloader = DataLoader(train_dl, shuffle = True, batch_size = 32)
next(iter(train_dataloader))
</code></pre>

username: luiejunior
cooked: <p>Fixed the issue in <strong>getitem</strong> by <code>return image, label.float()</code></p>
Regarding Conv2d input dimensions and weight dimensions

username: grndng
cooked: <p>Hey there - I’m currently trying to implement a UNET and have written the model, data loader and train loop. When I try to run the train loop, I get the following error message:</p>
<pre><code class="lang-auto">RuntimeError: Expected 4-dimensional input for 4-dimensional weight [64, 1, 3, 3], but got 3-dimensional input of size [1, 572, 572] instead
</code></pre>
<p>I’ve searched for solutions here as well as the rest of the searchable internet but can’t wrap my mind around it/can’t find my mistake. Apparently this happens when your model at some point expects a specific dimensionality which it does not receive (initially or from the previous layer).</p>
<p>I’m not too “fluent” when it comes to PyTorch so I’m not sure what information you would need, but I’ll take a shot. This is what my DataLoader for my training data looks like:</p>
<pre><code class="lang-python">train_loader = DataLoader(dataset=train_set, shuffle=False, batch_size=1, num_workers=1, pin_memory=True)
</code></pre>
<p>If I follow the <code>Traceback</code>, it leads me to this portion of my model:</p>
<pre><code class="lang-python">class UNET(nn.Module):
    def __init__(self):
        super(UNET, self).__init__()
        self.max_pool_2x2 = nn.MaxPool2d(kernel_size = 2, stride = 2)
        self.downwards_01 = double_conv(1,64) # this is the culprit, the first line of my forward()-function
</code></pre>
<p><code>double_conv</code> is a function since I’ll be using that a lot for my UNET to decode and encode. That function looks like this:</p>
<pre><code class="lang-python">def double_conv(input, output):
    conv = nn.Sequential(
        nn.Conv2d(input, output, kernel_size=3),
        nn.ReLU(inplace=True),
        nn.Conv2d(output, output, padding=0, kernel_size=3, stride=1),
        nn.ReLU(inplace=True),
    )
    return conv
</code></pre>
<p>Apparently there is something wrong with my <code>Conv2d</code>. What I understand from the error message is that pytorch expects a tensor with four dimensions while my script is returning [1, 572, 572]. If I understand the manual correctly, <code>Conv2d</code> expects four dimensions as <code>[batchsize, channels, heigth, width]</code>. So for some reason either my batchsize or my channels aren’t being taken care of and I can’t figure it out. Can someone help me figure out my mistake?</p>

username: my3bikaht
cooked: <p>Conv2d expects 4 dimensions (batch num of samples, num of channels, h, w)<br>
if you want to pass single tensor, use unsqueeze(0) to add extra dimension in front of your tensor.</p>
CPU Parallelism for Cumulative Loops

username: cw-tan
cooked: <p>Hi, I don’t have any practical experience parallelizing but I was hoping to parallelize the training of my model over multiple CPU cores in a specific way. The part I want to parallelize over is a for loop that contains a cumulative sum of the error. I thought that this was the most effective way to parallelize my code because my model is supposed to predict various high-dimensional quantities from high-dimensional inputs with the inputs and outputs having variable shapes. The code for the error gradient calculation of a single optimizer step would look something like</p>
<pre><code class="lang-auto">error = 0
for training_sample in training_samples:
    predicted_a, predicted_b, predicted_c = model.forward(training_sample.input)
    error += torch.mean((predicted_a - training_sample.expected_a).pow(2)) + 
             torch.mean((predicted_b - training_sample.expected_b).pow(2)) + ...
error.backward()
</code></pre>
<p>For large sets of training samples, I got the impression that it was possible to parallelize this by having each CPU core work on calculating the errors due to a fraction of the training samples before adding them all up to calculate the gradient. Is this possible and advisable as a way to parallelize my problem, and how should I start to go about it? (I put this under the category of “distributed” as it seemed related but I couldn’t find a neat example that covers what I want to achieve)</p>
<p>Thank you!</p>

username: my3bikaht
cooked: <p>Aren’t you bottlenecked by batch size (memory-wise) anyway?</p>

username: cw-tan
cooked: <p>By my estimations about 16 cores from a cluster I have access to is sufficient for the memory requirements. However, I’d rather not request for 16 cores just for the memory - might as well parallelize the training to make the most of the cores, hence the question. Otherwise, I guess I could just train in mini-batches, one mini-batch at a time without parallelism with 2 cores to avoid wasting resources.</p>

username: rvarm1
cooked: <p>This use case doesn’t exactly need distributed training, but you can try <code>torch.jit.fork</code> which is the main way to do task-level parallelism in PyTorch: <a href="https://pytorch.org/docs/stable/generated/torch.jit.fork.html" class="inline-onebox" rel="noopener nofollow ugc">torch.jit.fork — PyTorch 1.10.0 documentation</a>.</p>

username: cw-tan
cooked: <p>Cool, thanks! Let me try that.</p>
Pytorch Lightning : Confusion regarding metric logging

username: Ajinkya_Ambatwar
cooked: <p>Hi, I am a bit confused about metric logging in <code>training_step</code>/<code>validation_step</code>.<br>
Now a standard <code>training_step</code> is</p>
<pre><code class="lang-auto">def training_step(self, batch, batch_idx):
        labels=&lt;from somewhere&gt;
        logits = self.forward(batch)
        loss = F.cross_entropy(logits, labels)
        with torch.no_grad():
            correct = (torch.argmax(logits, dim=1) == labels).sum()
            total = len(labels)
            acc = (torch.argmax(logits, dim=1) == labels).float().mean()

        log = dict(train_loss=loss, train_acc=acc, correct=correct, total=total)
        self.log("train_loss", loss, on_step=True, on_epoch=True, prog_bar=True)
        self.log("train_acc", acc, on_step=True)
        return dict(loss=loss, log=log)
</code></pre>
<p>Now here my doubt is, the <code>train_loss</code> that I am logging here, is it the train loss for this particular batch or averaged loos over entire epoch. Now as I have set, <code>on_step</code> and <code>on_epoch</code> both True, what is actually getting logged and when(after each batch or at the epoch end)?</p>
<p>About <code>training_acc</code>, when I have set <code>on_step</code> to True, does it only log the per batch accuracy during training and not the overall epoch accuracy?</p>
<p>Now with this <code>training_step</code>, if I add a custom <code>training_epoch_end</code> like this</p>
<pre><code class="lang-auto">def training_epoch_end(self, outputs) -&gt; None:
        correct = 0
        total = 0
        for o in outputs:
            correct += o["log"]["correct"]
            total += o["log"]["total"]
        self.log("train_epoch_acc", correct / total)
</code></pre>
<p>Is the <code>train_epoch_acc</code> here same as the average of per batch <code>training_acc</code>?</p>
<p>I intend to put an <code>EarlyStoppingCallBack</code> with monitoring validation loss of the epoch, defined in a same fashion as for <code>train_loss</code>.<br>
If I just put <code>early_stop_callback = pl.callbacks.EarlyStopping(monitor="val_loss", patience=p)</code> , will it monitor per batch <code>val_loss</code> or epoch wise <code>val_loss</code> as logging for <code>val_loss</code> is happening during batch end and epoch end as well.<br>
Sorry if my questions are a little too silly, but I am confused about this!</p>
<p>Thank you!</p>

username: EvanZ
cooked: <p>Did you ever figure this out? I have a similar question about validation_step and validation_epoch_end.</p>

username: Ajinkya_Ambatwar
cooked: <p>Hi sorry for delayed reply,</p>
<p>You can check <a href="https://github.com/PyTorchLightning/pytorch-lightning/discussions/7722" rel="noopener nofollow ugc">this</a> out</p>
<p>Hope this helps!</p>
Confusion about loss objective with SGD

username: israrbacha
cooked: <p>Hi, I have confusion about optimizing the loss objective with SGD</p>
<ol>
<li>
<p>loss=1-F.cosine_similarity(x,y) and loss= -F.cosine_similarity(x,y), are these two represent to minimize the similarity using SGD? Apparently,  (1-Cosinesim) seems suitable to me  but i have seen the (-CosineSim) in a contrastive Self-Supervise learning objective. So how these two behave differently?</p>
</li>
<li>
<p>whats the difference between : loss = -F.crossentropy(logits, label) and loss=F.crossentropy(-logits, label) ? are these two represents to maximize the cross entropy ?</p>
</li>
</ol>

username: mMagmer
cooked: <ol>
<li>no difference</li>
<li>
<ul>
<li>-F.crossentropy is not bounded and minimizing loss will not converge</li>
<li>-F.crossentropy(logits, label) is not equal to  F.crossentropy(-logits, label)</li>
</ul>
</li>
</ol>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html#torch.nn.CrossEntropyLoss" rel="noopener nofollow ugc"><code>crossentropy</code></a> is equivalent to the combination of <code>LogSoftmax</code> and <code>NLLLoss</code><br>
So <code>logits</code> input first go through <code>LogSoftMax</code>.<br>
And for <code>F.crossentropy(-logits, labels)</code> think it this way: minimizing it will return estimator with minimum cross entropy between <code>softmax(-logits)</code> and <code>lables</code> .</p>
<blockquote>
<p>are these two represents to maximize the cross entropy?</p>
</blockquote>
<p>I don’t think so.</p>
Torch 0.3.1 undefined symbol: nvrtGetProgramLogSize + cuLaunchKernel

username: complexhhs
cooked: <p>Hello guys! I have an issue about torch 0.3.1 import process</p>
<p>Here is the thing.</p>
<p>Local server environment does not support CUDA(only using CPU) in RHEL6.9, glibc2.12 - too old system?! lol<br>
Anyway, to using torch framework, only torch v0.3.1 version supports my local server system.</p>
<p>And mixing nuitka builder to connect another job, program crashed showing below message.</p>
<pre><code class="lang-auto">[myaccount@my system]$ {program.exe}
Traceback (most recent call last):
   File "{myenvironment}/torch/__init__.py" line56, in &lt;module torch&gt;
ImportError: {myenvironment}/torch/_C.so: undefined symbol: nvrtGetProgramLogSize
[myaccount@my system]$
</code></pre>
<p><a href="https://forums.fast.ai/t/torch-undefined-symbol-nvrtcgetprogramlogsize/41037/2" rel="noopener nofollow ugc">Torch forum issue</a> reported similar problem, but I guessed it slightly differ with my situation. Anyway, I tried to search what is the problem in <strong>nvrtGetProgramLogSize</strong></p>
<pre><code class="lang-auto">[myaccount@my system]$ readelf -s {myenvironment}/torch/_C.cpython-36m-x86_64-linux-gnu.so | grep nvrtcGetProgramLogSize
    310: 0000000000000000    0 NOTYPE GLOBAL DEFAULT UND nvrtcGetProgramLogSize
14565: 0000000000000000    0 NOTYPE GLOBAL DEFAULT UND nvrtGetProgrramLogSize
</code></pre>
<p>I guess above terminal message means the linker was not able to link this symbol during pytorch installation.<br>
Is there anyone who knows that solution?</p>
<ul>
<li>update!<br>
So far, I realized that I have to set .basrc file information about LD_PRELOAD options</li>
</ul>
<pre><code class="lang-auto">[myaccount@my system]$ LD_PRELOAD={my_setting_path}/nuitka/{another}/torch/lib/* ./{exe_file}/libnvrtc-56d4825a.so.8.0.61 {exe_file}
</code></pre>
<p>Using LD_PRELOAD options, <strong>nvrtGetProgramLogSize</strong> message is disappeared.<br>
However, another bug message shows.</p>
<pre><code class="lang-auto">[myaccount@my system]$ LD_PRELOAD={torch lib path}/libnvrtc-56d4825a.so.8.0.61 {exe_file}
Traceback (most recent call last):
  File "/home/vinpremkumar/20211115/ALi/nuitka/ali.dist/torch/__init__.py", line 58, in &lt;module torch&gt;
ImportError: /home/vinpremkumar/20211115/ALi/nuitka/ali.dist/torch/_C.so: undefined symbol: cuLaunchKernel
</code></pre>
<p>As you see, <strong>cuLaunchKernel</strong> undefined symbol bug message appears. Any idea?</p>
How to add cosine similarity score in Cross Entropy Loss

username: Kaushal
cooked: <p>Hi, My apologies if my question is very naive. I am new to Pytorch.</p>
<p>I am trying to add cosine similarity score in cross entropy loss such a way that similarity score should be maximise. I am confused between fallowing two codes i.e.,</p>
<p><span class="hashtag">#code</span><br>
loss = criterion(pred, gtruth) + 0.01 * (1 - cosineSim(pred_repj, gtruth_repj))<br>
or<br>
loss = criterion(pred, gtruth) + 0.01 * ( cosineSim(pred_repj, gtruth_repj))<br>
<span class="hashtag">#code</span></p>
<p>Do I need to take care any other aspect if  I am adding similarity score in this way?</p>

username: ptrblck
cooked: <p>Since you would like to maximize the cosine similarity, I would go with the first approach, as in the worst case, you’ll add <code>0.01 * 2</code> to the loss and in the best (trained) case, it will be <code>1 - 1 = 0</code>. Here is a small dummy example of just rotating tensors:</p>
<pre><code class="lang-python">x = torch.tensor([[1., 1.]], requires_grad=True)
y = torch.tensor([[-1, 0.]])

criterion = nn.CosineSimilarity()
optimizer = torch.optim.SGD([x], lr=1.)

for _ in range(20):
    optimizer.zero_grad()
    loss = 1. - criterion(x, y)
    loss.backward()
    optimizer.step()
    print('x: {}, loss: {}'.format(x, loss.item()))
</code></pre>

username: Kaushal
cooked: <p>Thanks lot, It is helpful.</p>
Insert a fixed value into a tensor dimension before every N elements

username: iiSeymour
cooked: <p>What is the best way to insert a single fixed value into a tensor dimension before every <code>N</code> elements?</p>
<p>This is what I’ve come up with so far -</p>
<pre><code class="lang-auto">import torch

T, N, C = (720, 32, 256)

x = torch.empty((T, N, C))
M = 4

value = torch.tensor(24)

x = torch.cat([
    value.expand(T, N, C // M, 1),
    x.reshape(T, N, C // M, M)
], axis=-1).reshape(T, N, -1)


print(x.shape) # torch.Size([720, 32, 320])  
</code></pre>

username: Eta_C
cooked: <pre><code class="lang-python">x = torch.nn.functional.pad(x.view(T, N, C // M, M), (1, 0, 0, 0, 0, 0, 0, 0), value=24).view(T, N, -1)
</code></pre>
<p>returns the same result faster.</p>
Multiprocessing issue

username: hnt4499
cooked: <p>Hi,</p>
<p>I am running DDP for my training task. I observe that occasionally a python exception is raised:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File ".../python3.7/multiprocessing/queues.py", line 242, in _feed
    send_bytes(obj)
</code></pre>
<p>And I observe that this <strong>always</strong> occurs at the end of a <code>for</code> loop (e.g., training <code>for</code> loop or evaluation <code>for</code> loop). Moreoever, this exception does <strong>not</strong> cause the process to terminate.</p>
<p>May I know what is the root cause of the issue? Does it affect my training?</p>

username: pritamdamania87
cooked: <p>Could you share the complete traceback? Its not clear what the error might be from the traceback you have provided.</p>

username: hnt4499
cooked: <p>It’s literally the complete traceback:</p>
<pre><code class="lang-auto">2021-07-02 01:58:40,520 - some log
Traceback (most recent call last):
  File ".../lib/python3.7/multiprocessing/queues.py", line 242, in _feed
    send_bytes(obj)
2021-07-02 01:58:40,796 - another log
</code></pre>
<p>As you can see, the process does not terminate.</p>
<p>However, sometimes I see the “full” traceback as follows:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/usr/lib/python3.7/multiprocessing/queues.py", line 242, in _feed
    send_bytes(obj)
  File "/usr/lib/python3.7/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/usr/lib/python3.7/multiprocessing/connection.py", line 404, in _send_bytes
    self._send(header + buf)
  File "/usr/lib/python3.7/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
</code></pre>
<p>This does not cause the process to terminate either.</p>

username: pritamdamania87
cooked: <p>Thanks for providing additional information, although I’m not sure what could be the reason for this traceback. The weird thing is the traceback doesn’t have any application code/ DDP code indicating where this is coming from.</p>
<p>Is it possible to share minimal repro script to see if we I can repro it on my end?</p>

username: fermat97
cooked: <p>Hi <a class="mention" href="/u/hnt4499">@hnt4499</a> did you find the cause or any solution?</p>

username: hnt4499
cooked: <p>Hi <a class="mention" href="/u/fermat97">@fermat97</a> ,</p>
<p>No, I didn’t. Since it doesn’t affect my task, I didn’t bother trying to fix it.</p>

username: Hao_Fu1
cooked: <p><a class="mention" href="/u/hnt4499">@hnt4499</a> got exactly the same error with you. Can some one help with this issue?</p>

username: rvarm1
cooked: <p>Hi <a class="mention" href="/u/hao_fu1">@Hao_Fu1</a> , do you have a repro of the issue?</p>
Different output length on GPUs nn.DataParallel for paraphrasing

username: Muhammad_Naeem
cooked: <p>Hi, I am using nn.DataParallel module to use multiple GPUs for my Pegasus Model.</p>
<p>The problem I am facing is that when DataParallel divides a batch onto multiple GPUs then the length of paraphrases produced by the <code>PegasusForConditionalGeneration</code> on each GPU doesn’t have equal length as length of output depends upon input.<br>
I can not force model to produce a fixed-length output (by truncating longer ones and padding smaller ones) as I don’t want to truncate bigger paraphrases or unnecessarily pad paraphrases to a large number<br>
GPU:0 produces output of 48 length while GPU:1 of 26. Is there a way to solve this problem</p>
<pre><code class="lang-auto">/torch/nn/parallel/comm.py", line 235, in gather
    return torch._C._gather(tensors, dim, destination)
RuntimeError: Input tensor at index 1 has invalid shape [15, 48], but expected [15, 26]
</code></pre>

username: rvarm1
cooked: <p>Hi, this seems like it might be a possible bug with DataParallel, but a repro with some example model and training code would be needed to confirm. If you’re able to get that could you please post an issue to <a href="https://github.com/pytorch/pytorch/issues" class="inline-onebox" rel="noopener nofollow ugc">Issues · pytorch/pytorch · GitHub</a>?</p>
Convert from PyTorch toTensorflow

username: SamirMoustafa
cooked: <p>Dear All,</p>
<p>I would like to ask, Is there a way to convert the PyTorch model to TensorFlow 1.x / 2.x without passing through ONNX? Because the generated graph by ONNX is not optimized, it usually adds <code>cast</code>, <code>reshape</code>, <code>squeeze</code>, and <code>unsqueeze</code> operations for the graph.<br>
My main concern is about converting nested models that have <code>nn.sequential(*[layers])</code> inside their architecture!</p>
<p>Best,<br>
Samir</p>
Nn.functional.interpolate does not seem to work as it meant? or is it?

username: bigbreadguy
cooked: <p>I’m training some convolution-based networks with several datasets I’ve built(the project is open to public btw).</p>
<p>And this few lines in my script keep occurring RuntimeError: dimension does not match</p>
<p>Which can be simplified as below(to save time to browse).</p>
<pre><code class="lang-auto"># forward net
output = net(input_data) # tensor size : (4,4,256,256)

# Build target heatmap from pose labels
target = nn.functional.interpolate(target, (output.size()[2], output.size()[3]), mode="nearest") # tensor size has to be : (4,4,256,256) yet it have been (4,3,256,256)

loss = fn_pose(output, target)
</code></pre>
<p>The loss function squeezes the tensor to the size of (N,C,H*W)<br>
N : batch_size, H : height, W : Width</p>
<p>Then last dimensions of each tensor become output-&gt;(65536), target-&gt;(49152)<br>
65536 is the square number of 256, and 49152 is 256 times 192(3/4 of 256)</p>
<p>How come this happen? I have been worked on this problem few weeks.</p>
<p>For anybody who wants to check the repo : <a href="https://github.com/bigbreadguy/pose-estimator-test-loop" rel="noopener nofollow ugc">bigbreadguy/pose-estimator-test-loop: pytorch implementation of pose estimators for testing along datasets in the project (github.com)</a></p>

username: mMagmer
cooked: <p>I tied the fallowing code in torch1.9 and I can see no error</p>
<pre><code class="lang-auto">
output = torch.randn(4,4,256,256) # tensor size : (4,4,256,256)

target = torch.randn(4,4,256,192)

target = nn.functional.interpolate(target, (output.size()[2], output.size()[3]), mode="nearest")

print(target.shape)

</code></pre>
<p>Output&gt;&gt;&gt; torch.Size([4, 4, 256, 256])</p>
<p>And I don’t know how you expect that target size has to be a tensor with 4 channels.<br>
You can see in dataset definition that <a href="https://github.com/bigbreadguy/pose-estimator-test-loop/blob/917256898f0ac063cdee6b22fc1e0a8207b1aa0a/src/dataset.py#L19" rel="noopener nofollow ugc">hm_shape</a> has only 3 channels.</p>

username: bigbreadguy
cooked: <p>First of all, thanks for helping me.</p>
<p>I forgot to update the thread after I’ve solved the problem. Please forgive me. The problem was not because of nn.funtional.interpolate, it was just typical torchvision.transform mistakes.</p>
<p>I really appreciate you!</p>
TensorRT integration release date

username: mfoglio
cooked: <p>I followed the latest Nvidia conference and they announced an official PyTorch and TensorRT integration that should allow to convert PyTorch model with a single line of code. When will this be available? And what PyTorch version will include this feature?<br>
Thank you</p>

username: Dheeraj_Peri
cooked: <p>Here is the repository : <a href="https://github.com/NVIDIA/Torch-TensorRT" class="inline-onebox" rel="noopener nofollow ugc">GitHub - NVIDIA/Torch-TensorRT: PyTorch/TorchScript compiler for NVIDIA GPUs using TensorRT</a><br>
Release notes : <a href="https://github.com/NVIDIA/Torch-TensorRT/releases/tag/v1.0.0" class="inline-onebox" rel="noopener nofollow ugc">Release Torch-TensorRT v1.0.0 · NVIDIA/Torch-TensorRT · GitHub</a></p>

username: narendasan
cooked: <p><a class="mention" href="/u/dheeraj_peri">@Dheeraj_Peri</a> Linked the repo but to answer the questions:</p>
<blockquote>
<p>When will this be available?</p>
</blockquote>
<ul>
<li>The integration is distributed as a separate python package similar to <code>torchvision</code> or <code>torch-xla</code>. You can download Torch-TensorRT now. The following command will get you the latest release (the version mentioned at GTC)</li>
</ul>
<pre><code class="lang-auto">pip3 install torch-tensorrt -f https://github.com/NVIDIA/Torch-TensorRT/releases
</code></pre>
<blockquote>
<p>And what PyTorch version will include this feature?</p>
</blockquote>
<ul>
<li>The prebuilt wheels target PyTorch version 1.10.0+cu113 (The cuda version is important for TensorRT and builds of Torch-TensorRT are PyTorch version specific).</li>
<li>You can build the project from source for the specific version of PyTorch, CUDA, TensorRT and cuDNN you’d like fairly easily by modifying the WORKSPACE file (<a href="https://github.com/NVIDIA/Torch-TensorRT/blob/master/WORKSPACE" rel="noopener nofollow ugc">https://github.com/NVIDIA/Torch-TensorRT/blob/master/WORKSPACE</a>)
<ul>
<li>To support some older PyTorch versions might require small code changes, post a Github issue and we are happy to help you.</li>
</ul>
</li>
<li>There are older releases that target PyTorch versions back to PyTorch 1.4.0 if you quickly want to try out Torch-TensorRT but we would recommend you try to backport Torch-TensorRT v1.0.0 to an older PyTorch release because of the amount of features that have been added each version.</li>
</ul>
<p>More information can be found here: <a href="https://nvidia.github.io/Torch-TensorRT/index.html" rel="noopener nofollow ugc">https://nvidia.github.io/Torch-TensorRT/index.html</a><br>
Also feel free to tag us if you have other questions.</p>
RuntimeError: expected scalar type Long but found Float for modulo function

username: scubacharlee
cooked: <p>I’m trying to create a model that will learn the function x % 500 from the numbers between 0 - 2000.<br>
Regardless of my approach, which I am planning to experiment with, I am running into a type error that I’ve been stuck on for a while. No combination of <code>to(torch.Long)</code> or <code>type(torch.LongTensor)</code> seems to fix the issue. Any insight is appreciated.</p>
<pre><code class="lang-auto">X = np.arange(0, 2000)
y = X%500

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=42)

X_train = np.array([x // 10 ** np.arange(4) % 10 for x in X_train])
y_train = np.array([y // 10 ** np.arange(4) % 10 for y in y_train])

class Net(nn.Module):
    def __init__(self):
        super(Net,self).__init__()
        self.fc1 = nn.Linear(4, 40)
        self.fc2 = nn.Linear(40, 40)
        self.fc3 = nn.Linear(40, 40)
        self.fc4 = nn.Linear(40, 40)
        self.fc5 = nn.Linear(40, 40)
        self.relu = nn.ReLU()    
        
    def forward(self,x):
        out = self.relu(self.fc1(x))
        out = self.relu(self.fc2(x))
        out = self.relu(self.fc3(x))
        out = self.relu(self.fc4(x))
        out = self.relu(self.fc5(x))
        out = nn.Softmax(out)
        return out

#device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
device = torch.device("cpu")

lr = 0.001 # learning_rate
epochs = 10 # How much to train a model

model = Net().to(device)
model.train()

X_train_tensor = torch.from_numpy(X_train).type(torch.LongTensor)
sample_out = model(X_train_tensor[0])
</code></pre>
<p>It seems that the last line is causing the following error:<br>
<code>RuntimeError: expected scalar type Long but found Float</code></p>

username: ptrblck
cooked: <p>The error message points to the model parameters and expects them to be <code>LongTensor</code>s, since your input is already a <code>LongTensor</code>. Transform the inputs to<code>FloatTensor</code>s and it should work:</p>
<pre><code class="lang-python">X_train_tensor = torch.from_numpy(X_train).float()
</code></pre>
Torch.cuda.amp blocks gradient computation

username: eliaz
cooked: <p>Hi !</p>
<p>I had an issue while using <code>torch.cuda.amp.autocast()</code>. In the context manager I called <code>torch.no_grad()</code> and it blocked gradient computation in the rest of the context manager.</p>
<p>Example on how to reproduce :</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
net = nn.Linear(100, 10).cuda()
input = torch.randn(32, 100, device='cuda')
target = torch.randn(32,10, device='cuda')
crit = nn.MSELoss()

with torch.cuda.amp.autocast():
    with torch.no_grad():
        out = net(input)

    new_out = net(input)
    loss = crit(new_out, target)
</code></pre>
<p>Then <code>loss</code> will not have a <code>grad_fn</code><br>
But if we would not use autocast :</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
net = nn.Linear(100, 10).cuda()
input = torch.randn(32, 100, device='cuda')
target = torch.randn(32,10, device='cuda')
crit = nn.MSELoss()

with torch.cuda.amp.autocast(False):
    with torch.no_grad():
        out = net(input)

    new_out = net(input)
    loss = crit(new_out, target)
</code></pre>
<p>Then <code>loss</code> has a <code>grad_fn</code>.</p>
<p>Is it an expected behavior ?</p>
<p>Thanks for any help !</p>

username: ptrblck
cooked: <p>You would have to exit the autocast context before running the second forward pass as described <a href="https://discuss.pytorch.org/t/autocast-and-torch-no-grad-unexpected-behaviour/93475/3">here</a> due to the internal caching.</p>
<p>CC <a class="mention" href="/u/vadimkantorov">@vadimkantorov</a> as we’ve discussed more general use cases and this one might be important for you.</p>

username: eliaz
cooked: <p>Hi !</p>
<p>Thanks for the quick reply.<br>
Sorry I didn’t see it was a duplicate.</p>
<p>For this particular case how bad is to call <code>net.requires_grad_(False)</code> to avoid computing the activations and then reset it to true ?</p>
<p>Thanks</p>

username: vadimkantorov
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> And I guess this usecase is also an extra reason for also supporting autocast-handling module wrappers <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Tensorboard Cell Shows up Blank

username: Mickey_Damani
cooked: <p>Hi,</p>
<p>This is my first post here so I apologize if I am breaking any rules.<br>
I have this piece of code running in colab trying to initialize an instance of tensor board:</p>
<p>%load_ext tensorboard<br>
%tensorboard --logdir ‘logs’ --port 6006 --host localhost --reload_interval 1</p>
<p>This just produces a blank cell like below:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/3/4378a855da8d1f3829ebb19265c9b663f9049d36.png" data-download-href="https://discuss.pytorch.org/uploads/default/4378a855da8d1f3829ebb19265c9b663f9049d36" title="Screen Shot 2021-11-16 at 3.10.00 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/3/4378a855da8d1f3829ebb19265c9b663f9049d36_2_690x455.png" alt="Screen Shot 2021-11-16 at 3.10.00 PM" data-base62-sha1="9CSuST8Vpbse5QVOPP0KAhFw86O" width="690" height="455" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/3/4378a855da8d1f3829ebb19265c9b663f9049d36_2_690x455.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/3/4378a855da8d1f3829ebb19265c9b663f9049d36_2_1035x682.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/3/4378a855da8d1f3829ebb19265c9b663f9049d36_2_1380x910.png 2x" data-dominant-color="FDFDFD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-11-16 at 3.10.00 PM</span><span class="informations">1822×1204 79.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Here is the code int the file that is supposed to connect to the instance of tb.</p>
<pre><code class="lang-auto">train_logger = tb.SummaryWriter(path.join(args.log_dir, 'train'), flush_secs=1)
valid_logger = tb.SummaryWriter(path.join(args.log_dir, 'valid'), flush_secs=1)
</code></pre>
<p>Then later down in the file:</p>
<pre><code class="lang-auto"> # Log training Accuracy
        train_logger.add_scalar('global accuracy', training_confusion_matrix.global_accuracy, global_step)
        train_logger.add_scalar('average accuracy', training_confusion_matrix.average_accuracy, global_step)
        train_logger.add_scalar('IOU', training_confusion_matrix.iou, global_step)
</code></pre>