strange train_loss
🐛 Bug
Logging train_loss both from return and self.log gives different outputs. They are both also not a running avergae of my manually printed losses - can you please point me to how they are computed? it is not immediately clear from the outputs
    def training_step(self, batch, batch_idx) -> Dict:
        t0 = time.time()
        loss_tensors, logits = self._step(batch)
        train_time = time.time() - t0
        logs = {name: loss for name, loss in zip(self.loss_names, loss_tensors)}
        logs["tpb"] = batch["input_ids"].ne(self.pad).sum() + batch["labels"].ne(self.pad).sum()
        logs["batch_train_time"] = train_time

        print(loss_tensors[0])
        self.log("train_loss", loss_tensors[0], on_step=True, prog_bar=True, logger=True, sync_dist=True)
        return {"loss": loss_tensors[0], "log": logs, "progress_bar":{"global_step":self.global_step}}

Outputs look like:
tensor(695.5908, device='cuda:0', grad_fn=<AddBackward0>)
Epoch 0:   3%|▎         | 1/30 [00:01<00:48,  1.68s/it, loss=1409.850, v_num=87, train_loss=705, global_step=0]tensor(688.4814, device='cuda:0', grad_fn=<AddBackward0>)
Epoch 0:   7%|▋         | 2/30 [00:01<00:27,  1.03it/s, loss=1392.687, v_num=87, train_loss=688, global_step=1]tensor(645.7902, device='cuda:0', grad_fn=<AddBackward0>)
Epoch 0:  10%|█         | 3/30 [00:02<00:19,  1.35it/s, loss=1358.177, v_num=87, train_loss=645, global_step=2]tensor(711.6373, device='cuda:0', grad_fn=<AddBackward0>)
Epoch 0:  13%|█▎        | 4/30 [00:02<00:16,  1.61it/s, loss=1371.488, v_num=87, train_loss=706, global_step=3]tensor(680.3423, device='cuda:0', grad_fn=<AddBackward0>)
Epoch 0:  17%|█▋        | 5/30 [00:02<00:13,  1.82it/s, loss=1374.843, v_num=87, train_loss=694, global_step=4]tensor(672.1014, device='cuda:0', grad_fn=<AddBackward0>)
Epoch 0:  20%|██        | 6/30 [00:03<00:12,  2.00it/s, loss=1390.955, v_num=87, train_loss=736, global_step=5]tensor(700.9694, device='cuda:0', grad_fn=<AddBackward0>)
Epoch 0:  23%|██▎       | 7/30 [00:03<00:10,  2.14it/s, loss=1391.849, v_num=87, train_loss=699, global_step=6]tensor(702.0041, device='cuda:0', grad_fn=<AddBackward0>)
Epoch 0:  27%|██▋       | 8/30tensor(714.2592, device='cuda:1', grad_fn=<AddBackward0>)loss=672, global_step=7]
tensor(687.0416, device='cuda:1', grad_fn=<AddBackward0>)
tensor(643.3681, device='cuda:1', grad_fn=<AddBackward0>)
tensor(699.7834, device='cuda:1', grad_fn=<AddBackward0>)
tensor(707.9196, device='cuda:1', grad_fn=<AddBackward0>)
tensor(799.4126, device='cuda:1', grad_fn=<AddBackward0>)
tensor(696.2469, device='cuda:1', grad_fn=<AddBackward0>)
tensor(641.3730, device='cuda:1', grad_fn=<AddBackward0>)

Please reproduce using [the BoringModel and post here]
https://colab.research.google.com/drive/13Neon4ncn5iwYYY1giuNrOaXcrIx4VQu#scrollTo=quj4LUDgmFvj
Expected behavior
train_loss and loss should be same. I would simply not return {loss:loss} but I need to return some stats like tpb for processing and hence have to.
Environment

CUDA:

GPU:

Tesla T4


available:         True
version:           10.1


Packages:

numpy:             1.18.5
pyTorch_debug:     True
pyTorch_version:   1.7.0+cu101
pytorch-lightning: 1.0.7
tqdm:              4.41.1


System:

OS:                Linux
architecture:

64bit



processor:         x86_64
python:            3.6.9
version:           #1 SMP Thu Jul 23 08:00:38 PDT 2020