About "Init tensors using type_as"
As described in the doc: https://pytorch-lightning.readthedocs.io/en/stable/multi_gpu.html#init-tensors-using-type-as

When you need to create a new tensor, use type_as. This will make your code scale to any arbitrary number of GPUs or TPUs with Lightning.

# with lightning
def forward(self, x):
    z = torch.Tensor(2, 3)
    z = z.type_as(x, device=self.device)

However, it would be not convenient when there is no proper x , for example:
class Model(pl.LightningModule):
    def __init__(self, opt):
        super().__init__()
        self.opt = opt
        self.net = Net(opt)

    def forward(self, x):
        return self.net(x)
    def training_step(self, batch, batch_id):
        x, labels = batch
        ### labels: a tuple (0,0,0,....1,0)
        ### to(self.device) is right
        labels = torch.LongTensor(labels).to(self.device)  
        output = self(x)
        loss = F.cross_entropy(output, labels)
        return {"loss": loss}

class Net(nn.Module):
    def __init__(self, opt):

        super().__init__()
        self.opt = opt
        self.init_bert()
    def forward(self, x):
        x = self.get_bert(x)
        return x
    def init_bert(self):
        self.bert_tokenizer = AutoTokenizer.from_pretrained(self.opt.bert_path)
        self.bert = AutoModel.from_pretrained(self.opt.bert_path)
    def get_bert(self, sentence_lists):
        sentence_lists = [' '.join(x) for x in sentence_lists]
        ids = self.bert_tokenizer(sentence_lists, padding=True, return_tensors="pt")
        ###
        #  how to put the ids['input_ids'] into the right device ???
        #  .to(self.bert.device) is right
        #  .to(self.device) is wrong
        ###
        inputs = ids['input_ids'].to(self.bert.device)
        print("**************")
        print(self.device)    
        print(self.bert.device)
        print("**************")
        embeddings = self.bert(inputs)
        return embeddings[0]

The problems have been marked in the code.
My confusion is that when I run the code using multi-gpu:
(1) why "labels = torch.LongTensor(labels).to(self.device)  " is right while "inputs = ids['input_ids'].to(self.device)" is wrong.
In other words, in the multi-gpu running, the self.device is not consitent with self.bert.device.
(2) For the above case, are there any other ways to send the ids['input_ids'] into the right device in the multi-gpus running, rather than   inputs = ids['input_ids'].to(self.bert.device)  since the doc of lightning requires Delete .cuda() or .to() calls