Bert2bert EncoderDecoderModel from Huggingface is generating a zero tensor for any input
Author: Aakash12980
Pull request: no
Labels: name: wontfix

I am using Bert2Bert EncoderDecoderModel from Huggingface for sentence simplification. But my model is generating a zero tensor of the same length regardless of the input. Could someone help me what is wrong in my project. The GitHub link of my project is this https://github.com/Aakash12980/Sentence-Simplification-using-Transformer. Please help me through this.

My decode module is like this:
`
@task.command()
@click.option('--src_test', default="./drive/My Drive/Mini Project/dataset/src_test.txt", help="test source file path")
@click.option('--tgt_test', default="./drive/My Drive/Mini Project/dataset/tgt_test.txt", help="test target file path")
@click.option('--best_model', default="./drive/My Drive/Mini Project/best_model/model.pt", help="best model file path")

def test(**kwargs):

    print("Testing Model module executing...")
    logging.info(f"Test module invoked.")

    src_test = open_file(kwargs['src_test'])
    tgt_test = open_file(kwargs['tgt_test'])
    len_data = len(src_test)

    test_dataset = WikiDataset(src_test, tgt_test)

    test_dl = DataLoader(test_dataset, batch_size=TRAIN_BATCH_SIZE, collate_fn=collate_fn, shuffle=True)

    _,_,_ = load_checkpt(kwargs["best_model"])
    print("Model loaded...")
    model.to(device)
    model.eval()

    test_start_time = time.time()
    epoch_test_loss = evaluate(test_dl, 0)
    epoch_test_loss = epoch_test_loss/len_data
    print(f'avg. test loss: {epoch_test_loss:.5f} | time elapsed: {time.time() - test_start_time}')
    logging.info(f'avg. test loss: {epoch_test_loss:.5f} | time elapsed: {time.time() - test_start_time}')
    print("Test Complete!")
 `
and the output its generates is this: 

`input: tensor([[  101,  1188,  1108,  1272, 11310,  1125,  1136,  1151,  2548,  1106,
          2732, 20386,  2692,   117,  2693,  1117, 16975,   117,  1105,  1103,
          8183,  1571,  1105,  8183,  1545,  3784,   118,  8581,  1125,  3335,
          1136,  1678,  2629,   119,   102]])
output: tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])
input: tensor([[  101,  1109,  4665,  1108,  6497,  1173,  1112,  1126,   169,   169,
          6548, 21718, 24226, 25285,   112,   112,   117,  1112,  1103,  1586,
          1108,   169,   169, 16314,  1105,  4489,  1200,  1190,  1103,  6188,
           117,  4411,  1126,  5340,  1895,  4252, 20473,  4626,  2116,  1104,
          9494,   112,   112,   119,   102]])
output: tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])
input: tensor([[  101, 13030,  1643,  2883,  1144,  3567,  8593,  9648,  1958,   117,
          1259,  1210,  4748,  1453,  4278,   117,  1160,  1635,  8793,   112,
          1635,  4278,  1105,   170,  6068,  1635,  1509,   119,   102]])
output: tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])`


Comments:
Hey @Aakash12980,

Could you please provide a complete code examples with which I can reproduce the results? Did you train the model yourself or did you use a pretrained model? (Which one in case you did?)

> Hey @Aakash12980,
> 
> Could you please provide a complete code examples with which I can reproduce the results? Did you train the model yourself or did you use a pretrained model? (Which one in case you did?)

@patrickvonplaten I have used pretrained bert2bert model and I am fine-tuning it for sentence simplification. 

The full code in run.py is below

```
logging.basicConfig(filename="./drive/My Drive/Mini Project/log_file.log", level=logging.INFO, 
                format="%(asctime)s:%(levelname)s: %(message)s")
CONTEXT_SETTINGS = dict(help_option_names = ['-h', '--help'])

TRAIN_BATCH_SIZE = 4   
N_EPOCH = 5
LOG_EVERY = 11000

config_encoder = BertConfig()
config_decoder = BertConfig()
config_decoder.is_decoder = True
config_decoder.add_cross_attention = True
config = EncoderDecoderConfig.from_encoder_decoder_configs(config_encoder, config_decoder)
model = EncoderDecoderModel.from_encoder_decoder_pretrained('bert-base-cased', 'bert-base-cased', config=config)

device = "cuda" if torch.cuda.is_available() else "cpu"
print(f"Using {device} as device")


def collate_fn(batch):
    data_list, label_list = [], []
    for _data, _label in batch:
        data_list.append(_data)
        label_list.append(_label)
    return data_list, label_list

def save_model_checkpt(state, is_best, check_pt_path, best_model_path):
    f_path = check_pt_path
    torch.save(state, f_path)

    if is_best:
        best_fpath = best_model_path
        shutil.copyfile(f_path, best_fpath)

def load_checkpt(checkpt_path, optimizer=None):
    if device == "cpu":
        checkpoint = torch.load(checkpt_path, map_location=torch.device("cpu"))
        model.load_state_dict(checkpoint["model_state_dict"])
        if optimizer is not None:
            optimizer.load_state_dict(checkpoint["optimizer_state_dict"], map_location=torch.device("cpu"))
            
    else:
        model.load_state_dict(checkpoint["model_state_dict"])
        if optimizer is not None:
            optimizer.load_state_dict(checkpoint["optimizer_state_dict"])

    eval_loss = checkpoint["eval_loss"]
    epoch = checkpoint["epoch"]

    return optimizer, eval_loss, epoch



def evaluate(batch_iter, e_loss):
    was_training = model.training
    model.eval()
    eval_loss = e_loss

    with torch.no_grad():
        for step, batch in enumerate(batch_iter):
            src_tensors, src_attn_tensors, tgt_tensors, tgt_attn_tensors = generate_tokens(batch)
            loss, _ = model(input_ids = src_tensors.to(device), 
                            decoder_input_ids = tgt_tensors.to(device),
                            attention_mask = src_attn_tensors.to(device),
                            decoder_attention_mask = tgt_attn_tensors.to(device),
                            labels=tgt_tensors.to(device))[:2]
            
            eval_loss += (1/(step+1)) * (loss.item() - eval_loss)

    if was_training:
        model.train()

    return eval_loss 

@click.group(context_settings=CONTEXT_SETTINGS)
@click.version_option(version = '1.0.0')
def task():
    ''' This is the documentation of the main file. This is the reference for executing this file.'''
    pass


@task.command()
@click.option('--src_train', default="./drive/My Drive/Mini Project/dataset/src_train.txt", help="train source file path")
@click.option('--tgt_train', default="./drive/My Drive/Mini Project/dataset/tgt_train.txt", help="train target file path")
@click.option('--src_valid', default="./drive/My Drive/Mini Project/dataset/src_valid.txt", help="validation source file path")
@click.option('--tgt_valid', default="./drive/My Drive/Mini Project/dataset/tgt_valid.txt", help="validation target file path")
@click.option('--best_model', default="./drive/My Drive/Mini Project/best_model/model.pt", help="best model file path")
@click.option('--checkpoint_path', default="./drive/My Drive/Mini Project/checkpoint/model_ckpt.pt", help=" model check point files path")
@click.option('--seed', default=123, help="manual seed value (default=123)")
def train(**kwargs):
    print("Training data module executing...")
    logging.info(f"Train module invoked.")
    seed = kwargs["seed"]
    torch.manual_seed(seed)
    if device == "cuda":
        torch.cuda.manual_seed(seed)
    
    np.random.seed(seed)
    print("Loading dataset...")
    
    src_train = open_file(kwargs['src_train'])
    tgt_train = open_file(kwargs['tgt_train'])
    src_valid = open_file(kwargs['src_valid'])
    tgt_valid = open_file(kwargs['tgt_valid'])
    train_len = len(src_train)
    valid_len = len(src_valid)
    print("Dataset Loaded.")

    train_dataset = WikiDataset(src_train, tgt_train)
    valid_dataset = WikiDataset(src_valid, tgt_valid)
    del src_valid, src_train, tgt_train, tgt_valid

    print("Creating Dataloader...")
    train_dl = DataLoader(train_dataset, batch_size=TRAIN_BATCH_SIZE, collate_fn=collate_fn, shuffle=True)
    valid_dl = DataLoader(valid_dataset, batch_size=TRAIN_BATCH_SIZE, collate_fn=collate_fn, shuffle=True)
    print("Dataloader created.")

    model.to(device)
    param_optimizer = list(model.named_parameters())
    no_decay = ['bias', 'gamma', 'beta']
    optimizer_grouped_parameters = [
        {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],
        'weight_decay_rate': 0.01},
        {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)],
        'weight_decay_rate': 0.0}
    ]
    optimizer = torch.optim.AdamW(optimizer_grouped_parameters, lr=3e-3)
    eval_loss = float('inf')
    start_epoch = 0
    if os.path.exists(kwargs["checkpoint_path"]):
        optimizer, eval_loss, start_epoch = load_checkpt(kwargs["checkpoint_path"], optimizer)
        print(f"Loading model from checkpoint with start epoch: {start_epoch} and loss: {eval_loss}")
        logging.info(f"Model loaded from saved checkpoint with start epoch: {start_epoch} and loss: {eval_loss}")

    train_model(start_epoch, eval_loss, (train_dl, valid_dl), optimizer, kwargs["checkpoint_path"], kwargs["best_model"], (train_len, valid_len))
    print("Model Training Complete!")
    
    

@task.command()
@click.option('--src_test', default="./drive/My Drive/Mini Project/dataset/src_test.txt", help="test source file path")
@click.option('--tgt_test', default="./drive/My Drive/Mini Project/dataset/tgt_test.txt", help="test target file path")
@click.option('--best_model', default="./drive/My Drive/Mini Project/best_model/model.pt", help="best model file path")
def test(**kwargs):
    print("Testing Model module executing...")
    logging.info(f"Test module invoked.")

    src_test = open_file(kwargs['src_test'])
    tgt_test = open_file(kwargs['tgt_test'])
    len_data = len(src_test)

    test_dataset = WikiDataset(src_test, tgt_test)

    test_dl = DataLoader(test_dataset, batch_size=TRAIN_BATCH_SIZE, collate_fn=collate_fn, shuffle=True)

    _,_,_ = load_checkpt(kwargs["best_model"])
    print("Model loaded...")
    model.to(device)
    model.eval()

    test_start_time = time.time()
    epoch_test_loss = evaluate(test_dl, 0)
    epoch_test_loss = epoch_test_loss/len_data
    print(f'avg. test loss: {epoch_test_loss:.5f} | time elapsed: {time.time() - test_start_time}')
    logging.info(f'avg. test loss: {epoch_test_loss:.5f} | time elapsed: {time.time() - test_start_time}')
    print("Test Complete!")
 

#/drive/My Drive/Mini Project
@task.command()
@click.option('--src_file', default="./drive/My Drive/Mini Project/dataset/src_file.txt", help="test source file path")
@click.option('--best_model', default="./drive/My Drive/Mini Project/checkpoint/model_ckpt.pt", help="best model file path")
@click.option('--output', default="./drive/My Drive/Mini Project/outputs/decoded.txt", help="file path to save predictions")
def decode(**kwargs):
    print("Decoding sentences module executing...")
    logging.info(f"Decode module invoked.")
    src_test = open_file(kwargs['src_file'])
    print("Saved model loading...")
    _,_,_ = load_checkpt(kwargs["best_model"])
    print(f"Model loaded.")
    model.to(device)
    model.eval()
    inp_tokens = create_sent_tokens(src_test)
    predicted_list = []
    print("Decoding Sentences...")
    for tensor in inp_tokens:
        with torch.no_grad():
            predicted = model.generate(tensor.to(device), decoder_start_token_id=model.config.decoder.pad_token_id)
            print(f"input: {tensor}")
            print(f'output: {predicted.squeeze()}')
            predicted_list.append(predicted.squeeze())
    
    output = get_sent_from_tokens(predicted_list)
    with open(kwargs["output"], "w") as f:
        for sent in output:
            f.write(sent + "\n")
    print("Output file saved successfully.")

def train_model(start_epoch, eval_loss, loaders, optimizer, check_pt_path, best_model_path, len_data):
    best_eval_loss = eval_loss
    print("Model training started...")
    for epoch in range(start_epoch, N_EPOCH):
        print(f"Epoch {epoch} running...")
        epoch_start_time = time.time()
        epoch_train_loss = 0
        epoch_eval_loss = 0

        model.train()
        for step, batch in enumerate(loaders[0]):

            src_tensors, src_attn_tensors, tgt_tensors, tgt_attn_tensors = generate_tokens(batch)

            optimizer.zero_grad()
            model.zero_grad()
            loss = model(input_ids = src_tensors.to(device), 
                            decoder_input_ids = tgt_tensors.to(device),
                            attention_mask = src_attn_tensors.to(device),
                            decoder_attention_mask = tgt_attn_tensors.to(device),
                            labels = tgt_tensors.to(device))[0]
            
            loss.backward()
            optimizer.step()
            epoch_train_loss += (1/(step+1))*(loss.item() - epoch_train_loss)

            if (step+1) % LOG_EVERY == 0:
                print(f'Epoch: {epoch} | iter: {step+1} | avg. loss: {epoch_train_loss/TRAIN_BATCH_SIZE} | time elapsed: {time.time() - epoch_start_time}')
                logging.info(f'Epoch: {epoch} | iter: {step+1} | avg. loss: {epoch_train_loss/TRAIN_BATCH_SIZE} | time elapsed: {time.time() - epoch_start_time}')
                eval_start_time = time.time()
                epoch_eval_loss = evaluate(loaders[1], epoch_eval_loss)
                epoch_eval_loss = epoch_eval_loss/TRAIN_BATCH_SIZE
                print(f'Completed Epoch: {epoch} | avg. eval loss: {epoch_eval_loss:.5f} | time elapsed: {time.time() - eval_start_time}')
                logging.info(f'Completed Epoch: {epoch} | avg. eval loss: {epoch_eval_loss:.5f} | time elapsed: {time.time() - eval_start_time}')
                
                check_pt = {
                    'epoch': epoch+1,
                    'model_state_dict': model.state_dict(),
                    'optimizer_state_dict': optimizer.state_dict(),
                    'eval_loss': epoch_eval_loss
                }
                check_pt_time = time.time()
                print("Saving Checkpoint.......")
                if epoch_eval_loss < best_eval_loss:
                    print("New best model found")
                    logging.info(f"New best model found")
                    best_eval_loss = epoch_eval_loss
                    save_model_checkpt(check_pt, True, check_pt_path, best_model_path)
                else:
                    save_model_checkpt(check_pt, False, check_pt_path, best_model_path)  
                print(f"Checkpoint saved successfully with time: {time.time() - check_pt_time}")
                logging.info(f"Checkpoint saved successfully with time: {time.time() - check_pt_time}")
        
        gc.collect()
        torch.cuda.empty_cache()     


if __name__ == "__main__":
    task()
```

The GitHub link for my full project is here [https://github.com/Aakash12980/Sentence-Simplification-using-Transformer](url). 

Hey @Aakash12980 - I think you forgot to replace the pad_token_ids with -100 in your code. You can checkout the map function here: https://huggingface.co/patrickvonplaten/bert2bert-cnn_dailymail-fp16#training-script to see how this should be done.

> Hey @Aakash12980 - I think you forgot to replace the pad_token_ids with -100 in your code. You can checkout the map function here: https://huggingface.co/patrickvonplaten/bert2bert-cnn_dailymail-fp16#training-script to see how this should be done.

@patrickvonplaten I have included -100 as well in the labels and again I am getting the same weird outputs from my model.generate() method. 
```
Setting `pad_token_id` to 102 (first `eos_token_id`) to generate sequence
output: tensor([101, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119,
        119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119,
        119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119,
        119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119,
        119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119,
        119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119,
        119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119,
        119, 119], device='cuda:0')
Setting `pad_token_id` to 102 (first `eos_token_id`) to generate sequence
output: tensor([101, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119,
        119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119,
        119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119,
        119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119,
        119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119,
        119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119,
        119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119,
        119, 119], device='cuda:0')
```

My updated code is this link: [https://github.com/Aakash12980/Sentence-Simplification-using-Transformer](url)

Hey @Aakash12980, hmm - I'm not sure what to do here then either. I've done quite a lot of Bert2Bert fine-tune runs and they all worked for me. Also I will publish a more in-detail notebook in ~2 weeks on how to do Bert2Bert training. 

It's quite time-consuming and difficult for me to dive deeper into your code, so the best I can do for you at the moment is this training code snippet: https://huggingface.co/patrickvonplaten/bert2bert-cnn_dailymail-fp16#training-script

@patrickvonplaten Thank you so much for your time. 

@patrickvonplaten Could you please explain to me why you have assigned -100 to PAD tokens?
```
# mask loss for padding
    batch["labels"] = [
        [-100 if token == tokenizer.pad_token_id else token for token in labels] for labels in batch["labels"]
    ]
```
I don't quite understand what mask loss for padding means. Also, Do I need to pass labels when I am not using Trainer() method?
```
loss, logits = self.model(input_ids = src_tensors.to(device), 
                            decoder_input_ids = tgt_tensors.to(device),
                            attention_mask = src_attn_tensors.to(device),
                            decoder_attention_mask = tgt_attn_tensors.to(device),
                            labels = labels.to(device))[:2]
```
Why do we pass labels since we have already provided decoder_iniput_ids?

if label == -100, it means that on such tokens no loss is calculated and therefore does not affect the gradient. We don't want the model to learn to predict PAD tokens, so we make sure that no loss is calculated by putting a -100 on them. PyTorch's CE loss function uses -100 as the ignore token number.

Thank you @patrickvonplaten  I have trained my model but my model is generating tokens exactly similar to the input tokens. I tried to find out what have I done wrong but I couldn't. I don't know why it is generating the same tokens as inputs. Do you have any idea?
```
Model loaded.
Decoding Sentences...
input tokens: tensor([[  101,   140, 23339,  1706, 12788,   140, 22715,  8469,  1106,  2194,
          1103,  5080,  1104,   172, 23339, 17458,  1107,   170,   172, 22715,
         24759,  3536,   119,   102]])
Setting `pad_token_id` to 102 (first `eos_token_id`) to generate sequence
output tokens: tensor([  101,   140, 23339,  1706, 12788,   140, 22715,  8469,  1106,  2194,
         1103,  5080,  1104,   172, 23339, 17458,  1107,   170,   172, 22715,
        24759,  3536,   119,   102], device='cuda:0')
input tokens: tensor([[  101, 18959, 18072,  1108,  1549,  1103, 13852,  6607,  1104,   156,
         14046,  3263,  1670,  1118,  1624,  1600,  1107, 18563,  1571,   117,
          1106,  9489,  1105,  2669,  1103,  2226,   112,   188, 16358, 20408,
           119,   102]])
Setting `pad_token_id` to 102 (first `eos_token_id`) to generate sequence
output tokens: tensor([  101, 18959, 18072,  1108,  1549,  1103, 13852,  6607,  1104,   156,
        14046,  3263,  1670,  1118,  1624,  1600,  1107, 18563,  1571,   117,
         1106,  9489,  1105,  2669,  1103,  2226,   112,   188, 16358, 20408,
          119,   102], device='cuda:0')
input tokens: tensor([[  101,   153, 14272,  1633,  1110,   170,  5188,  1107,  1103,  1318,
         26042,  2853,  1107,  1564,   118,  2466,  1699,   119,   102]])
Setting `pad_token_id` to 102 (first `eos_token_id`) to generate sequence
output tokens: tensor([  101,   153, 14272,  1633,  1110,   170,  5188,  1107,  1103,  1318,
        26042,  2853,  1107,  1564,   118,  2466,  1699,   119,   102],
       device='cuda:0')
```
It is driving me crazy. I ran the training module for 4 epochs and this was the best model I got so far.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.