bert2bert-dialogsum

This is a BERT2BERT encoder-decoder model, warm-started from two bert-base-uncased checkpoints and fine-tuned for dialogue summarization on the DialogSum dataset.

Model Details

Model Description

The model takes a multi-turn dialogue as input and generates a short, single-sentence-style summary of the conversation. Both the encoder and decoder are initialized from bert-base-uncased; a randomly initialized cross-attention module was added to the decoder to let it attend over the encoder's output, following the standard Leveraging Pre-trained Checkpoints warm-start recipe for encoder-decoder models.

• Developed by: Cruciator
• Model type: Encoder-decoder (BERT2BERT) for text2text generation
• Language: English
• License: Apache 2.0 (inherited from the base bert-base-uncased checkpoints)
• Finetuned from model: bert-base-uncased (both encoder and decoder)

Model Sources

• Base model: bert-base-uncased
• Training data: knkarthick/dialogsum

Uses

Direct Use

Summarizing short, informal, multi-turn dialogues (e.g. chat transcripts, meeting snippets, customer support conversations) into a concise summary. Use the text2text-generation / EncoderDecoderModel.generate() API with an input dialogue truncated to 256 tokens.

Out-of-Scope Use

• Long-form documents, articles, or single-speaker text (the model was trained exclusively on short multi-turn dialogues, capped at 256 input tokens)
• Non-English text
• Domains far outside DialogSum's everyday-conversation style (e.g. legal, medical, or highly technical dialogue) without further fine-tuning

Bias, Risks, and Limitations

• Trained on a relatively small dataset (~12.5k training examples) for a small number of epochs, so summary quality and factual faithfulness are not guaranteed — always review generated summaries before relying on them.
• Inherits any biases present in bert-base-uncased and in the DialogSum dataset's crowd-sourced dialogues.
• Input dialogues are truncated at 256 tokens and output summaries are capped at 64 tokens, so longer conversations will be cut off rather than summarized in full.

How to Get Started with the Model

from transformers import EncoderDecoderModel, BertTokenizerFast

model = EncoderDecoderModel.from_pretrained("Cruciator/bert2bert-dialogsum")
tokenizer = BertTokenizerFast.from_pretrained("Cruciator/bert2bert-dialogsum")

dialogue = "#Person1#: Can we reschedule our meeting to Friday? #Person2#: Sure, works for me."
inputs = tokenizer(dialogue, return_tensors="pt", truncation=True, max_length=256)
summary_ids = model.generate(**inputs)
print(tokenizer.decode(summary_ids[0], skip_special_tokens=True))

Training Details

Training Data

DialogSum — 12,460 training examples and 500 validation examples of short, multi-turn dialogues paired with human-written summaries.

Training Procedure

• Inputs tokenized to a max length of 256 tokens; targets tokenized to a max length of 64 tokens
• Both encoder and decoder warm-started from bert-base-uncased; cross-attention weights randomly initialized
• Decoder inputs generated internally from labels (standard right-shifted teacher forcing)

Training Hyperparameters

• Epochs: 3
• Per-device train/eval batch size: 16
• Mixed precision: fp16
• Generation (eval-time): beam search, num_beams=2
• Generation (inference defaults on this checkpoint): beam search, num_beams=4, length_penalty=2.0, no_repeat_ngram_size=3, early_stopping=True
• Hardware: Google Colab, NVIDIA T4 GPU

Evaluation

Evaluated on the DialogSum validation split (500 examples) at the end of each epoch using ROUGE-2 F-measure.

Epoch	Training Loss	Validation Loss	ROUGE-2 F-measure
1	2.489	2.257	0.0612
2	2.072	2.091	0.0884
3	1.734	2.070	0.1069

ROUGE-2 was still improving at the final epoch, suggesting further training beyond 3 epochs would likely improve results further.

Technical Specifications

• Model size: ~0.2B parameters
• Tensor type: F32
• Framework: 🤗 Transformers (v5)