JayShah07/falconai-text-bullet-t5

Text-to-Bullets T5

A fine-tuned T5 encoder-decoder model for converting English prose into concise bullet points.

Model Summary

Property	Value
Architecture	T5 encoder-decoder
Parameters	~60.5M
Language	English
Task	Text-to-bullets generation
Framework	PyTorch / Transformers
Hugging Face task	text2text-generation

The model was fine-tuned specifically for source-to-target text transformation.

Unlike a decoder-only model, T5 separates the task into:

Source text
   ↓
Encoder
   ↓
Source representation
   ↓
Decoder
   ↓
Bullet points

This architecture is well suited to summarization-style tasks because the decoder generates output while cross-attending to the encoded source.

---

Training Results

The model was fine-tuned for seven epochs.

Epoch	Train Loss	Val Loss	ROUGE-1	ROUGE-2	ROUGE-L	BERT P	BERT R	BERT F1	Bullet Format	Pred. Bullets	Ref. Bullets	Bullet Error	Compression	Ref. Compression	Generated Tokens
1	0.5121	0.4222	0.8007	0.7850	0.7770	0.9626	0.9316	0.9457	1.000	1.24	4.54	3.36	0.2214	0.2501	54.47
2	0.3232	0.2900	0.9544	0.9503	0.9493	0.9962	0.9842	0.9900	1.000	4.13	4.54	0.45	0.2414	0.2501	60.86
3	0.2630	0.2173	0.9751	0.9725	0.9701	0.9976	0.9910	0.9942	1.000	4.37	4.54	0.29	0.2464	0.2501	64.47
4	0.2172	0.1788	0.9816	0.9796	0.9774	0.9982	0.9928	0.9954	1.000	4.38	4.54	0.22	0.2470	0.2501	64.36
5	0.1935	0.1619	0.9863	0.9845	0.9828	0.9984	0.9943	0.9963	1.000	4.42	4.54	0.18	0.2481	0.2501	64.99
6	0.1900	0.1563	0.9863	0.9845	0.9828	0.9984	0.9943	0.9963	1.000	4.42	4.54	0.18	0.2481	0.2501	64.99
7	0.1871	0.1555	0.9857	0.9837	0.9823	0.9984	0.9941	0.9962	1.000	4.42	4.54	0.18	0.2479	0.2501	64.83

Training observations

The largest improvement happened between epochs 1 and 2.

• ROUGE-L increased from 0.7770 → 0.9493
• Mean bullet-count error decreased from 3.36 → 0.45
• Performance largely plateaued around epochs 5–6
• Bullet-format compliance remained 100% in the reported validation evaluation

These are training-time validation results and should not be directly compared with later held-out evaluations unless the same split and generation settings are used.

---

Later 500-Example Evaluation

A later FP32 evaluation on 500 examples produced:

Metric	Value
ROUGE-1	0.6041
ROUGE-2	0.5330
ROUGE-L	0.5603
BERTScore F1	0.8826
Bullet Format	1.000
Median Latency	0.895 s
P95 Latency	1.947 s
Throughput	97.28 tokens/s

---

Comparison with Qwen3 0.6B

Metric	Qwen3 0.6B	Fine-Tuned T5
Approx. Parameters	~600M	~60.5M
Architecture	Decoder-only	Encoder-decoder
Task Specialized	No	Yes
ROUGE-1	0.6494	0.6041
ROUGE-2	0.4702	0.5330
ROUGE-L	0.5463	0.5603
Bullet Format	0.9999	1.0000

The T5 model is roughly an order of magnitude smaller by parameter count.

In the reported evaluations, Qwen retained the higher ROUGE-1 score, while the fine-tuned T5 produced higher ROUGE-2 and ROUGE-L.

These measurements were collected at different stages of the experiment, so they should be treated as experimental context rather than a perfectly controlled head-to-head benchmark.

The key takeaway is that a much smaller task-specific encoder-decoder model can be highly competitive with a larger general-purpose decoder model on a narrow text-transformation task.

---

Usage

Install dependencies:

pip install torch transformers sentencepiece

Load and run the model:

import torch
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

MODEL_ID = "JayShah07/falconai-text-bullet-t5"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForSeq2SeqLM.from_pretrained(MODEL_ID).eval()

text = """
The Solar System consists of the Sun and the objects gravitationally bound
to it, including eight planets, dwarf planets, moons, asteroids and comets.
It formed approximately 4.6 billion years ago. Mercury, Venus, Earth and
Mars are terrestrial planets, while Jupiter and Saturn are gas giants and
Uranus and Neptune are ice giants.
""".strip()

inputs = tokenizer(
    text,
    return_tensors="pt",
    truncation=True,
)

with torch.inference_mode():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=False,
        num_beams=1,
    )

output = tokenizer.decode(
    output_ids[0],
    skip_special_tokens=True,
)

print(output)

---

Example Output Style

- The Solar System contains the Sun, eight planets, dwarf planets, moons, asteroids and comets.
- It formed approximately 4.6 billion years ago.
- Mercury, Venus, Earth and Mars are terrestrial planets.
- Jupiter and Saturn are gas giants, while Uranus and Neptune are ice giants.

Exact wording and bullet segmentation may vary.

---

Intended Use

This model is intended for:

• converting paragraphs into bullet points
• restructuring reports and articles
• summarization-style text transformation
• extracting key information from English prose

Limitations

The model may omit details from dense text, merge related facts, or produce imperfect summaries on text outside its training distribution.

Generated outputs should be reviewed before high-stakes use.

---

Author

Developed by Jay Shah as an experiment in task-specific encoder-decoder fine-tuning.