alixagari2000/ts-content-v1

📝 Large-Scale Text Summarization Model

This repository contains a powerful transformer-based text summarization model trained on a large-scale dataset with millions of examples. Built using PyTorch and the Hugging Face Transformers library, it provides an end-to-end pipeline from preprocessing and tokenization to training and evaluation using ROUGE metrics.

🚀 Overview

This project includes:

• 📚 A large-scale summarization dataset (millions of examples)
• 🧠 Transformer encoder-decoder architecture (e.g., BART/T5)
• 🔁 Training loop using PyTorch (compatible with 🤗 Trainer)
• 🧼 Tokenization and preprocessing pipeline
• 📏 Evaluation using standard metrics (ROUGE)
• 🧪 Ready-to-run scripts for training, evaluation, and inference

---

📦 Dataset

The training data consists of millions of article-summary pairs, covering diverse domains such as news, scientific papers, and web content. Data sources may include:

• CNN/DailyMail
• XSum
• ArXiv & PubMed
• Custom datasets

All datasets are tokenized, cleaned, and formatted to be Hugging Face Dataset-compatible.

---

🧠 Model Architecture

I use a Transformer encoder-decoder architecture, ideal for abstractive summarization tasks. Supported architectures include:

• facebook/bart-large
• t5-base or t5-large
• Custom models supported via config files

You can switch models easily via CLI or config YAML.

---

⚙️ Tokenization & Preprocessing

Text is tokenized using the corresponding tokenizer from the Hugging Face model hub. Preprocessing includes:

• Lowercasing & punctuation cleaning
• Truncation & padding
• Conversion to input/output pairs
• Tokenized using AutoTokenizer

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("facebook/bart-large")

def preprocess_function(examples):
    inputs = [doc for doc in examples["document"]]
    model_inputs = tokenizer(inputs, max_length=1024, truncation=True)

    with tokenizer.as_target_tokenizer():
        labels = tokenizer(examples["summary"], max_length=128, truncation=True)

    model_inputs["labels"] = labels["input_ids"]
    return model_inputs