JanSchachtschabel/mt5-small-german-educational-metadata

mT5-small – Metadata for German Educational Materials (Test Model)

Test and research model — not for production use. The model generates titles, descriptions and keywords for German educational materials. Its quality is sufficient for a prototype, but not for unchecked use in production, editorial systems or automated decisions. Titles in particular are weak (see Evaluation and Weaknesses). All outputs must be reviewed by a human.

Demo

Try the model in the browser: mT5 Metadata Demo.

The Space runs on ZeroGPU and offers five sample texts from the test split plus free text input. You can generate the title, description and keywords individually or all three at once, and compare the result with the reference metadata. It is a test UI, not a production tool: outputs still require human review. Free ZeroGPU usage is subject to a daily quota (2 min/day unauthenticated vs. 5 min/day with a free Hugging Face account), so log in for the larger quota.

Overview

This model is a Seq2Seq model based on mT5-small that generates three metadata fields from the full text of a German-language educational resource:

• title
• description
• keywords

All three targets are learned by one shared model. A prefix in the input text (title:, description:, keywords:) selects which field is generated. The model was fine-tuned on synthetic educational materials and is specialised in German educational language.

Base model	deutsche-telekom/mt5-small-sum-de-mit-v1 (MIT), built on google/mt5-small
Training data	JanSchachtschabel/synthetic-german-educational-materials-50k (CC BY 4.0)
Parameters	300,164,480 (mT5-small, 8 encoder / 8 decoder layers, d_model 512)
Language	German
License	MIT (model weights)
Version	2026-11-05, 0.1-test

Training data

The dataset contains 50,000 synthetic German educational materials from all education sectors (school, higher education, vocational education and training, adult education, early childhood education). Each record consists of a full text of about one A4 page and the fields title, description and keywords derived from it.

Split	Materials
Train	45,144
Validation	2,463
Test	2,393

The splits are separated by topic: all materials generated from the same topic are in the same split, so the model sees no training topics during evaluation. The texts were generated by a language model; they are not real, expert-reviewed materials (see the dataset's limitations).

The three training targets

Field	Prefix	Max. target tokens	Output
title	title:	48	single-line title
description	description:	256	short description, several sentences
keywords	keywords:	96	comma-separated keywords

Each prefix is followed by a single space. The prefix is placed directly before the full text, e.g. title: <full text>. The input is truncated to 1024 tokens (mode head, i.e. the beginning of the text). All settings are stored in the repository in metadata_generation.json.

Usage

import json

import torch
from huggingface_hub import hf_hub_download
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

repo_id = "JanSchachtschabel/mt5-small-german-educational-metadata"

tok = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForSeq2SeqLM.from_pretrained(repo_id)
device = "cuda" if torch.cuda.is_available() else "cpu"
if device == "cuda" and torch.cuda.is_bf16_supported():
    model = model.to(torch.bfloat16)  # bf16 as in training; fp16 is unstable with mT5
model = model.to(device).eval()

settings_path = hf_hub_download(repo_id, "metadata_generation.json")
with open(settings_path, encoding="utf-8") as fh:
    settings = json.load(fh)
text = """The full text of the educational material goes here ..."""

for task, prefix in settings["tasks"].items():
    enc = tok(prefix + text, max_length=settings["max_source_length"],
              truncation=True, return_tensors="pt").to(device)
    with torch.no_grad():
        ids = model.generate(
            **enc,
            max_new_tokens=settings["max_target_lengths"][task],
            num_beams=4,
            no_repeat_ngram_size=3,
        )
    print(f"{task}: {tok.decode(ids[0], skip_special_tokens=True)}")

Notes:

• Input: use clean plain text where possible (no navigation, cookie notices or HTML residue). During training, Markdown was reduced to plain text and the text was cut after 1024 tokens.
• Beams: num_beams=4 is the default. 2 is usually similarly good and slightly faster; 1 (greedy) is the fastest but tends to repeat phrases and invent words. 8 brings little improvement for this model.
• Length: for long texts only the beginning is used. For long web pages, trim to the relevant sections first.
• Review: outputs are suggestions, not verified metadata.

Training

Setting	Value
Base model	deutsche-telekom/mt5-small-sum-de-mit-v1
Training examples	135,432 (45,144 materials × 3 targets)
Epochs	3
Batch size	6 per GPU × 2 GPUs = 12
Gradient accumulation	1
Learning rate	1e-4
Precision	bf16
Gradient checkpointing	yes
Max. source length	1024 tokens
Max. target lengths	48 / 256 / 96 (title / description / keywords)
Seed	42
Training steps	33,858
Runtime	11 h 59 min 43 s on 2 × RTX 5060 Ti (16 GB)
Validation loss	3.402 → 3.164 → 3.100 (epochs 1–3)

The final model corresponds to the best checkpoint (checkpoint-33858, epoch 3, validation loss 3.100).

The three targets contribute unequally to the token-level loss: description about 79 %, keywords about 15 %, title only about 6 %. The model therefore optimises mostly for descriptions; titles benefit least from training.

Evaluation

Evaluated on the first 200 materials of the test split (2,393 materials, topics not seen in training), 4 beams, bf16, with the same preprocessing and length limits as in training:

Field	Metric	Value
Title	exact match (strict)	0.005
Title	word-overlap F1 (loose)	0.419
Description	ROUGE-L F1	0.296
Keywords	keyword F1, whole terms (strict)	0.327
Keywords	word-overlap F1 (loose)	0.390

Metric definitions:

• Title strict: exact match (case-insensitive).
• Title loose: token F1 between generated and reference title.
• Description: word-level ROUGE-L F1.
• Keywords strict: F1 over the comma-separated keyword lists (whole terms).
• Keywords loose: token F1 over all keyword words.

Interpretation: descriptions reach ROUGE-L 0.296, just below the level of a usable prototype; keywords are solid but could be better. Titles are the weakest point: only 0.5 % exactly match the reference title, although word overlap is 0.419. The model formulates titles that are semantically close but rarely word-identical. Reference titles in the dataset are often concatenated keywords; natural formulations are penalised heavily by exact match.

Strengths

• Produces fluent, topically appropriate descriptions for German educational texts.
• Usually returns plausible keywords from educational and subject-specific vocabulary.
• A small model (300M parameters) that runs on a single GPU or CPU.
• All three fields with one model and a uniform prefix scheme.

Weaknesses and limitations

• Titles are weak. Titles are often topically appropriate but phrased differently; exact matches are rare (0.005). Titles should not be used without review. Beam search (2–4) improves quality over greedy decoding but does not solve the problem.
• Repetitions occur in descriptions (e.g. duplicated words or phrases).
• Synthetic training data: the texts come from a single language model and have a uniform style. The model may invent facts or reuse formulations that are not supported by the specific material.
• Domain shift: real websites, scanned scripts or very long documents differ from the training data. Inputs are truncated after 1024 tokens; the rest is ignored.
• German only / German education system. Other languages or education systems are not supported.
• No human evaluation: all metrics come from the synthetic test split. Quality on real materials has not been systematically evaluated.
• Not production-ready: no guarantee of correctness, completeness, consistency or freedom from copyright issues in the outputs. Do not use for automated grading, selection or other high-stakes decisions.

License and provenance

• Model weights: MIT, derived from the MIT-licensed base model deutsche-telekom/mt5-small-sum-de-mit-v1.
• Training data: JanSchachtschabel/synthetic-german-educational-materials-50k, licensed under CC BY 4.0. The data is synthetic; the dataset card describes its origin, limitations and license terms in detail.
• Base model provenance: the base model is a German summarization model by Deutsche Telekom AG, based on google/mt5-small.

Feedback

Errors, unsuitable outputs or suggestions can be reported through the Discussions of this repository. Production use would require a larger, curated dataset, a better title metric (e.g. keyword coverage instead of exact match) and a human evaluation.