UMUTeam/catalan_capitalization_punctuation_restoration_int8

Catalan Capitalization and Punctuation Restoration (8-bit quantized)

Model description

UMUTeam/catalan_capitalization_punctuation_restoration_int8 is a Transformer-based model (8-bit quantized) for restoring capitalization and punctuation in Catalan text. It is intended primarily for post-processing automatic speech recognition (ASR) transcripts, which often contain words without punctuation or correct capitalization. The model can also be applied to other unpunctuated Catalan text.

The model approaches restoration as a token-classification task and uses the BERTa Catalan language model (a RoBERTa-based architecture) as its pre-trained backbone, as described in the accompanying publication. It predicts capitalization and the punctuation mark following each word using its surrounding context.

Five punctuation marks are supported: period (.), comma (,), question mark (?), exclamation mark (!), and colon (:). Combined with two capitalization labels (lowercase and uppercase), they form 12 prediction classes:

l, u, ?l, ?u, !l, !u, ,l, ,u, .l, .u, :l, :u.

The suffix l indicates lowercase and u indicates uppercase/capitalized; any preceding symbol indicates the punctuation mark to place after the corresponding word. A label without a punctuation symbol means that no punctuation mark is inserted.

Example

• Input: què estàs fent
• Restored text: Què estàs fent?

This is an illustrative example from the task description, not a guarantee of a particular inference result.

Intended uses

• Improving the readability of Catalan ASR transcripts.
• Restoring capitalization and punctuation in unformatted Catalan text.
• Preparing text for downstream natural language processing tasks.

The model is not a speech-to-text system: it accepts text, not audio.

Training data

The model was fine-tuned using the Catalan portion of OpusParaCrawl, a collection of parallel texts gathered from web crawls as part of the ParaCrawl project (Bañón et al., 2020). The source text already included sentence boundaries, capitalization, and punctuation, allowing the creation of training examples with these features removed from the input and retained as labels.

After cleaning, extraction, and selection, the Catalan dataset contained 50,000 sentences and 1,136,708 words. Sentences containing less frequent target punctuation marks (especially question marks, exclamation marks, and colons) were preferentially selected to improve class balance.

The publication describes a 60% / 20% / 20% sentence-level split for training, validation, and testing. The corresponding word/token-label counts, as reported in Table 2 of the paper, are:

Split	Word/token-label count
Training	726,486
Training after data augmentation	981,345
Validation	181,517
Test	228,705
Total before augmentation	1,136,708

The augmented training set was obtained using three techniques:

1. Back-translation: Catalan text was translated into English and back into Catalan using Helsinki-NLP translation models, introducing lexical variation.
2. Random insertion: Words were inserted to approximate some errors observed in ASR output.
3. Random deletion: Words were removed to approximate some errors observed in ASR output.

The paper assigns an approximately equal 0.33 probability to each augmentation technique. Augmentation applies to the training data; validation and test data are reported separately.

Important: This model was trained using text extracted from web corpora and synthetically augmented examples, rather than exclusively on real ASR transcripts.

Training procedure

The pre-trained BERTa model was adapted through supervised fine-tuning for token classification. Each input word is associated with a label specifying its capitalization and following punctuation. Model-specific subword tokenization is used; special tokens are ignored during training (label -100), and the paper describes assigning a word's label to its subwords.

The accompanying paper does not provide sufficient information in its text to state the exact learning rate, batch size, number of epochs, or random seed for this released checkpoint. These settings are therefore not claimed here.

Evaluation

The model was evaluated on the held-out Catalan test split of OpusParaCrawl using the macro-averaged and weighted-averaged F1 scores across all 12 classes.

Model and training data	Macro-F1	Weighted-F1
BERTa with data augmentation	69.343%	90.858%
BERTa without data augmentation	68.762%	90.735%

How to use

from transformers import pipeline, AutoModelForTokenClassification, AutoTokenizer
import torch

def get_result_text_ca (list_entity, text):
    result_words = []
    punc_tags = ['?', '!', ',', '.', ':']
    tmp_word = ""
    for idx, entity in enumerate(list_entity): 
        start = entity["start"]
        end = entity["end"]
        tag = entity["entity"]
        word = entity["word"]
        
        # check punctuation
        punc_in = next((p for p in punc_tags if p in tag), "")
                
        subword = False
        # check subwords
        if word[0] != "Ġ": 
            subword = True
            if tmp_word == "":
                p_s = list_entity[idx-1]["start"]
                p_e = list_entity[idx-1]["end"]
                tmp_word = text[p_s:p_e] + text[start:end]
            else: 
                tmp_word = tmp_word + text[start:end]
            word = tmp_word
        else:
            tmp_word = ""
            word = text[start:end]
        
        if tag == "l": 
            word = word 
        elif tag == "u":
            word = word.capitalize()
        # case with punctuation
        else:
            if tag[-1] == "l":
                word = (punc_in + word) if punc_in in ["¿", "¡"] else (word + punc_in)
            elif tag[-1] == "u":
                word = (punc_in + word.capitalize()) if punc_in in ["¿", "¡"] else (word.capitalize() + punc_in)     
            
        if subword == True: 
            result_words[-1] = word
        else:
            result_words.append(word)

    return " ".join(result_words)


model_path = "UMUTeam/catalan_capitalization_punctuation_restoration_int8"

model = AutoModelForTokenClassification.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path)

pipe = pipeline("token-classification", model=model, tokenizer=tokenizer)

text = "el pacient presenta els símptomes següents febre dispnea nàusees i vòmits"
result = pipe(text)

print("Source text: "+ text)
result_text = get_result_text_ca(result, text)
print("Restored text: " +result_text)

You can also explore the project's demonstration Space.

Authors and affiliations

• Ronghao Pan — University of Murcia, Spain
• José Antonio García-Díaz — University of Murcia, Spain
• Pedro José Vivancos-Vicente — VÓCALI Sistemas Inteligentes S.L., Spain
• Rafael Valencia-García — University of Murcia, Spain

Funding and acknowledgements

• Research project 2021/C005/00150076, funded by the Spanish Government, through the Ministerio de Asuntos Económicos y Transformación, and by the European Union — NextGenerationEU/PRTR.
• LaTe4PSP research project PID2019-107652RB-I00, funded by MCIN/AEI/10.13039/501100011033.

Citation

If you use this model in research, please cite:

@article{pan2023evaluation,
  title   = {Evaluation of Transformer-Based Models for Punctuation and Capitalization Restoration in Catalan and Galician},
  author  = {Pan, Ronghao and Garc{\'i}a-D{\'i}az, Jos{\'e} Antonio and Vivancos-Vicente, Pedro Jos{\'e} and Valencia-Garc{\'i}a, Rafael},
  journal = {Procesamiento del Lenguaje Natural},
  number  = {70},
  pages   = {27--38},
  year    = {2023},
  doi     = {10.26342/2023-70-2}
}

Resources

• Paper: Evaluation of Transformer-Based Models for Punctuation and Capitalization Restoration in Catalan and Galician
• Hugging Face model: UMUTeam/catalan_capitalization_punctuation_restoration
• Pre-trained Catalan BERTa model: PlanTL-GOB-ES/roberta-base-ca