Transformer-eng-por

Model Overview

The transformer-eng-por model is a transformer trained for text classification.

Details

Size: 23,805,216 parameters
Model type: Transformer
Optimizer: rmsprop
Number of Epochs: 30
Embbedding dimensionality: 256
Dense dimensionality: 2048
Attention heads: 8
Vocabulary size: 20000
Sequence lenght: 20
Hardware: Tesla V4
Emissions: Not measured
Total Energy Consumption: Not measured

How to Use

import tensorflow as tf
import numpy as np
import string
import keras
import re

strip_chars = string.punctuation
strip_chars = strip_chars.replace("[", "")
strip_chars = strip_chars.replace("]", "")


def custom_standardization(input_string):
    lowercase = tf.strings.lower(input_string)
    return tf.strings.regex_replace(lowercase, f"[{re.escape(strip_chars)}]", "")

portuguese_vocabulary_path = hf_hub_download(
    repo_id="AiresPucrs/transformer-eng-por",
    filename="keras_transformer_blocks.py",
    repo_type='model',
    local_dir="./")

from keras_transformer_blocks import TransformerEncoder, PositionalEmbedding, TransformerDecoder

transformer = keras.models.load_model("/content/transformer-eng-por/transformer-eng-por.h5",
    custom_objects={"TransformerEncoder": TransformerEncoder,
        "PositionalEmbedding": PositionalEmbedding,
        "TransformerDecoder": TransformerDecoder})

with open('portuguese_vocabulary.txt', encoding='utf-8', errors='backslashreplace') as fp:
    portuguese_vocab = [line.strip() for line in fp]
    fp.close()

with open('english_vocabulary.txt', encoding='utf-8', errors='backslashreplace') as fp:
    english_vocab = [line.strip() for line in fp]
    fp.close()


target_vectorization = tf.keras.layers.TextVectorization(max_tokens=20000,
                                        output_mode="int",
                                        output_sequence_length=21,
                                        standardize=custom_standardization,
                                        vocabulary=portuguese_vocab)

source_vectorization = tf.keras.layers.TextVectorization(max_tokens=20000,
                                        output_mode="int",
                                        output_sequence_length=20,
                                        vocabulary=english_vocab)

portuguese_index_lookup = dict(zip(range(len(portuguese_vocab)), portuguese_vocab))
max_decoded_sentence_length = 20


def decode_sequence(input_sentence):
    tokenized_input_sentence = source_vectorization([input_sentence])
    decoded_sentence = "[start]"

    for i in range(max_decoded_sentence_length):
        tokenized_target_sentence = target_vectorization([decoded_sentence])[:, :-1]
        predictions = transformer([tokenized_input_sentence, tokenized_target_sentence])
        sampled_token_index = np.argmax(predictions[0, i, :])
        sampled_token = portuguese_index_lookup[sampled_token_index]
        decoded_sentence += " " + sampled_token
        if sampled_token == "[end]":
            break
    return decoded_sentence


eng_sentences =["What is its name?",
                "How old are you?",
                "I know you know where Mary is.",
                "We will show Tom.",
                "What do you all do?",
                "Don't do it!"]

for sentence in eng_sentences:
    print(f"English sentence:\n{sentence}")
    print(f'Portuguese translation:\n{decode_sequence(sentence)}')
    print('-' * 50)

This will output the following:

English sentence:
What is its name?
Portuguese translation:
[start] qual é o nome dele [end]
--------------------------------------------------
English sentence:
How old are you?
Portuguese translation:
[start] quantos anos você tem [end]
--------------------------------------------------
English sentence:
I know you know where Mary is.
Portuguese translation:
[start] eu sei que você sabe onde mary está [end]
--------------------------------------------------
English sentence:
We will show Tom.
Portuguese translation:
[start] vamos ligar para o tom [end]
--------------------------------------------------
English sentence:
What do you all do?
Portuguese translation:
[start] o que vocês todos nós têm feito [end]
--------------------------------------------------
English sentence:
Don't do it!
Portuguese translation:
[start] não faça isso [end]
--------------------------------------------------

Intended Use

This model was created for research purposes only. Specifically, it was designed to translate sentences from English to Portuguese. We do not recommend any application of this model outside this scope.

Performance Metrics

Accuracy: 76,46%

Training Data

English-portuguese translation.

The dataset consists of a set of English and Portuguese sentences.

Limitations

In What is its name? to [start] o que é o seu nome [end], the transformer model makes a gender assumption, even though the source sentence wasn't gendered ([start] qual é o nome dele [end]). Errors like these are common in NLP, algorithmic bias being one of the great problems associated with using language models in real applications. In conclusion, we do not recommend using this model in real-world applications. It was solely developed for academic and educational purposes.

Cite as

@misc{teenytinycastle,
    doi = {10.5281/zenodo.7112065},
    url = {https://github.com/Nkluge-correa/teeny-tiny_castle},
    author = {Nicholas Kluge Corr{\^e}a},
    title = {Teeny-Tiny Castle},
    year = {2024},
    publisher = {GitHub},
    journal = {GitHub repository},
}

License

The transformer-eng-por is licensed under the Apache License, Version 2.0. See the LICENSE file for more details.