language: pt
license: cc-by-nc-4.0
co2_eq_emissions: 710
QUOKKA
Model description
QUOKKA is the pioneering generative model for Portuguese from Portugal (PT-PT) to the best of our knowledge. Our model is a fine-tuned version of Phoenix that was released on 04/08/2023. The backbone of Phoenix is BLOOMZ, which was fine-tuned using a vast dataset consisting of 267k samples of instructions and 189k samples of conversations.
Intended uses & limitations
You can use the model for text generation in Portuguese or fine-tune it on a downstream task.
How to use
You can use this model directly with a pipeline for text generation:
from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline
checkpoint = "automaise/quokka-7b"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(checkpoint)
generator = pipeline("text-generation", model=model, tokenizer=tokenizer, device=0)
generator("<human>Olá, consegues ajudar-me?<bot>")
Limitations and bias
- Language: the model was fine-tuned on Portuguese data only and might not generalize appropriately to other languages.
- Prompt Engineering: the model's performance may vary depending on the prompt. We recommend writing clear and specific instructions.
- Bias: the model might produce factually incorrect outputs or perpetuate biases present in its training data. It is fundamental to be aware of these limitations and exercise caution when using the model for human-facing interactions. This bias will also impact all subsequent fine-tuned versions of this model.
Training data
QUOKKA was fine-tuned on a dataset collected from different sources:
Initially, we used the Bactrain-X dataset, which involves the translation of 67k English instructions (52k from Alpaca and 15k from Dolly v2) into 51 languages using Google Translate API. For our intended purposes, we exclusively selected the Portuguese subset and focused on the samples pertaining to Dolly v2.
Then, we incorporated the Cabrita dataset that consists of a translation of Alpaca's training data. The Portuguese translation was generated using ChatGPT. Therefore, it is important to note that these translations may not be of the highest quality.
Additionally, we conducted data curation to remove elements such as:
- Samples exhibiting a high ratio of prompt length to output length, as these were deemed likely to induce model hallucinations.
- Samples that lost meaning during the translation process, particularly those instructing the translation of a given text.
As a result, our final dataset comprises 56k samples.
Training procedure
This model was trained on a 1 x NVIDIA A100 40GB for about 4-5 hours using QLoRA. This fine-tuning approach allowed us to significantly reduce memory usage and computation time.
Evaluation results
Environmental impact
Carbon emissions were estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019). The hardware, runtime, cloud provider, and compute region were utilized to estimate the carbon impact.
- Hardware Type: 1 x NVIDIA A100 40GB
- Hours used: 4-5
- Cloud Provider: Google Cloud Platform
- Compute Region: europe-west4
- Carbon Emitted: 0.71 kg eq. CO2