BackyardLabs/Eluwa-1.3b

Eluwa: A Conversational LoRA for Facebook's OPT Architecture

Eluwa is a fine-tuned Low-Rank Adapter (LoRA) model for Facebook's OPT 1.3b, 2.7b and 6.7b architectures. It is trained on the Stanford Alpaca dataset.

This repository contains the Eluwa 1.3b 2 epoch model, which represents a significant improvements in question-answering ability compared to the default OPT 1.3b model. Below are the results of Vicuna-style testing: 80 questions in various categories, with the responses rated by GPT-4.

Model	OPT 1.3b base	Eluwa 1.3b 1000 iter	Eluwa 1.3b 2 epoch
Generic	53	58	59
Knowledge	71	69	78
Roleplay	32	44	54
Common sense	41	55	57
Fermi	21	19	17
Counterfactual	28	29	42
Coding	16	7	8
Math	3	3	3
Writing	22	11	54
Total	287	295	372

Within the bounds of our testing, it was, in fact, suprisingly good, exceeding its 2.7b cousin.

A csv of questions, answers and GPT's reviews are also included in this repo in the /TestResults/ folder of the Eluwa github repo, along with the base model for comparison.

Using Eluwa

I used oobabooga's text generation UI for testing, because it lets me easily regenerate outputs, modify the conversation history passed to the model, and mess with parameters.

To load Eluwa, download OPT 1.3b from Huggingface and download both the .bin and .json file from the /model folder on this Github. Follow the instructions on the text generation UI repository to figure out where the model goes and how to load a LoRA. Eluwa goes in the /loras folder.

Training and notes

Training Eluwa is a straightforward process. It is essentially Facebook's GPT-like OPT 6.7b model, loaded in 8-bit and trained using Stanford's Alapaca dataset. The training code is available on the Eluwa github repo and will as-is in Google Colab.

Why "Eluwa"?

Well, the whole thing was inspiration from Alpaca, which is a LoRA based on Llama. Others adopted the trend (Cabrita, Vicuna etc). Now, in Sri Lanka, we don't have llamas (at least, I've never seen any), but we do have goats. Goats are spectacular animals. In Ragama I once beheld a goat fighting a pack of stray dogs (and winning). Then it came for me. I hit it on the head with my umbrella, whereupon which it ate the umbrella and chased me the length and breadth of the entire village.

If you can't beat em, join em. "Eluwa" means goat. Goats are fearsome, versatile, and double as the essential ingredient in mutton rolls. Everything in the known universe is either a goat, or not a goat. They're not as nice as llamas or alpacas, but they'll do.

License

Facebook's OPT has its own license. Please read it here. Alpaca is licensed for research use only. The dataset is CC BY NC 4.0 (allowing only non-commercial use) and they note that models trained using the dataset should not be used outside of research purposes.

Eluwa, therefore, is only for research and non-commercial use, under CC BY NC 4.0. Go experiment with it, but don't use it commercially. This applies to the testing dataset.