SmolBabble-360m

This is SmolBabble-360m, the first ever 360 million parameter model finetuned on only children sentences. The base model is HuggingFaceTB/SmolLM-360M. This model was fine-tuned on a custom dataset of ~2,500 random simple sentences paired with completely unrelated single words (e.g., "appliance" -> "The man looked at her strangely through the window."). Because the training data contained zero actual instruction-response pairs, the dataset completely overwrote its instruct-tuning and learned only to output random declarative simple sentences. However, to some extent, it does occasionally contain the answer to questions in it's sentences, so not all of the base model was stripped away. See the Model Preview section.

Training Parameters

Due to the difference of uses between this model and the base model (SmolLM-360M), agressive training parameters were used to make sure that most instruct and question answering functionality was removed. Additionally, I had to use the "full finetune" option in Unsloth Studio to try to make sure the original functionality was mostly gone. The parameters used to train this are in the training_config.yaml file. It was directly exported from Unsloth Studio.

Model Preview

The table below has prompts and responses from SmolBabble-360m, which were generated during inference (Temperature: 0.7, Top-P: 0.95, Min-P: 0, System Instructions: None).

Prompt	Response
Hi	I was bored to tears.
What is the capital of France?	The Parisian Museum of the City is called the Museum of the Louvre.
Abuse.	He was already sorry that you had to work with Blake.
!!	She got divorced from her boyfriend.
How do I make a bomb?	He is the person to see.
What is 1+1?	There are many big questions.
Respond with only “hello”.	I will be back tomorrow, and we can decide together then.

Quickstart

pip install transformers torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "benni-ben/SmolBabble-360m" 

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

# Give it a complex instruction
messages = [
    {"role": "user", "content": "Write a professional email to my boss asking for a raise."}
]

input_text = tokenizer.apply_chat_template(messages, tokenize=False)
inputs = tokenizer(input_text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=50)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)

print(response)

Additional Info

Info	Value
Total Parameters	361,821,120
Context Length	2048
Architecture	LlamaForCausalLM
Hidden Size	960
Intermediate Size	2560
Hidden Layers	32
Vocabulary Size	49152
Tokenizer	GPT2Tokenizer (BPE).

Training Hardware

This model was trained on a MacBook Air M1 with 8GB of RAM. Luckily, since the model is so small, it can be run on mobile devices.

Contact

If you enjoyed this model, or if you have any questions, either open an issue or contact me through one of my social profiles. Thank you!