BananaMind-1.0-Instruct

BananaMind-1.0-Instruct is a full-finetuned GPT-2 XL instruction model trained on Alpaca-style instruction data.

It is our first Scaling Up Plan Model. We plan to do full pretraining in the future.

The model is based on openai-community/gpt2-xl and is intended for basic English instruction-following text generation.

It is our first usable chat model.

It includes a 1024 token context window.

📚 Training Data

For the Full Finetune, we used the Alpaca Cleaned Dataset for ~0.75 Epochs not full because it started plateauing.

Evaluation

Evaluations were run with lm-evaluation-harness using the Hugging Face model backend in bfloat16.

Benchmark Summary

Benchmark	Setting	Metric	Score
ARC-Easy	0-shot	acc	57.62%
ARC-Easy	0-shot	acc_norm	50.38%
ARC-Challenge	0-shot	acc	27.13%
ARC-Challenge	0-shot	acc_norm	28.24%
HellaSwag	0-shot	acc	40.33%
HellaSwag	0-shot	acc_norm	50.68%
PIQA	0-shot	acc	71.00%
PIQA	0-shot	acc_norm	70.13%
Winogrande	0-shot	acc	58.25%
MMLU	0-shot	acc	25.54%

Prompt Format

Use this format:

Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
{instruction}

### Response:

Example Usage

Install dependencies:

pip install -U transformers accelerate safetensors torch

Run inference:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

repo = "BananaMind/BananaMind-1.0-Instruct"

tokenizer = AutoTokenizer.from_pretrained(repo, use_fast=True)
tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(
    repo,
    dtype=torch.bfloat16,
    device_map="auto",
)

instruction = "Explain what photosynthesis is in simple terms."

prompt = f"""Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
{instruction}

### Response:
"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=140,
        do_sample=False,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

text = tokenizer.decode(output[0], skip_special_tokens=True)
print(text.split("### Response:", 1)[-1].strip())

Suggested Generation Settings

For stable answers:

• do_sample=False
• repetition_penalty=1.1
• max_new_tokens=80 to 160

For slightly more creative answers:

• do_sample=True
• temperature=0.4
• top_p=0.85
• repetition_penalty=1.15
• max_new_tokens=80 to 160

BananaMind-1.0-Instruct was trained as a full finetune rather than a LoRA adapter. The model was trained using an Alpaca-style instruction-response format.

Older training checkpoints may still be available under the checkpoints/ folder.

Samples

The sky can vary in color from blue to purple, depending on the time of day and the location. The sky can also be cloudy or clear, with stars visible through the clouds. The number of r’s in the word "strawberry" is 3. (yes i dont know how it knows this correctly) I am an AI assistant designed to assist users in various tasks and provide them with information, entertainment, and assistance. 1 + 1 is equal to 2. Here's a simple Python script that will print "Hello World" to the console:

#!/usr/bin/env python

import time

print("Hello World")

License

Apache 2.0