ILoveBuns/llama-3.3-70b-python-mental-execution-lora

Llama 3.3 70B Python Mental Execution LoRA

A PEFT/LoRA adapter trained for mentally executing self-contained Python 3 snippets, predicting exact stdout, and producing concise variable traces. This repository contains adapter and tokenizer artifacts only; it does not contain the Llama 3.3 base-model weights.

Base model

togethercomputer/Meta-Llama-3.3-70B-Instruct-Reference

Access to and use of the base model remain subject to the Meta Llama 3.3 Community License and the base-model provider terms.

Training

• Method: supervised fine-tuning (SFT)
• Adapter: LoRA
• Rank: 8
• Alpha: 8
• Dropout: 0
• Target modules: q_proj, v_proj
• Epochs: 1
• Learning rate: 1e-4
• Scheduler: cosine
• Training steps: 21
• Final validation loss: 1.048828125
• Dataset: 12,000 Python mental-execution prompt/completion pairs

Evaluation

Adaption reported a dataset win rate of:

• Base model: 45%
• Adapted model: 55%

This is a 10 percentage-point increase, or approximately 22.2% relative improvement over the base win rate.

A separate one-example sanity check was unsuccessful: for a list comprehension yielding [1, 4], the model incorrectly reported the sum as 8 rather than 5. The adapter should therefore be treated as experimental and outputs must be independently verified.

Usage

Load the compatible Llama 3.3 base model under its applicable access and license terms, then attach this repository with PEFT. Exact compatibility may depend on the reference checkpoint and tokenizer revision used by the training provider.

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_id = "togethercomputer/Meta-Llama-3.3-70B-Instruct-Reference"
adapter_id = "ILoveBuns/llama-3.3-70b-python-mental-execution-lora"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
base = AutoModelForCausalLM.from_pretrained(base_id, device_map="auto")
model = PeftModel.from_pretrained(base, adapter_id)

Limitations

• Exact-output reasoning remains unreliable.
• Training and platform evaluation used a small number of optimizer steps.
• The adapter may inherit limitations and risks of the base model.
• Do not use unverified outputs for grading, security decisions, or execution of untrusted code.

License

The adapter is distributed subject to the Meta Llama 3.3 Community License and any applicable base-model provider terms. No base-model weights are included.