codewarnab/qwen2.5-coder-7b-latex-sft

Qwen2.5-Coder-7B LaTeX SFT

A LoRA adapter fine-tuned by Arnab Mondal (codewarnab) for generating complete, long-form LaTeX documents and TikZ diagrams. This repository contains the adapter and its saved tokenizer, not the full base-model weights or the training dataset.

Full training write-up: How I Fine-Tuned Qwen to Output Production-Ready LaTeX Documents

Base model and training

The adapter was trained with Unsloth on a 4-bit Qwen2.5-Coder-7B-Instruct base. Its saved configuration references unsloth/qwen2.5-coder-7b-instruct-bnb-4bit, a quantized distribution of Qwen/Qwen2.5-Coder-7B-Instruct.

• Method: QLoRA supervised fine-tuning, with loss on assistant responses.
• LoRA rank: 16; alpha: 32; dropout: 0; bias: none.
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj.
• Training sequence limit: 32,768 tokens.
• Training data: complete LaTeX documents checked for compilation; the dataset is private.

The intended output is a complete document, including its document class, packages, body, and closing document environment. It is not tuned primarily for short snippets.

Evaluation

The accompanying held-out evaluation recorded 156 compilable outputs from 176 generated documents (88.6%). There were 20 compilation failures and one additional held-out prompt without a generated document. Successful outputs had a median length of about 32 pages and 16.6k output tokens.

Those results used adapters merged into BF16 base weights, served through vLLM on an L40S with a 40,960-token context and YaRN factor 1.25. They are not an independent evaluation of every possible adapter-loading or quantization configuration. Training used a 32k sequence limit; quality beyond that length was not separately established.

Browse the evaluation PDFs.

Loading with Transformers and PEFT

Install PyTorch suitable for your GPU, then install transformers, accelerate, and peft>=0.20.0. This example loads the original BF16 base explicitly; it needs enough memory for that model and its generation cache.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

adapter_id = "codewarnab/qwen2.5-coder-7b-latex-sft"
base_id = "Qwen/Qwen2.5-Coder-7B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
base = AutoModelForCausalLM.from_pretrained(
    base_id,
    dtype=torch.bfloat16,
    device_map="auto",
)
model = PeftModel.from_pretrained(base, adapter_id)
model.eval()

messages = [
    {
        "role": "system",
        "content": (
            "You are a LaTeX/TikZ expert. When asked to create a document or diagram, "
            "output a complete, compilable LaTeX document. Always include "
            "\\documentclass, required packages, and \\end{document}."
        ),
    },
    {
        "role": "user",
        "content": "Write a complete textbook chapter on Fourier series with worked examples and TikZ figures.",
    },
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

# The base configuration's 32k context includes both prompt and generated output.
max_new_tokens = min(32768, 32768 - inputs.input_ids.shape[-1])
with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=max_new_tokens,
        do_sample=False,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.eos_token_id,
    )
latex = tokenizer.decode(output[0, inputs.input_ids.shape[-1]:], skip_special_tokens=True)
print(latex)

Use the exact system message above and the saved chat template to match training. Short generation budgets can truncate long documents before closing their environments. The example does not enable the extended-context evaluation configuration.

Limitations

• Compilation can still fail because of missing packages, undefined commands, syntax errors, or truncated output.
• A compilable PDF does not establish factual correctness, mathematical correctness, good pagination, or accurate diagrams.
• Long generations can repeat themselves or exceed the context limit.
• Content beyond the 32k training sequence length has not been independently validated.
• Compile generated source in an isolated environment with resource limits and shell escape disabled.

License and attribution

The adapter is released under Apache 2.0. The Qwen base model is also distributed under Apache 2.0. Its weights must be obtained separately. This release does not include or grant access to the private training dataset.