anjohn0077/NEXS-lora-adapters

NEXS LoRA Adapters — Serving & Evaluation Guide

Rank-128 LoRA adapters extracted with mergekit from domain fine-tunes and sanitized for vLLM serving (pure lora_A/lora_B weights only; modules_to_save tensors stripped, modules_to_save: null in configs; no resize_token_embeddings() needed). Two families:

• Llama-3.1-8B — five adapters (finance, legal, medical, toxicity, truthfulness), 224 A/B pairs each (32 layers × q/k/v/o/gate/up/down).
• Qwen3-32B — six adapters (instruction-following, medical ×2, theorem proving, Russian ×2), 448 A/B pairs each (64 layers × 7 projections).

Llama-3.1-8B adapters on the Hub

Domain	HF repo	Extracted from
finance	anjohn0077/NEXS-finance-lora	mukaj/Llama-3.1-Hawkish-8B
legal	anjohn0077/NEXS-legal-lora	MaziyarPanahi/calme-2.3-legalkit-8b
medical	anjohn0077/NEXS-medical-lora	TsinghuaC3I/Llama-3.1-8B-UltraMedical
toxicity	anjohn0077/NEXS-toxicity-lora	K-intelligence/Llama-SafetyGuard-Content-Binary
truthfulness	anjohn0077/NEXS-truthfulness-lora	HiTZ/Llama-3.1-8B-Instruct-multi-truth-judge

Index/manifest repo: anjohn0077/NEXS-lora-adapters

Qwen3-32B adapters on the Hub

Name	HF repo	Extracted from	Notes
IF	anjohn0077/NEXS-qwen3-32b-IF-lora	qihoo360/Light-IF-32B	instruction following
medical-openmedzoo	anjohn0077/NEXS-qwen3-32b-medical-openmedzoo-lora	OpenMedZoo/MedGo
medical-tachyhealth	anjohn0077/NEXS-qwen3-32b-medical-tachyhealth-lora	TachyHealth/Gazal-R1-32B-sft-merged-preview	fp32 tensors (4.3 GB)
prover	anjohn0077/NEXS-qwen3-32b-prover-lora	Goedel-LM/Goedel-Prover-V2-32B	not yet benchmarked
russian-refalmachine	anjohn0077/NEXS-qwen3-32b-russian-refalmachine-lora	RefalMachine/RuadaptQwen3-32B-Instruct
russian-t-tech	anjohn0077/NEXS-qwen3-32b-russian-t-tech-lora	t-tech/T-pro-it-2.0

Serve the five Llama adapters with vLLM

Directly from the Hub (vLLM downloads the adapters itself; requires hf auth login or HF_TOKEN for the gated base model):

python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3.1-8B \
    --enable-lora \
    --lora-modules \
        finance=anjohn0077/NEXS-finance-lora \
        legal=anjohn0077/NEXS-legal-lora \
        medical=anjohn0077/NEXS-medical-lora \
        toxicity=anjohn0077/NEXS-toxicity-lora \
        truthfulness=anjohn0077/NEXS-truthfulness-lora \
    --port 8000 \
    --max-loras 5 \
    --max-lora-rank 128 \
    --gpu-memory-utilization 0.85

Or from the local sanitized copies (on this cluster):

python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3.1-8B \
    --enable-lora \
    --lora-modules \
        finance=/scratch/shared_dir/lora/llama/finance_sanitized \
        legal=/scratch/shared_dir/lora/llama/legal_sanitized \
        medical=/scratch/shared_dir/lora/llama/medical_sanitized \
        toxicity=/scratch/shared_dir/lora/llama/toxicity_sanitized \
        truthfulness=/scratch/shared_dir/lora/llama/truthfulness_sanitized \
    --port 8000 \
    --max-loras 5 \
    --max-lora-rank 128 \
    --gpu-memory-utilization 0.85

To fetch the Hub copies to a local directory instead:

for d in finance legal medical toxicity truthfulness; do
    hf download "anjohn0077/NEXS-${d}-lora" --local-dir "./adapters/${d}"
done

Serve the six Qwen3-32B adapters with vLLM

The base model is ~65 GB in bf16; one 141 GB H200 fits the model, all six adapters, and KV cache at --gpu-memory-utilization 0.85.

python -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen3-32B \
    --enable-lora \
    --lora-modules \
        IF=anjohn0077/NEXS-qwen3-32b-IF-lora \
        medical_openmedzoo=anjohn0077/NEXS-qwen3-32b-medical-openmedzoo-lora \
        medical_tachyhealth=anjohn0077/NEXS-qwen3-32b-medical-tachyhealth-lora \
        prover=anjohn0077/NEXS-qwen3-32b-prover-lora \
        russian_refalmachine=anjohn0077/NEXS-qwen3-32b-russian-refalmachine-lora \
        russian_t_tech=anjohn0077/NEXS-qwen3-32b-russian-t-tech-lora \
    --port 8000 \
    --max-loras 6 \
    --max-lora-rank 128 \
    --gpu-memory-utilization 0.85

Or from the local sanitized copies (on this cluster), replace each repo id with /scratch/shared_dir/lora/qwen/<dir>_sanitized (dirs: IF_sanitized, medical_OpenMedZoo_sanitized, medical_TachyHealth_sanitized, prover_sanitized, russian_RefalMachine_sanitized, russian_t-tech_sanitized).

Evaluate with lm-evaluation-harness

With a server running on port 8000, each adapter is addressed by the name given in --lora-modules. Llama family:

lm_eval --model local-completions \
    --model_args model=finance,base_url=http://localhost:8000/v1/completions,tokenizer=meta-llama/Llama-3.1-8B,num_concurrent=10 \
    --tasks mmlu_econometrics \
    --output_path results/vllm_finance

lm_eval --model local-completions \
    --model_args model=legal,base_url=http://localhost:8000/v1/completions,tokenizer=meta-llama/Llama-3.1-8B,num_concurrent=10 \
    --tasks mmlu_professional_law \
    --output_path results/vllm_legal

lm_eval --model local-completions \
    --model_args model=medical,base_url=http://localhost:8000/v1/completions,tokenizer=meta-llama/Llama-3.1-8B,num_concurrent=10 \
    --tasks mmlu_professional_medicine \
    --output_path results/vllm_medical

lm_eval --model local-completions \
    --model_args model=toxicity,base_url=http://localhost:8000/v1/completions,tokenizer=meta-llama/Llama-3.1-8B,num_concurrent=10 \
    --tasks sst2 \
    --output_path results/vllm_toxicity_sst2

lm_eval --model local-completions \
    --model_args model=truthfulness,base_url=http://localhost:8000/v1/completions,tokenizer=meta-llama/Llama-3.1-8B,num_concurrent=10 \
    --tasks truthfulqa_mc2 \
    --output_path results/vllm_truthfulness

Qwen3-32B family (note tokenizer=Qwen/Qwen3-32B):

lm_eval --model local-completions \
    --model_args model=IF,base_url=http://localhost:8000/v1/completions,tokenizer=Qwen/Qwen3-32B,num_concurrent=10 \
    --tasks ifeval \
    --output_path results/vllm_qwen_IF

lm_eval --model local-completions \
    --model_args model=medical_openmedzoo,base_url=http://localhost:8000/v1/completions,tokenizer=Qwen/Qwen3-32B,num_concurrent=10 \
    --tasks medqa_4options \
    --output_path results/vllm_qwen_medical_openmedzoo

lm_eval --model local-completions \
    --model_args model=medical_tachyhealth,base_url=http://localhost:8000/v1/completions,tokenizer=Qwen/Qwen3-32B,num_concurrent=10 \
    --tasks medqa_4options \
    --output_path results/vllm_qwen_medical_tachyhealth

lm_eval --model local-completions \
    --model_args model=russian_refalmachine,base_url=http://localhost:8000/v1/completions,tokenizer=Qwen/Qwen3-32B,num_concurrent=10 \
    --tasks m_mmlu_ru \
    --output_path results/vllm_qwen_russian_refalmachine

lm_eval --model local-completions \
    --model_args model=russian_t_tech,base_url=http://localhost:8000/v1/completions,tokenizer=Qwen/Qwen3-32B,num_concurrent=10 \
    --tasks m_mmlu_ru \
    --output_path results/vllm_qwen_russian_t_tech

(The prover adapter's intended minif2f evaluation has not produced results yet, so it ships unbenchmarked.)

Benchmark results

Accuracy of each LoRA served on the base model via vLLM, compared against the plain base model and the original full fine-tune it was extracted from.

Llama-3.1-8B family:

Domain	Benchmark	Base	LoRA (vLLM)	Full fine-tune
finance	mmlu_econometrics	0.4825	0.4561	0.5526
legal	mmlu_professional_law	0.4941	0.4915	0.4948
medical	mmlu_professional_medicine	0.7169	0.7059	0.7721
toxicity	sst2	0.6732	0.8991	0.8899
truthfulness	truthfulqa_mc2	0.4416	0.7164	0.7307

Note: toxicity and truthfulness recover nearly all (or more than) the fine-tune's gains; finance/medical/legal LoRAs land close to base — most of those fine-tunes' improvement lived in the full-rank embedding/head deltas that vLLM cannot serve and were stripped during sanitization.

Qwen3-32B family (from qwen32b_adapter_acc.csv):

Adapter	Benchmark	Base	LoRA (vLLM)	Full fine-tune
IF	ifeval	0.8336	0.2737	0.8669
medical-openmedzoo	medqa_4options	0.7494	0.7604	0.7596
medical-tachyhealth	medqa_4options	0.7494	0.7455	0.7478
russian-refalmachine	m_mmlu_ru	0.7517	0.7524	0.7370
russian-t-tech	m_mmlu_ru	0.7517	0.7542	0.7679
prover	minif2f	—	—	—

Notes: the IF LoRA scores far below base on ifeval (0.2737 vs 0.8336) — the low-rank approximation appears to break the fine-tune's instruction-following behavior rather than approximate it; it is published for completeness with these honest numbers. The medical and russian adapters track their full fine-tunes closely. prover has not been benchmarked (its minif2f run produced no results), but it passes structural validation and serves through vLLM.

Dependencies (for replication)

All extraction, sanitization, serving, and evaluation ran in a single Python virtualenv on Linux with NVIDIA H200 GPUs (driver 595.71.05). Exact versions:

Component	Version	Notes
Python	3.12.3
torch	2.9.1	CUDA 12.8 build (2.9.1+cu128)
mergekit	0.1.4	fork: ikhyunAn/mergekit @ e85a454, editable install — not upstream arcee-ai/mergekit
vllm	0.15.1
transformers	4.57.1
peft	0.15.2
huggingface-hub	0.35.3	pinned <1.0 (required by transformers 4.57.1)
safetensors	0.5.3
accelerate	1.6.0
tokenizers	0.22.1
xformers	0.0.32.post1
numpy	2.2.6
lm_eval	0.4.9.1	EleutherAI/lm-evaluation-harness @ ad3f4d0c, editable install

To recreate the core environment:

python3.12 -m venv lora-env && source lora-env/bin/activate
pip install torch==2.9.1 --index-url https://download.pytorch.org/whl/cu128
pip install vllm==0.15.1 transformers==4.57.1 peft==0.15.2 \
    "huggingface_hub==0.35.3" safetensors==0.5.3 accelerate==1.6.0
pip install -e "git+https://github.com/ikhyunAn/mergekit.git@e85a454#egg=mergekit"
pip install -e "git+https://github.com/EleutherAI/lm-evaluation-harness.git@ad3f4d0c#egg=lm_eval"

Caveats:

• Do not upgrade huggingface_hub past 1.0 in this environment — transformers 4.57.1 pins huggingface-hub<1.0 and both transformers and mergekit break at import time with 1.x.
• Extraction used a single GPU (CUDA_VISIBLE_DEVICES=<idx> with mergekit-extract-lora --cuda); each 8B extraction takes ~4 minutes on an H200 and peaks around 2×16 GB of downloaded weights in the HF cache.

Provenance / pipeline

1. Extraction: `mergekit-extract-lora --model --base-model --out-path --cuda` with base `meta-llama/Llama-3.1-8B` for the llama family (see `extract_and_upload_loras.sh`; raw outputs in `/scratch/shared_dir/lora/llama//`) and `Qwen/Qwen3-32B` for the qwen family (raw outputs in `/scratch/shared_dir/lora/qwen/`).
2. Sanitization (same script set in /scratch/shared_dir/lora/llama/ and /scratch/shared_dir/lora/qwen/): fix_adapters.py (drop norm layers) → fix_vllm_weights.py / final_clean.py (drop all non-lora_* keys, null out modules_to_save) → verify_all_clean.py. For llama, toxicity's extended vocab (128258) was truncated to the base 128256 first via fix_toxicity_vocab.py.
3. Upload: upload_sanitized_loras.py (llama) and upload_qwen_loras.py (qwen), both in this directory, push each sanitized adapter to its Hub repo with an updated README.