gemma-4-31b-indic-rys

Gemma 4 31B Indic SFT + gated Layer 1 (lambda=0.25).

Release status: public research/development candidate. It improved the inspected equal-benchmark macro by 1.945 percentage points over stock. The observed IndicParam regression was 1.364 points. This repository is published for reproducibility and further evaluation on Indic languages, not as a general replacement for the stock model.

This release combines:

1. a rank-8 multilingual and instruction-following LoRA trained on google/gemma-4-31B-it; and
2. one memory-sharing repetition of zero-based text layer 1, with only the second occurrence gated at lambda=0.25.

The final text stack has 61 virtual layers:

[0, 1, 1, 2, 3, ..., 59]

For the second occurrence of source layer 1:

h_out = h_in + 0.25 * (F(h_in) - h_in)

Both occurrences retain the checkpoint's trained Layer-1 scalar (0.0654296875) and use the same trained LoRA tensors. No scalar-root correction is applied.

Important: this is an adapter plus a runtime architecture transform, not a merged 31B checkpoint. Do not load it with plain AutoPeftModel.from_pretrained(...): that silently produces the SFT-only 60-layer model and omits the gated repetition. Use one of the included inference scripts.

Weight-loading contract

The base checkpoint is intentionally not duplicated in this repository. At runtime, each included loader constructs:

pinned Gemma 4 31B base + trained LoRA deltas + gated Layer-1 transform

Transformers attaches the original 60-layer LoRA before building its memory-sharing execution view. vLLM constructs the 61-layer graph first and attaches the audited, index-remapped LoRA from vllm_adapter/. These paths implement the same trained updates and gate used in evaluation.

Merging the LoRA would require publishing roughly a full 31B BF16 checkpoint and would still require custom code for the gate, because lambda=0.25 changes the layer forward pass rather than an ordinary parameter tensor. The adapter-plus-transform format is therefore the canonical release.

Repository contents

Path	Purpose
adapter_model.safetensors	Original 60-layer SFT LoRA used by the Transformers loader
adapter_config.json	PEFT rank-8 LoRA configuration
vllm_adapter/	Audited 60→61 remap used only by vLLM
gated_gemma4_transformers.py	Memory-sharing Transformers implementation
gated_gemma4_vllm.py	Native vLLM 0.23.0 implementation
inference_transformers.py	Minimal Transformers text-generation CLI
inference_vllm.py	Minimal vLLM text-generation CLI
evaluation/	Frozen-split scores, paired comparisons, and run metadata
training_metadata.json	Data, training, selection, and provenance metadata
release_manifest.json	SHA-256 and byte size of every release-managed file (excluding Hugging Face's generated .gitattributes)

Quick start: vLLM

The validated and recommended runtime is a single 96 GB NVIDIA GPU.

huggingface-cli download GarvinBhati/gemma-4-31b-indic-rys \
  --local-dir ./gemma-4-31b-indic-rys
pip install -r ./gemma-4-31b-indic-rys/requirements-vllm.txt
python ./gemma-4-31b-indic-rys/inference_vllm.py \
  --prompt "भारत में मानसून के महत्व को संक्षेप में समझाइए।"

The release was evaluated with:

• NVIDIA RTX PRO 6000 Blackwell Server Edition
• vLLM 0.23.0
• PyTorch 2.11.0+cu129
• CUDA 12.9
• Transformers 5.12.1
• BF16, max_model_len=6144

The full 5,242-row evaluation peaked at 88.83 GB allocated VRAM. Smaller generation queues do not duplicate model weights, but the 31B BF16 base model still requires a high-memory GPU.

Quick start: Transformers

Install a CUDA-compatible PyTorch build first. Google Colab images may include torchao==0.10.0, which is incompatible with PEFT 0.19.1 when present. This BF16 LoRA path does not use TorchAO, so remove that optional package before installing the pinned runtime:

# Required on Colab only when an incompatible preinstalled TorchAO is present.
pip uninstall -y torchao

huggingface-cli download GarvinBhati/gemma-4-31b-indic-rys \
  --local-dir ./gemma-4-31b-indic-rys
pip install -r ./gemma-4-31b-indic-rys/requirements-transformers.txt
python ./gemma-4-31b-indic-rys/inference_transformers.py \
  --prompt "தமிழில் இரண்டு வரிகளில் நீர் பாதுகாப்பை விளக்கவும்."

Restart the Python process after changing TorchAO or the pinned packages. If a previous model load failed with CUDA OOM, restart before retrying so that the notebook traceback cannot retain a partial model.

The Transformers implementation first injects the original LoRA into the 60-layer base model, then creates the gated, memory-sharing 61-layer execution view. This is semantically equivalent to the explicit 61-layer LoRA remap used by vLLM. The released benchmark outputs were generated with vLLM; the Transformers path has passed topology and shared-parameter validation but has not been used to reproduce the complete benchmark suite.

By default, the included loader temporarily disables Transformers 5.12.1's optional model-sized CUDA allocator warm-up. This avoids an unnecessary loading-time memory spike in notebooks and changes loading speed only—not weights, precision, topology, generation, or inference memory. Pass --enable-cuda-allocator-warmup to the CLI, or skip_cuda_allocator_warmup=False to load_model_and_processor(...), to restore the upstream loading optimization.

Both included examples are text-only. The underlying Gemma checkpoint is multimodal, but multimodal behavior of this fine-tune plus layer transform has not been evaluated.

Evaluation

All numbers below use the same frozen 5% manifest:

• MILU: 3,980 examples
• IndicParam: 660 examples
• IndicIFEval: 602 examples
• total: 5,242 examples

The equal-benchmark macro is the unweighted mean of the three displayed benchmark scores.

Model	MILU	IndicParam	IndicIFEval strict	Equal-benchmark macro
Stock Gemma 4 31B	75.879%	45.758%	61.794%	61.144%
Selected SFT only	76.382%	44.242%	66.113%	62.246%
SFT + Layer 1, λ=0.25	76.432%	44.394%	68.439%	63.088%

Observed changes:

• final versus stock: +1.945 percentage points on the equal-benchmark macro;
• final versus SFT-only: +0.842 macro points;
• final versus stock on IndicIFEval strict: +6.645 points; and
• final versus stock on IndicParam: -1.364 points.

The final-versus-SFT paired differences did not survive Holm correction across the three benchmarks. The stock IndicIFEval reference also used max_tokens=2048, while the SFT and final candidates used max_tokens=4096; the SFT-only comparison is therefore the cleaner architecture ablation.

Evaluation status

These results are development evidence, not an untouched final test. The 5% split was inspected during method development and gate selection. Independent evaluation on an untouched split is still required and we hope community will evaluate this on more benchmarks.

Training

The adapter was trained for one epoch with assistant-only loss:

Setting	Value
LoRA rank / alpha / dropout	8 / 16 / 0
Target modules	410 language-layer attention and MLP projections
Maximum sequence length	1,024
Micro-batch / gradient accumulation	1 / 16
Learning rate	5e-5
Packed training blocks	1,367
Optimizer steps	86 (the final step used 7 of 16 blocks)
Packed compute-token slots	1,399,808

The actual block mixture was:

• 752 blocks (55.0%) from willhx/if_oracle_sft;
• 342 blocks (25.0%) from nvidia/Nemotron-RL-knowledge-mcqa; and
• 273 blocks (20.0%) from akoksal/muri-it-language-split.

Published dataset licenses and attribution:

Dataset	Published license	Source
willhx/if_oracle_sft	Apache-2.0	dataset card
nvidia/Nemotron-RL-knowledge-mcqa	CC BY 4.0	dataset card and attribution
akoksal/muri-it-language-split	Apache-2.0	dataset card and MURI paper

The frozen target manifest was used only as a prompt decontamination deny-list. No target answers or exact target evaluation rows were used for training. Pinned source revisions and the complete selection record are in training_metadata.json.

Intended use

This release is intended for:

• research on multilingual and Indic-language instruction following;
• reproducible study of gated layer repetition;
• text generation and evaluation on high-memory research hardware; and
• further fine-tuning where the architecture transform is retained.

It is not intended as a drop-in replacement for the stock model in high-stakes, safety-critical, legal, medical, or financial workflows.

Limitations

• IndicParam regressed relative to the stock checkpoint.
• The gate was selected on an already-inspected development split.
• Only the pinned Transformers and vLLM versions have been structurally validated.
• Text inference was evaluated; multimodal behavior was not.
• The adapter inherits limitations, biases, safety risks, and usage requirements from Gemma 4.
• Generated content can be incorrect, outdated, biased, or unsafe and should be independently verified.

Reproducibility and integrity

The release is locked to:

• base model: google/gemma-4-31B-it;
• base revision: 842da3794eaa0b77d5f08bae87a17459d91ff475;
• source adapter SHA-256: 069d935a301f23a3fa3a32c2c56484159bc7a7b41b1a3f81af060e4aa9f63aeb;
• vLLM runtime adapter SHA-256: 440b15c4dc8f0e3bb32fa87aa39c991e67925479ee7f41394a1795ea13d1cbf4;
• repeated source layer: zero-based layer 1;
• gate: 0.25; and
• scalar policy: trained_scalar_on_both_gate_second_only.

Run the local integrity check before uploading or after downloading:

python validate_release.py

License

The release is provided under Apache 2.0. Use of the base checkpoint remains subject to the Gemma 4 license and terms linked in the model-card metadata. Training-data licenses and attribution are listed above; in particular, the NVIDIA MCQA source is credited under CC BY 4.0.