SASVAAI/gemma-4-26b-jUnitTestGeneration

Gemma-4-26B-A4B JUnit Test Generator

Given a Java focal method, a skeleton of its class, and the existing test class the new test will be inserted into, emits one JUnit test method starting with @Test. A test counts as correct only if it compiles in the real project, passes against the real implementation, and kills at least one PIT mutant of the focal method.

This repository holds two forms of the same model:

Where	What	Use it when
repository root	google/gemma-4-26B-A4B-it with the LoRA merged in, bfloat16, 11 safetensors shards (~51.6 GB)	you want one directory to load or serve, e.g. with vLLM
adapter/	the LoRA adapter itself (35.5 MiB, QLoRA-trained) plus the training run's state, results and every validation prediction with its harness verdict	you already have the base model, or need the base in 4-bit on one GPU

The adapter was trained with QLoRA (4-bit NF4 base, bf16 compute) via TRL SFT and merged with PeftModel.merge_and_unload. Every number in this card was measured on the merged weights at the root, served with vLLM, and graded by building and running each test inside its own Maven project.

The headline number assumes you can run the tests. 0.503 is pass@5: the share of 332 held-out focal methods for which at least one of five candidates (one greedy, four sampled at temperature 0.8) compiles, passes and kills a mutant. A single greedy answer succeeds on 0.247. Use 0.503 only if your deployment generates several candidates, runs them, and keeps one that passes; otherwise the honest figure is 0.247. The untuned base model was never scored on these rows, so this card does not establish how much of either number the fine-tune is responsible for. Read How these values were chosen and Evaluation before quoting anything.

Model details

Developed by	SASVA AI Model Cognition Labs (MCL) Team
Base model	google/gemma-4-26B-A4B-it
Base revision	4d7ae4984b7db7de8f8457170b3f1a419ee76d52
Base parameters	25,805,936,206 total (Hub safetensors metadata); mixture-of-experts, ~4B active per token
Architecture family	gemma4 (Gemma4ForConditionalGeneration; text tower with 30 layers, 128 experts, hidden size 2816, vocabulary 262,144)
Adaptation	LoRA (r=16, alpha=32, dropout=0.05, rsLoRA off, DoRA off), merged into the root weights; adapter kept under adapter/
Trainable modules	q_proj, k_proj, o_proj, gate_proj, up_proj, down_proj on all 30 layers; v_proj on the 25 sliding-window layers (see below)
Excluded modules	.*vision_tower.* (the vision tower is untouched; training and evaluation were text-only)
Training method	qlora (--load-in-4bit, run 14 / trial 1)
Refinement	none
Precision	training: 4-bit NF4 base with double quantisation, bf16 compute; adapter saved in bfloat16; root weights: bfloat16 merge
Language	English prompts, Java output
License	Apache-2.0 (inherited from the base model); training data methods2test is MIT

Trainable parameters: 18,585,600 across 205 modules, 0.072% of the base model's parameters. adapter/adapter_model.safetensors is 37,232,104 bytes (410 tensors, lora_A + lora_B per module, all bfloat16). Every tensor sits under base_model.model.model.language_model.

One Gemma 4 structural fact shapes the module list. The 30 text layers alternate 5 sliding-window attention layers with one full-attention layer, so layers 5, 11, 17, 23 and 29 are global attention, and the global layers have no v_proj weight. PEFT therefore attached v_proj adapters to 25 layers and the other six projections to 30. The tensor counts match: 60 per projection for 30 layers × A/B, 50 for v_proj.

Intended use

Direct use. Draft a unit test for one Java method, for a developer to review, run and keep. The model is most useful inside a loop that compiles and runs its output: generate several candidates, run them, keep one that passes. That loop is what pass@5 measures.

The model was trained on a specific prompt shape and that shape is part of the contract:

• System prompt (verbatim, from the training and evaluation command lines): "You are a Java test generation expert. Given a focal method with its class skeleton and target test class, write a single JUnit test method (starting with @Test) that compiles, passes against the correct implementation, and is designed to kill a PIT mutant by testing boundary conditions, edge cases, and specific logic branches. Output only the raw Java test method starting with @Test, with no explanation or surrounding text."
• User turn: the instruction, a blank line, then the context block inside a bare ``` fence (instruction + "\n\n```\n" + context + "\n```"). The instruction is, with the three names filled in: "Write one JUnit test method for the focal method {focal_method} of class {focal_class}. The method will be inserted into the existing test class {test_class} shown after the focal class, so use only what is visible there and in the focal class. Output only the Java test method, starting with @Test. No class declaration, no imports, no explanation."
• The context block is, in this order:
  1. // Focal class: <path> and the focal class skeleton: fields, every method's signature, and the full body of the focal method only.
  2. A blank line, // Test class: <path>.
  3. // Java source level: <level>, with a reminder of what the level forbids for 1.5 to 1.7 (for example 1.6 (no diamond operator <>, lambdas, method references, try-with-resources, multi-catch or switch on String)). Every evaluation row has this line; 38% of training rows have no level because their parent POMs no longer resolve.
  4. The test class's import lines as they are at the project's commit.
  5. The test class skeleton: its declaration and fields, no test methods.
• Applied through the tokenizer's chat template (chat_template.jinja, shipped in this repo) with add_generation_prompt=True. Do not concatenate strings by hand.
• The output is one Java method starting with @Test, meant to be inserted into the test class as-is. It may only use imports already in the test class.
• Greedy decoding for a single answer; the four extra pass@5 candidates were sampled at temperature=0.8, top_p=0.95. Scored with max_new_tokens=1024.

Out-of-scope use. Generating whole test classes or test suites. Projects that are not Maven, or that need a JDK newer than the one you compile with. Accepting a generated test without running it. Using "the test passes" as evidence that the code under test is correct: the model writes tests that pass against the current implementation, bugs included.

How to get started

The prompt pieces are the same in every path. The example is validation row 292 (ArrayMatcher.matches, from the arquillian/algeron project).

SYSTEM = (
    "You are a Java test generation expert. Given a focal method with its class "
    "skeleton and target test class, write a single JUnit test method (starting with "
    "`@Test`) that compiles, passes against the correct implementation, and is designed "
    "to kill a PIT mutant by testing boundary conditions, edge cases, and specific logic "
    "branches. Output only the raw Java test method starting with `@Test`, with no "
    "explanation or surrounding text."
)
INSTRUCTION = (
    "Write one JUnit test method for the focal method `matches` of class `ArrayMatcher`. "
    "The method will be inserted into the existing test class `ArrayMatcherTest` shown "
    "after the focal class, so use only what is visible there and in the focal class. "
    "Output only the Java test method, starting with @Test. No class declaration, no "
    "imports, no explanation."
)
context = """// Focal class: pact/provider/core/src/main/java/org/arquillian/algeron/pact/provider/core/ArrayMatcher.java
class ArrayMatcher {
    public static final Optional<String> firstMatch(String[] patternArray, String value);

    public static final boolean matches(String[] patternArray, String value) {
        return firstMatch(patternArray, value).isPresent();
    }
}

// Test class: pact/provider/core/src/test/java/org/arquillian/algeron/pact/provider/core/ArrayMatcherTest.java
// Java source level: 1.8
import org.junit.Test;
import static org.assertj.core.api.Assertions.assertThat;

class ArrayMatcherTest {
}"""
messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": f"{INSTRUCTION}\n\n```\n{context}\n```"},
]

1. Fused weights with transformers (about 52 GB of accelerator memory in bfloat16):

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

REPO = "SASVAAI/gemma-4-26b-jUnitTestGeneration"
tokenizer = AutoTokenizer.from_pretrained(REPO)
model = AutoModelForCausalLM.from_pretrained(REPO, dtype=torch.bfloat16, device_map="auto")
model.eval()

inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)
out = model.generate(**inputs, max_new_tokens=1024, do_sample=False)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True).strip())

2. Fused weights with vLLM (how the numbers below were produced; the tensor-parallel degree must divide the 16 attention heads):

pip install "vllm>=0.19.1"
vllm serve SASVAAI/gemma-4-26b-jUnitTestGeneration --tensor-parallel-size 4 --max-model-len 10240

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
# one greedy answer
r = client.chat.completions.create(
    model="SASVAAI/gemma-4-26b-jUnitTestGeneration",
    messages=messages, temperature=0, max_tokens=1024,
)
# four more candidates, as in pass@5
extra = client.chat.completions.create(
    model="SASVAAI/gemma-4-26b-jUnitTestGeneration",
    messages=messages, temperature=0.8, top_p=0.95, n=4, max_tokens=1024,
)
candidates = [r.choices[0].message.content] + [c.message.content for c in extra.choices]

3. Adapter on the base model (base in 4-bit fits one 24 GB-class GPU, about 16 GB):

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE = "google/gemma-4-26B-A4B-it"
REPO = "SASVAAI/gemma-4-26b-jUnitTestGeneration"
tokenizer = AutoTokenizer.from_pretrained(REPO, subfolder="adapter")
model = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, REPO, subfolder="adapter")
model.eval()
# then generate exactly as in path 1

For the 4-bit base pass quantization_config=BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16), which reproduces the training-time numerics. vLLM cannot attach a LoRA to this architecture (its Gemma4ForConditionalGeneration does not list LoRA support), which is why the fused weights are at the root.

The greedy output recorded for this row during evaluation (vLLM, fused weights). It compiled, passed, and killed 1 of the focal method's 2 mutants:

@Test
public void shouldMatchWhenValueIsPresentInPatternArray() {
    assertThat(ArrayMatcher.matches(new String[]{"a", "b", "c"}, "b")).isTrue();
}

Decoding matters. pass@1 was scored greedily and pass@5 added four samples at T=0.8, top_p 0.95, all with max_new_tokens=1024 through the chat template. No other sampling setting was validated.

Training details

Data. 5,488 training rows and 332 validation rows built from methods2test (MIT), through andstor/methods2test_raw (revision 1a0f08c6), _meta (ae2bdff5) and _runnable (ed8cded6), by a deterministic script (autocatalyst.datagen.methods2test, seed 0). No model generated any training content: every target is a test a developer wrote in an open-source project.

Row construction, as recorded in the builder's manifest:

• Training rows come from methods2test's validation split: 44,298 rows scanned, 26,752 eligible, at most 25 rows per project, 5,488 kept from 394 projects. Dropped: 10,754 with no resolvable commit, 2,684 tests without an assertion, 225 constructor focal methods, 3,361 inputs over 7,000 characters, 522 targets over 2,500 characters, 141 inputs that grew too long once the test class's imports were added.
• Validation rows come from methods2test's test split, restricted to the _runnable ids (Maven projects that build): 78,388 scanned, 2,903 eligible, at most 6 rows per project, 400 requested. Each candidate's reference test was then run through the full harness below; 75 were rejected (23 did not compile on the pinned toolchain, 24 killed no mutant, 9 failed, 19 PIT errors) and 332 kept from 86 projects. Every validation reference passes all three gates, so a perfect model scores 1.0.
• No project appears in both splits (394 training projects, 86 validation projects, overlap 0).
• Input never contains the reference test or any other test method of the test class.
• Java levels in validation: 1.8 (149 rows), 1.7 (80), 1.6 (72), 1.5 (31).

Train samples	5,488 (394 projects)
Validation samples	332 (86 projects)
Project overlap	0
Prompt format	chat template + system prompt + instruction / fenced-context user turn (see Intended use)
Loss masking	answer tokens only; prompt tokens set to -100
Truncation	sequences cut to max_seq_len 4096; no sample was dropped for length

Source-code licences. methods2test is MIT, but each row's focal class and test come from a third-party Java project with its own licence. The licence field in andstor/methods2test_raw reads only licensed for 391 of the 394 training projects (2 are Apache-2.0, 1 "Other") and for 85 of the 86 validation projects (1 is GPL-3.0), so per-project licences were not verified for this release. Review this before a public release.

An LLM (Claude Opus 4.6, claude-opus-4-6, via an internal inference gateway) proposed the hyperparameters the search tried and wrote the system prompt from the project's problem statement. It generated no training content and computed no metric; every metric was computed by compiling and running Java.

Method

SFT method	qlora
Base quantisation during training	4-bit NF4, double quantisation, bf16 compute (bitsandbytes)
Refinement stage	none
Auto class	AutoModelForCausalLM (resolves to Gemma4ForConditionalGeneration)
Hardware	6x NVIDIA H100 80GB HBM3, torchrun --nproc_per_node=6

The project allowed one method (qlora). No refinement stage ran; the published adapter is the SFT adapter and the root weights are its merge.

Final hyperparameters

Hyperparameter	Value	Source
learning_rate	0.0002	[TRAIN] cmdline
lr_scheduler_type	cosine	[TRAIN] cmdline
num_train_epochs	2	platform hyperparameter record, adapter/trainer_state.json
per_device_train_batch_size	1	platform hyperparameter record, adapter/trainer_state.json
gradient_accumulation_steps	4	platform hyperparameter record
max_seq_length	4096	platform hyperparameter record
warmup_ratio	0.05	platform hyperparameter record
weight_decay	0.01	platform hyperparameter record
loraplus_lr_ratio	1.0 (off)	platform hyperparameter record
lora_r / lora_alpha / lora_dropout	16 / 32 / 0.05	[TRAIN] cmdline, adapter/adapter_config.json
use_rslora / use_dora	false / false	adapter/adapter_config.json
target_modules	the 7 listed in Model details	adapter/adapter_config.json
load_in_4bit	true (NF4, double quant, bf16 compute)	[TRAIN] cmdline

Effective batch size: 24 (1 x 4 x 6). Optimizer steps: 458 (229 per epoch).

neftune_noise_alpha (0.0), use_dora, use_rslora and lora_init (default) were left at their no-op defaults. KD parameters are omitted deliberately: this is a qlora run, not a distillation run.

Provenance note: every value above was recovered from the platform database (runs, experiments, events tables for run 14) and cross-checked against the [TRAIN] command line recorded in the run log and against the shipped adapter/adapter_config.json. The logged command line is cut off after --epochs; the values after it come from the run's recorded hyperparameter JSON.

How these values were chosen

These hyperparameters were selected by an automated search (autocatalyst.cli.run_autoresearch): an agent proposes one change at a time, runs train then eval, and keeps or discards on junit_pass_at_k (higher is better).

Run 14 ran 6 trials in 12 h 25 m (2026-10-09 18:09 to 2026-10-10 06:35 UTC); all 6 scored. This checkpoint is trial 1, the search's starting configuration, and no later trial beat it. Every trial trained on the same 5,488 rows and was scored on the same 332 validation rows.

#	r	Dropout	LR	Epochs	Train	Eval	pass@5	Kept
1	16	0.05	2e-4	2	92 m	29 m	0.503012	yes
2	16	0.05	2e-4	3	137 m	30 m	0.445783	no
3	16	0.05	1e-4	2	92 m	14 m	0.466867	no
4	32	0.05	2e-4	2	92 m	15 m	0.481928	no
5	16	0.10	2e-4	2	92 m	15 m	0.430723	no
6	16	0.05	3e-4	2	92 m	43 m	0.478916	no

All trials: QLoRA, lora_alpha = 2 x r, cosine schedule, batch 1 per device, gradient accumulation 4, warmup 0.05, weight decay 0.01, max_seq_len 4096, LoRA+ off, rsLoRA and DoRA off.

What the search actually established: nothing beyond noise. Five single changes from the starting point (one more epoch, half or 1.5x the learning rate, rank 32, dropout 0.10) all scored lower, between 0.431 and 0.482. pass@5 mixes four temperature-0.8 samples into each row's result, so two evaluations of the same weights differ from run to run; the spread of these six trials (0.43 to 0.50) is the best available estimate of that noise. Treat trials within about 0.05 of each other as tied, and read 0.503 as a high draw.

The previous search on this task. Run 12 (2026-10-08) searched the same space on an earlier version of the dataset (333 validation rows, no Java level line in the prompt) and kept on greedy pass@1. Its best checkpoint scored pass@1 0.276. In a separate check that sampled it five times at temperature 0.8, it reached pass@5 0.486 (0.505 counting its greedy answer too). So this checkpoint is not better than run 12's: its greedy pass@1 is lower (0.247), and its pass@5 is about level. The jump from "0.276" to "0.503" between the two runs is the change of metric, not of model.

Search space. Four knobs were varied (LORA_R, LORA_DROPOUT, LEARNING_RATE, EPOCHS). LR_SCHEDULER, GRAD_ACCUM, WARMUP_RATIO, WEIGHT_DECAY, MAX_SEQ_LEN, LORAPLUS_LR_RATIO, USE_RSLORA, USE_DORA, BATCH_SIZE and the training method were never moved.

Observed training metrics (this checkpoint).

Final train loss (mean over the run)	0.7416771020431185
Last logged train loss (step 450)	0.5607 (grad norm 0.31, token accuracy 0.864)
Final eval loss (teacher-forced, answer tokens)	0.5725216269493103
Eval mean token accuracy	0.8667
Train runtime	5,444.8575 s
Total FLOPs	1.6958247325338173e+18
Throughput	2.016 samples/s, 0.084 steps/s

458 optimizer steps ran. The logged train loss fell from 5.8956 at step 10 (grad norm 8.0) to 0.5607 at step 450, with a minimum of 0.5260; the reported train loss is the mean over the run, not a converged value.

Evaluation

Protocol. All 332 validation rows, prompts rendered through the chat template, max_new_tokens=1024. The platform merged the adapter into the base and generated with vLLM 0.19.1 in bfloat16 at tensor-parallel 4 on four H100s: one greedy answer per row, then four more sampled at temperature 0.8, top_p 0.95. Generation plus grading took 1,722 s. The merged weights it served are the ones at this repository's root.

The harness (junit_mutation evaluator) grades each candidate inside its own project, never by comparing text:

1. Check out the project at the row's commit and insert the candidate method into the test class.
2. Compile: mvn test-compile (JDK 8, Maven 3.6.3).
3. Pass: run that one test method (mvn test -Dtest=Class#method); it must pass against the real implementation.
4. Kill: mutate the focal class with PIT 1.14.4 and run only the candidate test against the mutants; it must kill at least one mutant of the focal method.

A candidate scores 1 only if it clears all three gates. pass@1 is the greedy candidate's score; pass@5 counts a row as solved if any of the five candidates scores 1.

Metric	Value
pass@5, greedy + 4 samples (the search metric)	0.503012 (167 / 332)
pass@1, greedy	0.246988 (82 / 332)
Greedy compile rate	0.421687 (140 / 332)
Greedy test-pass rate	0.246988 (82 / 332)
Greedy mean kill fraction (killed / mutants of the focal method)	0.206093

Where the greedy answers fail (from adapter/predictions.jsonl):

Greedy outcome	Rows	Share
Does not compile	192	57.8%
…of which ran into the 1,024-token cap	66	19.9%
Compiles but the test fails	58	17.5%
Passes but kills no mutant	0	0%
Clears all three gates	82	24.7%

The 66 capped outputs are repetition loops; every one fails to compile. The median greedy answer is 119 tokens against a reference median of 105. Of the 58 test failures, 35 are failed assertions (AssertionError, ComparisonFailure, AssertionFailedError): the test ran but expected a different value from the one the code returns. The rest are exceptions thrown during the test. Every greedy test that passed also killed a mutant, so the mutation gate rejected nothing here.

Where the other 85 pass@5 wins come from. Of the 167 solved rows, 82 were solved by the greedy answer and 85 only by a sampled one (attempt 2: 33, 3: 21, 4: 15, 5: 16).

Baseline for comparison. Not measured. The untuned google/gemma-4-26B-A4B-it was never scored on these 332 rows, so nothing here quantifies how much of the score the fine-tuning is responsible for. This is the most important gap in this card.

This is a validation split the search selected against. 6 trials were scored on these same 332 rows and the best was kept, so expect optimistic bias on top of the sampling noise already described. The rows come from projects that do not appear in training, but from the same methods2test snapshot, so they are unseen projects, not code written after the training data.

The evaluation set is reproducible. adapter/predictions.jsonl holds every one of the 332 rows: instruction, context, greedy prediction, reference test, and the harness verdict (stage, compiler or surefire error text, mutants, kills, per-attempt stages and the winning candidate). Local paths in error text are replaced with <project>/.

Limitations and bias

pass@5 needs a test runner. Without one, you get the greedy answer and pass@1 0.247: three of four single answers fail.

No baseline, so no established gain. See Evaluation.

One in five greedy answers never stops. Repetition loops consume the whole 1,024-token budget and never compile. Treat a capped output as a failure.

Most failures are compile errors. Using a class or method that is not visible, an import the test class lacks, or a language feature above the project's Java level. Keep the // Java source level: line and the test class's imports in the prompt; they are what the model uses to avoid this.

Passing tests encode current behaviour. The model writes tests that pass against the implementation as it is. If the method has a bug, a passing test locks the bug in.

Killing one mutant is a low bar. The gate needs one killed mutant, not all; the mean kill fraction is 0.21. A passing test is not evidence of thorough coverage.

Prompt shape is the contract. Change the system prompt, the instruction sentence, the context block's order, or the code fence, and you are evaluating a model nobody measured.

Domain narrowness. Maven projects from GitHub mined by methods2test, Java 1.5 to 1.8 in evaluation (training also saw 9, 10 and 11), with the JUnit 4, JUnit 5 or TestNG setup and assertion libraries those projects use. Gradle builds, newer Java, Kotlin and non-English identifiers are unmeasured.

Inherits all biases and limitations of the base model. This adapter changes 0.07% of the parameters and was not evaluated for safety or fairness. The base model's own card governs those properties.

Merged-weights equivalence

The root weights are adapter/ merged into the base in bfloat16 (PeftModel.merge_and_unload through autocatalyst.serving.merge, transformers 5.6.2). The merge was written as one 49 GB shard plus one 2.3 GB shard; for this release its tensors were copied unchanged into 11 shards of at most 5 GB (same tensor names, dtypes and values; 26 sampled tensors were checked equal to the source). They are the weights the evaluation above was served from, so the numbers in this card are the fused model's numbers. The merged directory keeps the untouched vision tower and the processor config so it loads as the base does.

Environmental impact

Hardware	6x NVIDIA H100 80GB HBM3
Training time	90.7 minutes (5,444.86 s)
Cloud provider / region	on-premise

Covers this trial only. The full 6-trial search that selected it took 12 h 25 m on the same hardware.

Framework versions

• PEFT 0.18.1
• TRL: 1.0.0
• Transformers: 5.7.0.dev0 (git main) for training; 5.6.2 for the merge
• Pytorch: 2.5.1+cu121
• bitsandbytes: 0.49.2
• flash-attn: 2.8.3
• vLLM: 0.19.1 (evaluation)
• Evaluation toolchain: JDK 8, Maven 3.6.3, PIT 1.14.4
• Python: 3.12.3

PEFT's version is the one recorded in adapter/adapter_config.json at save time; the rest are the pinned versions of the training and evaluation environments. transformers is a git-main build: the gemma4 architecture is not in the stable PyPI release used for training.

Citation

@misc{gemma4_junit_test_generation_2026,
  title  = {Gemma-4-26B-A4B JUnit Test Generator},
  author = {{SASVA AI Model Cognition Labs}},
  year   = {2026},
  url    = {https://huggingface.co/SASVAAI/gemma-4-26b-jUnitTestGeneration}
}

Training and evaluation data: methods2test, Tufano et al., "Methods2Test: A dataset of focal methods mapped to test cases", MSR 2022, https://github.com/microsoft/methods2test (MIT).