Atlas3D/llama.cpp-sm70-cuda12.9

llama.cpp for NVIDIA Volta (sm_70)

Linux x86-64 binaries of llama.cpp built with CUDA 12.9 for NVIDIA Tesla V100 GPUs. CUDA 13.0 removed offline compilation and CUDA library support for Volta, so this build uses the CUDA 12.x toolchain. See the CUDA 13.0 release notes.

Status: released 2026-10-20 at the maintainer's decision, with one open caveat. All three fresh kernel suites (29,352 cases, 0 failures) and the model-agreement gate pass. One Q5_1 MUL_MAT failure from the first validation pass (NMSE 1.48× the limit) could not be replayed, because its input was not saved. Our internal requirement to capture and explain every failure is therefore not met, and no BUILD-OK marker was issued; see the Q5_1 caveat below. The copies of this README and the release notes inside the archive were written at review time and call the build a review candidate. This page supersedes only that status line; the archive itself is unchanged.

Source and build identity

The source is upstream f74e2fa6a31cc33ab4ee4ae516641cfe62df3924, with two local patches. The build includes the upstream qwen35moe architecture.

• source-local.patch skips LoRA adapter matrix operations when the adapter's effective scale is zero, in ordinary, indexed/MoE, and embedding adapter paths in src/llama-graph.cpp.

• sm70-flash-attention.patch corrects CUDA flash-attention dispatch in ggml/src/ggml-cuda/fattn.cu. Volta MMA kernels require at least 32 columns; the patch prevents selection of the unsupported 16-column variant and falls through to the existing 32-column kernel. The triggering case has F16 K/V, head size 128, GQA ratio 8, KV length 5003, and query batch 4. The exact regression passed again, and all 12,012 supported Flash Attention cases across the three fresh full runs passed. This is a candidate upstream fix.

• Reported version: 0.7.0-dev (build 1, commit f74e2fa-modified).

• Build number 1 comes from the shallow source checkout's commit count; it is not an upstream release number.

• source-local.patch SHA256: f1886f55c4bc527b6c801001f4d22d661fc7023d10a22c1eb3299e2680bfdee2.

• sm70-flash-attention.patch SHA256: fd61cc17fd0c902052900a576344ea6cfbb729280b838f6400466163f8be2cfa.

• The initial source transfer was verified by identical relative paths and SHA256 hashes for all 3,825 files, excluding .git and directories named build*. This comparison included the LoRA patch and preceded the additional Volta dispatch patch.

• Initial source manifest SHA256 (before the Volta dispatch patch): 61baf1ca095793b2f32dece65dd4473d3dd849cf06ac687a300f629e9c69ce8f.

• Compiler: GNU C/C++ 13.3.0 and NVIDIA CUDA compiler 12.9.86.

• Embedded CUDA machine code: sm_70 only.

Requirements

• Linux x86-64 with glibc 2.38 or newer and a compatible libstdc++ exposing GLIBCXX_3.4.32 or newer.
• CPU support for SSE4.2, AVX, AVX2, FMA, F16C, and BMI2. GGML_NATIVE=OFF avoids host-specific tuning but does not disable these instruction sets.
• A Volta GPU with sufficient VRAM for the selected model and context. The numerical validation target is a Tesla V100 PCIe 32 GB.
• For a toolkit-matched deployment, use a Volta-compatible NVIDIA driver at least 575.57.08, the Linux driver baseline listed for CUDA 12.9 Update 1. Validation uses 580.178.04. NVIDIA identifies R580 as the last branch supporting Volta; a newer branch is not automatically compatible. See NVIDIA's CUDA 13.0 guidance.
• External CUDA 12 runtime and cuBLAS libraries, preferably those supplied with CUDA 12.9: libcudart.so.12, libcublas.so.12, and its dependency libcublasLt.so.12. The NVIDIA driver supplies libcuda.so.1. These libraries must be discoverable by the system loader; CUDA 13 libraries do not replace the required CUDA 12 libraries.
• Standard Linux runtime libraries, including libgomp for OpenMP.

NCCL is disabled. OpenSSL was unavailable during compilation, so HTTPS support is disabled. Use local model files; the examples below bind the HTTP server to the local machine.

Contents and use

The package layout is bin/ for executables and lib/ for llama.cpp shared libraries, plus license notices. It includes llama-server, llama-cli, llama-bench, llama-quantize, llama-perplexity, and test-backend-ops. Keep bin/ and lib/ together when relocating the package.

To inspect the review archive and verify its checksum:

sha256sum -c SHA256SUMS
tar --zstd -xf llama.cpp-f74e2fa-cuda12.9-sm70-linux-x64.tar.zst
./bin/llama-server --version

Place a separately obtained model in ./models/. For example:

CUDA_VISIBLE_DEVICES=0 ./bin/llama-server \
  -m ./models/MiniCPM-V-4_7-35B-A3B-Q4_K_M.gguf \
  -ngl 99 -c 4096 -fa on -ctk f16 -ctv f16 \
  --host 127.0.0.1 --port 8080

Or run a text prompt:

CUDA_VISIBLE_DEVICES=0 ./bin/llama-cli \
  -m ./models/MiniCPM-V-4_7-35B-A3B-Q4_K_M.gguf \
  -ngl 99 -c 4096 -fa on -ctk f16 -ctv f16 \
  --temp 0 --seed 20261016 -n 64 \
  -p 'Explain why the sky appears blue.'

Select the intended GPU with CUDA_VISIBLE_DEVICES. The model is not included in the binary package. These examples use text input; no image, audio, video, or projector validation is claimed.

Validation

Fresh numerical tests pass. One historical Q5_1 failure remains unexplained (see the Q5_1 caveat).

The tested model is MiniCPM-V-4_7-35B-A3B-Q4_K_M.gguf, the Q4_K_M quantization of MiniCPM-V 4.7 35B-A3B. Its size is 21,166,255,008 bytes and its freshly verified SHA256 is 9ecf91e814d7e9adc64325b1b65f8628a9b1bb8da4f61a92869ca7489bfd888e, matching the publisher’s file checksum. Its GGUF architecture is qwen35moe. These checks cover text inference only; they do not establish compatibility with other models or multimodal inputs.

Model agreement check	Measured result	Acceptance limit	Result
Mean KLD	0.000597	≤ 0.001	Pass
99th-percentile KLD	0.016200	Report only	Informational
Same top token (“same top p”)	99.510%; 1,015/1,020 positions	≥ 99.0%	Pass
CPU reference perplexity	1.155534	Reference	—
V100 perplexity	1.154694	Compare with CPU	—
Absolute relative PPL difference	0.072694%	≤ 0.5%	Pass

The comparison uses the included deterministic original English text: four 512-token windows and 1,020 scored positions, batch 512, microbatch 128, Flash Attention enabled, F16 K/V caches, and seed 20261016. The CPU reference disables GPU and operation offload; the V100 offloads all 41/41 layers. The base file is generated by --kl-divergence-base and compared using --kl-divergence. Its scaled 16-bit log-probability format and the upstream KLD estimator are described in REVALIDATION.md. This small, low-perplexity corpus is a numerical check, not a model-quality benchmark. The initial four-decimal PPL display gave a 0.0692% difference; the values above use the more precise output from the paired KLD run.

Three complete fresh backend runs used the same operation-family filter and no parameter filter. Each executed 9,784 supported cases, with zero failures and 1,793 unsupported combinations. The interrupted partial attempts are excluded. The table totals all three complete runs; fusion variants are grouped with their corresponding operation.

CUDA operation family	Passed	Failed	Unsupported/skipped
MUL_MAT	11,754	0	1,260
MUL_MAT_ID	2,886	0	42
FLASH_ATTN_EXT	12,012	0	4,017
SSM_CONV	405	0	0
GATED_DELTA_NET	144	0	0
RMS_NORM	159	0	0
ROPE	1,350	0	60
SOFT_MAX	642	0	0
Total	29,352	0	5,379

The fresh MUL_MAT failure fraction is 0/11,754; the Q5_1 subset is 0/168. Unsupported combinations are not counted as passes or executed-case denominator entries. Each run uses fresh std::random_device tensor values. Identical CSV hashes across clean runs reflect the fact that CSV records parameters and outcomes, not random values. The separate capture harness links the unchanged release libraries and inserts capture calls only when a comparison fails. No new failures occurred, so there were no new inputs requiring replay.

Q5_1 caveat. The initial run recorded one Q5_1/F32 MUL_MAT failure at m=16, n=1, k=256: NMSE 0.000740110 against 0.000500000, or 1.48022× the limit. Including the initial full run and the three fresh runs gives 1/15,672 supported MUL_MAT cases, below the proposed pooled 0.1% limit. That does not prove its cause: the original random input was not saved and cannot be replayed exactly on CPU. CPU/CUDA quantization arithmetic differences are plausible contributors, but input conditioning has not been established for that failure. The earlier 64 captured diagnostics all passed. Separately reported Q5_1 repetitions passed 1,120 V100 cases and 280 sm_86 cases; those reports are not pooled into the fresh counts above. The tested model contains Q4_K, Q6_K, and F32 tensors, with no Q5_1 weights. The written requirement to retain and explain every failure therefore remains unmet; passing random retries cannot repair the missing original evidence.

Greedy caveat. The initial 64-token prompt comparisons first diverged at token 26 for reasoning and 12 for prose; the code prompt matched all 64 tokens. Positions are one-based and informational, not a gate. CPU and GPU floating-point summation order can change near-tied token choices, so greedy sequences can diverge even when aggregate agreement remains within tolerance.

The exact former Flash Attention crash case passed again using the unpacked release executable, and all 12,012 supported cases of the full family passed across the three fresh runs. The patch keeps Volta away from an unsupported 16-column MMA dispatch and uses the existing supported 32-column path.

All 18 unpacked ELF files match the tested build. All six tools resolve dependencies and pass their supported version or usage probes with LD_LIBRARY_PATH unset and no CUDA compiler in PATH. The machine has a toolkit installed, so this verifies relocation and runtime-only dependencies, not a separately provisioned runtime-only OS image. The package contains no CUDA runtime libraries, model weights, private execution logs, or lease records.

REVALIDATION.json and REVALIDATION.md contain the current results and protocol. VALIDATION.json preserves the initial results, including the superseded greedy-equality decision. The internal release gate remains false because the original Q5_1 input is unavailable. The build was published at the maintainer's decision with that caveat stated; no success marker is bundled or implied.

Speed benchmarks

Measured on one Tesla V100 PCIe 32 GB (driver 580.178.04), with the card reserved for the run. The model is MiniCPM-V 4.7 35B-A3B Q4_K_M with all layers offloaded and Flash Attention on.

llama-bench -ngl 99 -fa 1 -r 3, in tokens/s:

Test	Tokens/s
pp512	837.51 ± 3.49
pp2048	827.25 ± 5.39
pp8192	813.15 ± 4.07
tg128	110.86 ± 0.88

llama-server /completion with n_predict 512, ignore_eos, and the prompt cache off. Values are prompt processing / generation in tokens/s; the prompt size is the actual token count:

Prompt	No draft	With DSpark draft	Draft acceptance
714 tokens	625.7 / 102.4	572.4 / 121.0	49.8% (mean length 2.49)
5,614 tokens	898.9 / 100.2	840.3 / 106.1	40.9% (mean length 2.22)
16,814 tokens	888.3 / 96.9	837.8 / 86.2	29.3% (mean length 1.88)

Peak VRAM was 21.4 GB without a draft and 23.7 GB with it; the peak GPU temperature was 51 °C. The draft file is the publisher's dspark-MiniCPM-V-4_7-35B-A3B-Q4_K_M.gguf, passed with -md … -ngld 99; the server auto-detects it as draft-dspark block drafting. --spec-type draft-mtp is rejected for this model because it has no MTP layers. The draft helps at short contexts and costs throughput at long ones.

Second model: Tess-4.5-27B (dense) on the same card

Same card, binaries and method. The model is the publisher's Tess-4.5-27B-Q4_K_M.gguf from migtissera/Tess-4.5-27B-GGUF (Apache-2.0), a dense qwen35 27B. Its draft is the publisher's MTP head, mtp-Tess-4.5-27B-Q4_K_M.gguf, run with --spec-type draft-mtp -md … -ngld 99. It is a speed comparison only; the numerical validation above covers MiniCPM.

llama-bench, in tokens/s, with MiniCPM for comparison:

Test	Tess-4.5-27B (dense)	MiniCPM-V 4.7 35B-A3B (MoE)	Tess / MiniCPM
pp512	859.73 ± 30.46	837.51 ± 3.49	1.03
pp2048	854.14 ± 6.84	827.25 ± 5.39	1.03
pp8192	806.09 ± 2.32	813.15 ± 4.07	0.99
tg128	33.04 ± 0.33	110.86 ± 0.88	0.30

llama-server /completion for Tess, prompt processing / generation in tokens/s:

Prompt	No draft	With MTP draft	MTP acceptance
714 tokens	582.2 / 32.7	469.5 / 66.1	88.3% (mean length 3.65)
5,614 tokens	797.8 / 31.1	737.0 / 69.0	96.7% (mean length 3.90)
16,814 tokens	715.3 / 26.7	695.8 / 63.0	90.3% (mean length 3.70)

Peak VRAM was 18.5 GB without a draft and 22.0 GB with it; the peak temperature was 71 °C. Prompt processing is similar for the two models. Generation is about 3.3× slower for the dense model, which reads all 27B weights per token, while the MoE reads about 3B active ones. Tess's MTP head raises its generation 2.0-2.4×, which closes part of that gap.

Build configuration

The following reproduces the compiler settings, using generic paths. Apply both source-local.patch and sm70-flash-attention.patch to the exact upstream commit before configuring. $src must resolve to that source directory.

src="$(pwd)"
cuda=/usr/local/cuda-12.9
export PATH="$cuda/bin:$PATH"
export CUDACXX="$cuda/bin/nvcc"
export CUDA_HOME="$cuda"
export CUDA_VISIBLE_DEVICES=''

nice -n 15 cmake -S "$src" -B "$src/build-volta" -G Ninja \
  -DCMAKE_CUDA_COMPILER="$cuda/bin/nvcc" -DCUDAToolkit_ROOT="$cuda" \
  -DGGML_CUDA=ON -DGGML_CUDA_NCCL=OFF -DCMAKE_CUDA_ARCHITECTURES=70 \
  -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=OFF \
  -DBUILD_SHARED_LIBS=ON -DGGML_NATIVE=OFF -DGGML_CCACHE=OFF \
  -DLLAMA_BUILD_NUMBER=1 -DLLAMA_BUILD_COMMIT=f74e2fa-modified \
  '-DCMAKE_BUILD_RPATH=$ORIGIN;$ORIGIN/../lib' \
  '-DCMAKE_INSTALL_RPATH=$ORIGIN;$ORIGIN/../lib' \
  -DCMAKE_BUILD_WITH_INSTALL_RPATH=ON -DCMAKE_INSTALL_RPATH_USE_LINK_PATH=OFF \
  "-DCMAKE_C_FLAGS=-ffile-prefix-map=$src=llama.cpp" \
  "-DCMAKE_CXX_FLAGS=-ffile-prefix-map=$src=llama.cpp" \
  "-DCMAKE_CUDA_FLAGS=-Xcompiler=-ffile-prefix-map=$src=llama.cpp"

nice -n 15 cmake --build "$src/build-volta" --parallel 40 --target \
  llama-server llama-cli llama-bench llama-quantize llama-perplexity test-backend-ops

LLAMA_CURL=OFF is retained from the requested build configuration; this source revision deprecates that option. HTTPS availability is controlled by OpenSSL detection, as noted above. Prefix mapping removes the original source location from compiler-generated file strings. Build and install RPATH both use $ORIGIN and $ORIGIN/../lib.

License

llama.cpp is distributed under the MIT License, copyright (c) 2023-2026 The ggml authors. Include and retain the complete upstream LICENSE with copies or substantial portions of the software. The software is provided without warranty under that license.

Bundled third-party components retain their respective licenses and notices; preserve the accompanying THIRD-PARTY-NOTICES. NVIDIA runtime libraries and separately downloaded models are governed by their own licenses and are not relicensed by this package.

GitHub release: https://github.com/Atlas3DSS/llama.cpp/releases/tag/sm70-cuda12.9-f74e2fa