Ling-3.0-tiny, IQ8R, for radiance

ling-3.0-tiny-iq8r.rad (7.9 GiB) is inclusionAI/Ling-3.0-tiny converted for the radiance inference engine, version 2.1.0 or later. It is a quick conversion for trying the model: round to nearest with a scale search, no calibration data.

What is in it

  • IQ8R trunk: the KDA projections (fused q|k|v, forget, gate, output), the latent attention's output, the dense layer's feed-forward and the shared expert's gate/up as int8 codes with one u8 scale per (row, 256 columns) under one bf16 scale a row, each 128 columns rotated by a Walsh-Hadamard transform before rounding.
  • The 128 routed experts of every layer and the shared expert's down projection as int8 on the same rotated grid, one bf16 scale per (row, 128 columns).
  • In bf16: the KDA beta projection, the latent attention's low-rank projections and gate, the convolutions, the router, the embedding and the lm_head.

Converted with the recipe the engine ships:

rad-convert <Ling-3.0-tiny checkpoint> --recipe data/recipes/ling-3.0-tiny-iq8r.recipe \
    -o ling-3.0-tiny-iq8r.rad

Serving

One card, or two:

models:
  ling-3.0-tiny:
    model: /models/ling-3.0-tiny-iq8r.rad
    tp: 1

What was checked

On Radeon AI PRO R9700 cards: the container loads at tp: 1 and tp: 2, answers short factual and coding prompts correctly, finds a string hidden in a 4.8k-token prompt, and decodes one stream at 4.3 ms a step on one card (230 tokens/s) and 3.7 ms on two (270 tokens/s). The model has no draft head. Not measured: KL divergence against the bf16 model, and any benchmark.

License

MIT, as the source model.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for StillDeadcode/ling-3.0-tiny-iq8r

Quantized
(43)
this model