philipp-zettl/vrom-registry huggingface dataset:
- 

    
    Integers: Training Neural Networks Without Floating Point | Phil's Blog
    
- 
    
- 
    
- 
    
- 

    
- 
    
- 
    
- 
    
- 

    Skip to main content
    Back

    
        
            
                # Integers: Training Neural Networks Without Floating Point
                
                
                    
                        By Philipp
                    
                    Mar 11, 2026
                
                
            
            
                

Part 1 of a series on native integer machine learning

When you train a neural network today, you're almost certainly doing it in float32,
or if you're lucky enough to have the hardware, bfloat16. The floating point unit
is so deeply assumed in modern ML that most frameworks don't even expose a seam where you could swap it out.
But floating point is not free, and it's not universal.
This series is about what happens when you take that assumption away entirely.
## The Problem with Float

The case for integer arithmetic in neural networks is usually framed around inference:
quantize your trained float model down to integers, ship it to a phone, profit.
This is well-understood, well-supported and wildely implemented. HuggingFace provides a great overview of these quantization methods
in their transformer documentation.

What's less explored is whether you can skip the float phase entirely, i.e. training, backpropagation, and weight updates all in integer arithmetic,
on hardware that might not have a floating point unit (FPU) at all.
The motivation isn't exotic. A large and growing class of compute targets, i.e. microcontrollers, FPGAs, custom ASICs, and edge accelerators,
either lack FPUs entirely or pay a significant power and area penalty to use them.
The standard answer is "train in float, quantize to int for deployment,"
but this creates a fundamental mismatch: you optimize a float model, then approximate it as integers and hope the gap is small.
You also need float hardware to train in the first place, which rules out the scenario where the
training itself needs to happen on the edge, e.g. on-device fine tuning.

Beyond hardware constraints, there's a more fundamental question:
is floating point load-bearing for learning, or is it just what we've always used because the math looks nicer?
The smooth gradients, the dynamic range, the precise intermediate values — how much of that actually matters for convergence?

That's the question I started trying to answer about a month ago.
## What "Native Integer" Means

"Native Integer" in this context means something specific. It doesn't mean:

- Training in float, then quantization to integer afterward

- Using floats for gradients and integers for activations

- Simulating integer arithmetic in float space

It means the core tensor operations, forward pass, backward pass and weight updates are comptued directly in integer
types, with no floating point involved in the hot path.

The main technical challenge this creates is scale management.
Floating point values handle scale implicitly through their exponent field. Integers don't do that.
When you multiply two i32 values together, the product has roughly double the bit-width of meaningful signal, and
you need to decide how to downcast it back to a representable range without losing too much information.
If this is done natively, the activations either explode to saturation or collapse to zero within a few layers.
A numerical example motivates this better. Say you have a two-layer network where weights and inputs are stored
as i32 values in range [−127,127] (roughly one signed byte of useful signal).

Layer 1 has 128 inputs. A single dot product accumulates 128 multiplications, each up to 127⋅127=16129,
giving a maximum accumulator value around 128⋅(127⋅127)=2064512≈2.1⋅106.

This still fits in a i32 (max. ≈2.1⋅109), so layer 1 is fine.

Now pass that accumulator without scaling into layer 2 as the input.
Layer 2 also has 128 input, and now each element of the input vector is up to ≈2.1⋅106, rather than 127.
A single dot product accumulates to 128⋅(2.1⋅106)⋅127≈32⋅109. That overflows i32.
Every output neuron in layer 2 saturates to the maximum value, and all gradient signal through that layer is dead.
The opposite failure is just as easy to each. If you right-shift the layer  1 output aggressively to bring it back into [−127,127] range,
you're dividing by a power of two. Shift by too many bits and the rounded-down values are mostly zero. You've discarded the signal before
layer 2 ever sees it.

With three or four layers and naively chosen shifts, the activations reliably collapse to an all-zero tensor within the first few forward passes.

The solution that we will use for now is shift-based fixed-point arithmetic:
Every tensor carries an associated `shift` value that represents its effective scale as a power of two.
Multiplying two tensors at shifts `s_a` and `s_b` produces an accumulator at shift `s_a + s_b`, and downcasting
applies stochastic rounding to avoid the systematic bias that straight truncation introduces.
## The Project: `integers`

For compatibility reasons, and because I really want to get better in Rust, I decided to build the codebase in Rust.
The library is completely built from scratch, with no ML framework dependencies.

The code base abstractions are:
### `Scalar` and `Numeric` traits

Rathert than hardcoding a type, every module is generic over a `Scalar`, currently `i32` and `f32`.
This means the same `Linear` layer, the same `ReLU`, the same optimizer can run in either mode.
These traits implement mathematical operators for the underlying data types, e.g. `matmul`.

The `f32` path serves as a correctness baseline. If integer training diverges wildly from float training on the same architecture and seed, that's an indicator
something is wrong in the shift logic.

`Scalar`s implement types for inference, `Numeric` for accumulators.
### `Tensor&lt;T&gt;`

A simple row-major flat array with a shape vector. No striding, no accelerator backend.
The shift value is not stored in the tensor itself, it's tracked as a `u32` value that threads
through the computation graph alongside the tensor.

This is deliberate: keeping shifts out of the tensor type means fewer abstraction layers
and easier reasoning about where scale changes happen.
### `Params&lt;S&gt;`

The weight store, which maintains both a `master` copy (full `S::Acc` precision for accumulating gradient updates)
and a `storage` copy (quantized `S` for use in forward/backward).
The `quant_shift` on a parameter set controls how aggressively the master weights are compressed when synced to storage.
`Params` are used in objects that implement the `Module` trait.
### `Module&lt;S&gt;` trait

The standard forward/backward interface, both methods carry shift values, `s_x` the input shift and `s_g` the output shift.
```
`fn forward(&amp;mut self, input: &amp;Tensor&lt;S&gt;, s_x: u32, rng: &amp;mut XorShift64) -&gt; (Tensor&lt;S&gt;, u32);
fn backward(&amp;mut self, grad: &amp;Tensor&lt;S::Acc&gt;, s_g: u32) -&gt; (Tensor&lt;S::Acc&gt;, u32);`
```
Apart from the incoming shift values each methods returns the shift value of the resulting output tensor.
### Sequential

A container that chains modules and threads shifts through the full forward and backward passes automatically.
### Optimizers

The optimizers (SGD with optional momentum, Adam) are also integer-native,
with learning rates expressed as bit shifts: `lr_shift = 4` means a learning rate of `1/16`.
This sounds limiting, but it aligns with how you would implement a fixed-point optimizer on hardware anyway.

Note: The `f32` implementation of both optimizers use "raw" arithmetic, only the `i32` version uses our internal
`Scalar` and `Numeric` operations.
### Random number generator

Random numbers are generated using a shift-register generator (Xorshift), specifically the 64 bit version.
```
`#[derive(Debug, PartialEq)]
pub struct XorShift64 {
    pub state: u64,
}

impl XorShift64 {
    pub fn new(seed: u64) -&gt; Self {
        // edge case handleing for state = 0
        let state = if seed == 0 { 0xC0FFEE } else { seed };
        Self { state }
    }

    pub fn next(&amp;mut self) -&gt; u64 {
        let mut x = self.state;
        x ^= x &lt;&lt; 13;
        x ^= x &gt;&gt; 7;
        x ^= x &lt;&lt; 17;
        self.state = x;
        x
    }

    /// Random value generator for value in range [0, range)
    #[inline(always)]
    pub fn gen_range(&amp;mut self, range: u32) -&gt; u32 {
        (self.next() as u32) % range
    }
}`
```
### Stochastic rounding

The downcast path uses the `XorShift64` random number generator to round fractional remainders
probabilistically rather than always truncating. Over many samples, this gives unbiased estimates
where truncation would introduce a systematic downward drift.
```
`pub fn stochastic_downcast(val: i32, shift: u32, rng: &amp;mut XorShift64) -&gt; i32 {
    if shift == 0 { return val; }

    let mask = (1 &lt;&lt; shift) - 1;
    let frac = val &amp; mask;

    let thresh = rng.gen_range(1 &lt;&lt; shift) as i32;
    let round_bit = if frac.abs() &gt; thresh { 1 } else { 0 };

    let shifted = (val &gt;&gt; shift) + round_bit;

    shifted
}`
```
Again a quick numerical example.
Let's say we want to downscale a 5 dimensional vector with elements i=0,...,4 and xi=5 with a shift value of 1.
In the naive approach we do xi=5&gt;&gt;1=2 on every element, losing the information about the lost 0.5 for every element.
With stochastic downcasting some elements are scaled to 2 some to 3, retaining the lost information from the naive shift.
## Current State: A Baseline

At this point the project is a working baseline, not a polished library.
The immediate goals were to prove out the architecture and establish that integer training can learn something on real tasks.

Both experiments use the same architecture and batch size accross float and integer runs.
The f32 path serves as the baseline; the i32 path uses the same `Sequential` container and module implementations,
with the `Scalar` type swapped out.
### Iris

The Iris dataset has 150 samples, 4 continuous features, and 3 classes. Features are z-score
quantized before loading.

Architecture: `4 -&gt; 8 -&gt; 8 -&gt; 3` with ReLU activations and MSE loss.

f32
i32

Test accuracy
90%
90%

Epochs
500
5000

`lr_shift`
7
3

`momentum_shift`
1
0

Batch size
32
32

Grad. Clip value
-
512

Both runs reach the same final accuracy. The integer model gets there, but needs 100x more epochs to do it.

The hyperparameters also diverge significantly:

f32 runs comfortably with samall learning rate (`lr_shift: 7` ≈1128) and light momentum,
while i32 needs a much larger effective learning rate (`lr_shift: 3` ≈18) and stronger momentum to make progress.
This points to a noisier, coarser update signal in the integer path, each weight step is quantized to a power-of-two
granularity, so small gradient signals that would nudge a float weight by a tiny amount either round to zero or overshoot.
That i32 converges at all on Iris is encouraging.
That it needs this much more iteration to do so is the first concrete measurement of what integer training costs.
## MNIST

The MNIST dataset has 60000 training samples and 10000 test samples, with 784 pixel features per image.
Features are z-score quantized.

Architecture: `784 -&gt; 128 -&gt; 128 -&gt; 10` with ReLU activations, MSE loss, and early stopping at 95% test accuracy.

f32
i32

Test accuracy
95.89%
~20%

Epochs
31 (early stopping)
-

Time per epoch
~7.5s
increasing

`lr_shift`
6
5

`momentum_shift`
None
None

Batch size
32
32

The f32 model converges cleanly, hitting the early stopping threshold at epoch 31.

The i32 model currently doesn't. Loss does decrease, the network is learning something, but test accuracy plateaus at 20%, barely above random for a 10-class problem.

There's also an unexpected symptom:

Per epoch compute time grows as training progresses, which shouldn't happen with a fixed architecture and batch size.
That points to a memory or bookkeeping issue in the integer path that needs investigation before the results can be trusted.

The honest picture after this round of experiments:
Integer training works on small problems, and the scaling behavior on larger ones is an open question, for me.
Getting MNIST i32 to match the f32 baseline is the central problem we will work on next.
## What's Next

This post is only a starting point. The subsequent posts in this series will cover:

- i32 MNIST What needs to be done to get to 95% accuracy for i32?

- Shift management in depth. The relationship between `input_shift`, `output_shift`,
`quant_shift` and `grad_shift` deserves a full treatment. Getting this wrong is the most
common failure mode, and the current codebase has some heuristics (auto-detecting shifts
from weight magnitude after Xavier init) that need to be documented and justified properly.

- Other layers. The repo contains already an implementation of a `RNNCell` and will soon receive
`Conv2d`. These will receive dedicated posts, too.

- The hardware question. What would it actually look like to run this inference (and eventually training)
on a microcontroller of FPGA? What changes when you drop from `i32` to `i8`?
The code is open and the approach is straightforward. The point of writing
this up isn't to claim that integer training is solved, it clearly isn't,
but to document what a ground-up integer ML stack might look like, what problems it runs into, and whether
the core assumption holds: that you can train a useful neural network without ever touching a floating point number.

So far, for small problems, the answer looks like yes. Part 2 will find out how far that scales.

The source code is available at github.

            

        

        
        
            &gt;&gt; SEE_ALSO:
            
            
                
                
                    
                    Mar 12, 2026 | 
                    Integers: The Theory
                
                
                
                    
                    Mar 27, 2026 | 
                    Integers: An integer native ML engine
- 

    
    Integers: The Theory | Phil's Blog
    
- 
    
- 
    
- 
    
- 

    
- 
    
- 
    
- 
    
- 

    Skip to main content
    Back

    
        
            
                # Integers: The Theory
                
                
                    
                        By Philipp
                    
                    Mar 12, 2026
                
                
            
            
                

Part 2 of a series on native integer machine learning/deep learning
#### Debug notes

After the previous post I jumped right into
debugging.

The increasing runtime in combination with the drastic difference between `lr_shift` and `momentum_shift` in the Iris example
gave me reason to believe that the code has a bug in regards to scaling (shifting values).

I am making this assumption out of two reasons.

First, both versions use different arithmetic operations. The `f32` uses raw arithmetic operators, e.g. `+` and `*`
whereas the `i32` uses saturating/wrapping arithmetic, i.e. values are either clipped or wrapped when overflowing.

Second, the `i32` version implements gradient clipping.

To get a clearer picture of the current state I added debugging statements to Iris test, checking runtime, overflow stats and outgoing shift of predictions.
The results were rather sobering; no wrapping, compute time per epoch is between 2700 - 3000µs (micro seconds). This could be influenced by a gamma ray from outer space.
So not really a great measurement for tracking down bugs.
Looking at the shifts of the network revealed something different.

For every training sample we see inside the logs
```
`Epoch #     Loss   Shift In    Forward Shift   Backward Shift
Epoch 1     XXXXXX      5           16              16
Epoch 2     XXXXXX      5           16              16
Epoch 3     XXXXXX      5           16              16
...`
```
This doesn't look correct. Especially the backward shift looks completely wrong.

The layers are configured the following way

Layer
Quant Shift
Incoming Shift
Outgoing Shift

L1
0
3
4

L2
0
4
4

L3
0
4
2

Our input is created with the `QuantizationMethod::StandardScore`, so it carries a shift of `5`.

The calculation for the final shift of the forward pass so is
so=si+∑k=1Nsik+sqk
with si the original input shift, sik the input shift of the k-th layer and sqk the quantization
shift of layer k.

In our example
so=5+(3+0)+(4+0)+(4+0)=5+3+4+4=16

This shows that our forward pass is already computing the correct shift.
The backward pass should reduce this shift again, but it doesn't. Hence an issue is inside the backward pass.

A fresh set of eyes was one of the main drivers behind the initial solution. After adjustment we ran the experiment again.

Architecture: `4 -&gt; 8 -&gt; 8 -&gt; 3` with ReLU activations and MSE loss.
We now reach higher accuracy on i32 compared to its f32 counter implementation.
The i32 branch converges within 239 epochs reaching an accuracy of 95%, whereas f32 never reaches this level.

f32
i32

Test accuracy
90%
95%

Epochs
500
239 (early stopped)

`lr_shift`
7
7

`momentum_shift`
1
1

Batch size
32
32

Grad. Clip value
-
213

Reason for this "success" were two things. Fixing the internal shift bookkeeping, `Linear&lt;S&gt;`'s backward pass was using `Linear::output_shift` to transform the gradient w.r.t inputs. This resulted in wrong shifts for subsequent layers. And `Linear&lt;S&gt;`'s backward pass did not return a reduced `s_g` (gradient shift), but rather just forwarded the incoming value.
Apart from the bookkeeping, drastically increasing the gradient clip value to allow more information bytes flowing through the network helped to get more stable results.

The diff for the changes inside `Linear&lt;S&gt;::backward`
```
`-        let local_delta_w = s_g + s_x;
+        let local_delta_w = s_g - s_x;
-        let local_delta_x = self.weights.quant_shift + self.output_shift;
+        let local_delta_x = self.weights.quant_shift + self.input_shift;
...
-        let s_g_prev = s_g;//.saturating_sub(local_delta_x);
+        let s_g_prev = s_g.saturating_sub(local_delta_x);
`
```
Especially the update to `s_g` was something I had implemented earlier, but forgot an didn't discover up until today.
#### Impact on MNIST

The change was clearly fixing a bug, otherwise our performance on Iris would not look like this.
Running quickly some epochs shows that at least the increasing compute time issue that we had in the previous post is
resolved and we have a more stable value there
```
` Epoch         Loss     Accuracy    Time(s)   Clamps FW Shift BW Shift
────────────────────────────────────────────────────────────────────────────────
     0    1612.0000         9.4%       9.12s       -1    31    5
     1    1612.0000         9.4%       9.09s       -1    31    5
     2    1612.0000         9.4%       9.08s       -1    31    5
     3    1612.0000         9.4%       9.09s       -1    31    5
     4    1612.0000         9.4%       9.08s       -1    31    5
     5    1612.0000         9.4%       9.08s       -1    31    5
     6    1612.0000         9.4%       9.07s       -1    31    5
     7    1612.0000         9.4%       9.10s       -1    31    5
     8    1612.0000         9.4%       9.08s       -1    31    5
     9    1612.0000         9.4%       9.09s       -1    31    5`
```
Tip
        
        

Obviously, the network in these logs did not learn anything, instead pay attention to the now more or less stable runtime.
## The core math

Right now, we're adjusting things and seemingly fix issues that we don't fully understand.
Instead of running around like a headless chicken, we will now look closer into the theory
behind the implementation of native integer training.
After we defined all the required theory, we will implement it and hopefully resolve all issues we faced so far, for good.
### Motivation

The fundamental problem with integer arithmetic is that multiplication accumulates scale.
When you multiply two integers, the result lives in a different "unit" than either operand,
and without an explicit mechanism to track and correct for that, values either explode or collapse
across layers in a artificial neural network.
The structure we want is one where the scale is part of the value itself, so that every operation
is well-defined in terms  of what the numbers actually mean, not just what bits they hold.
### The carrier set

Let Z be the integers. Define the carrier set as the Cartesian product

D=Z×Z

An element x=(v,s)∈D consists of a mantissa v∈Z and a scale exponent s∈Z. The rational
number it encodes is given by the interpretation map

[[x]]=v⋅2−s

So a larger s means the mantissa is "finer-grained", the unit each integer step represents is smaller.
A smaller s (or negative s) means the value lives at a corser scale.

        

          
            

          
          Note
        
        

The formal definition admits s∈Z, allowing negative scale exponents, which would represent values scaled up by a power of two rather than down. In the implementation, shift values are typed as `u32`, restricting s≥0. This means the system can only represent dyadic rationals of magnitude ≤|v|, never larger. The formal generality is retained here because the mathematical properties of the operations hold regardless, but any concrete instantiation should be understood as working in Z×N0.
### Non-uniqueness

It is critical to note that D is a set of representations, not a set of distinct values.
Two elements x1=(v1,s1) and x2=(v2,s2) are said to be equivalent, written x1∼x2, if and only if they encode
the same rational number

x1∼x2⇔v1⋅2−s1=v2⋅2−s2

For instance, (3,2),(6,3) and (12,4) all represent the rational 34. This non-uniqueness is not a flaw, it is precisely what
gives the system its flexibility. The scale can be chosen to match the magnitude of the data at hand.
### Scale alignment

Before we can define addition and substraction, we need a way to bring two elements into a common representation.

Given x1=(v1,s1) and x2=(v2,s2), define the target scale s∗=max(s1,s2), which is the coarser of the two scales.
The alignment of xi to scale s∗ is

align(xi,s∗)=(⌊vi2s∗−si⌉,s∗)
where ⌊...⌉ denotes a stochastic down-scaling^1 operation and vi2s∗−si the aligned value according to the
target scale.

The element with the finer scale has its mantissa right-shifted (divided by 2k) to match the coarser unit.
This is the exact analogue of lining up decimal points before adding:

You can only add 1.27+1.3 once they share the same number of decimal places.

Alignment is lossy in general, i.e. the bits shifted out are gone for good.
This precision loss during alignment is the fundamental source of rounding error in this system, which is the equivalent
to floating-point roundoff.
### Stochastic scaling

Stochastic scaling/Requantization handles two things. Rescaling (adjusting the shift to a new target) and clipping (enforcing the hardware bit-width boundaries).
#### Rescaling

Let x=(vx,sx)∈D be an accumulated tuple, and st∈Z be the target shift such that k=st−sx≥0.

We define the stochastic rounding function SR(v,k):D→D as:

SR(v,k)=⌊v⋅2−k⌋+I((vmod2k)&gt;U)

where U∼U{0,2k−1} and I(⋅) is the indicator function.

The requantization operator R:D×Z→D is then defined as

R(x,st)=(clip(SR(vx,st−sx),Qmin,Qmax),st)

where Qmin and Qmax represent the representable bounds of the target integer precision.
The function clip is defined as:

Let v∈Z be an integer value, and let Qmin,Qmax∈Z represent the lower and upper bounds of the target
scale, where Qmin&lt;Qmax.

clip(v,Qmin,Qmax)={Qminif v&lt;Qminvif Qmin≤v≤QmaxQmaxif v&gt;Qmax

We can derive the values of Qmin,Qmax∈Z based on the target bit-width b∈N+.

For targets for a signed two's complement integer (e.g. INT8 where b=8):

Qmin=−2b−1Qmax=2b−1−1

For unsigned integer targets (e.g. UINT8):

Qmin=0Qmax=2b−1