BEP: A Binary Error Propagation Algorithm for Binary Neural Networks Training
Authors: Luca Colombo
Fabrizio Pittorino
Daniele Zambon
Carlo Baldassi
Manuel Roveri
Cesare Alippi
Venue: ICLR 2026 Poster (2026)
Keywords: Binary Neural Networks
Fully binary training
Binary error backpropagation
Gradient-free optimization
Binary Recurrent Neural Networks
Resource-Constrained Machine Learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Binary Neural Networks (BNNs), which constrain both weights and activations to binary values, offer substantial reductions in computational complexity, memory footprint, and energy consumption. These advantages make them particularly well suited for deployment on resource-constrained devices. However, training BNNs via gradient-based optimization remains challenging due to the discrete nature of their variables. The dominant approach, quantization-aware training, circumvents this issue by employing surrogate gradients. Yet, this method requires maintaining latent full-precision parameters and performing the backward pass with floating-point arithmetic, thereby forfeiting the efficiency of binary operations during training. While alternative approaches based on local learning rules exist, they are unsuitable for global credit assignment and for back-propagating errors in multi-layer architectures. This paper introduces Binary Error Propagation (BEP), the first learning algorithm to establish a principled, discrete analog of the backpropagation chain rule. This mechanism enables error signals, represented as binary vectors, to be propagated backward through multiple layers of a neural network. BEP operates entirely on binary variables, with all forward and backward computations performed using only bitwise operations. Crucially, this makes BEP the first solution to enable end-to-end binary training for recurrent neural network architectures. We validate the effectiveness of BEP on both multi-layer perceptrons and recurrent neural networks, demonstrating gains of up to $+6.89$% and $+10.57$% in test accuracy, respectively. The proposed algorithm is released as an open-source repository.

Published as a conference paper at ICLR 2026
BEP: A BINARY ERROR PROPAGATION ALGORITHM FOR BINARY NEURAL NETWORKS TRAINING
Luca Colombo
Fabrizio Pittorino'
Daniele Zambon"
Carlo Baldassi²
Manuel Roveri'
Cesare Alippi
"Department of Electronics, Information and Bioengineering, Politecnico di Milano, Italy FFaculty of Informatics and IDSIA, Università della Svizzera italiana, Switzerland 3 DDeparmeent of Computing Sciences and BIDSA, Bocconi University, Italy
ABSTRACT
Binary Neural Networks (BNNs), which constrain both weights and activations binary values, offer substantial reductions in computational complexity, memory footprint, and energy consumption. These advantages make them particularly well suited for deployment on resource-constrained devices. However, training BNNs via gradient- optimization remains challenging due to the discrete nature of their variables. The dominant approach, quantization-awarc training, circumvents this issue by employing surrogate gradients. Yet, this method requires maintaining latent full-precision parameters and performing the backward pass with floating- point arithmetic, thereby forfeiting the efficiency of binary operations during training. While alternative approaches based on local learning rules exist, they are unsuitable for global credit assignment and for back- proccpppppggaing errors in multi- layer architectures. This paper introduces Binary Error Propagation (BEP), the first learning algorithm to establish a principled, discrete analog of the backpropagation chain rule. This mechanism enables error signals, represented as binary vectors, to be propagated backward through multiple layers of neural network. BEP operates entirely on binary variables, with all forward and backward computations performed using only bitwise operations. Crucially, this makes BEP the first solution to enable end-to-end binary training for recurrent neural network architectures. We validate the effectiveness of BEP both multi-layer perceptrons and recurrent neural networks, demonstrating gains of up to +6.89% and + +10.57% in test accuracy, respectively. The proposed algorithm is released an open-source repository.
INTRODUCTION
The design of Neural Networks (NNs) with weights and activations constrained to binary values, typically +1, is a promising direction for building models suited to essource-consrrained environments. In particular, Binary Neural Networks (BNNs) offer compelling solution for deploying Deep Learning (DL) models on edge devices and specialized hardware, where computational efficiency, power consumption, and memory footprint are critical design constraints (Courbariaux al. 2015
Rastegari et al. 2016 Hubara et al. 2016). Their primary advantage lies in replacing costly Floating- Point (FP) arithmetic with lightweight bitwise operations such XNOR and Popcount, resulting in
substantial reductions in computational complexity (Qin et al. 2020 Lucibello et al., 2022).
Despite these advantages, effectively training BNNs remains challenging due the non-differentiable nature of their activation functions. Consequently, conventional gradient-based learning algorithms cannot be directly applied, leading two main classes of solutions. The most prevalent is Quan- tization Aware Training (QAT), which formulates the problem within a continuous optimization framework. In this paradigm, full-precision latent weights are maintained, and non-differentiable activations are bypassed during the backward pass using the Straight- Throuhh Estimator (STE) (Ben) gio al. 2013). However, reliance on real-valued computations for gradient calculation and weight updates confines the efficiency of binary arithmetic to the forward pass only (Sayed et al. 2023)
Coorreponniig author. Email: cola..co polimi.it

---

Published as a conference paper at ICLR 2026
An alternative line of research has explored purely binary, gradient- free learning rules, often inspired by principles from statistical physics (Baldassi et al., 2015; Baldassi 2009). These methods operate directly binary weights and avoid continuous surrogates. A recent extension applied this approach to binary Multi-LLayer Perceptrons (MIPs) by generating local error signals at each layer using fixed random classifiers (Colombo al. 2025). However, fundamental limitation of this approach is that credit assignment remains local and error information does not propagate from the final output layer through the NN. This constraint makes such rules inapplicable to architectures where learning depends on end and-to-end error propagation across layers, such as Recurrent Neural Networks (RNNs).
From this perspective, this paper addresses the following research question: it possible to formulate a multi-layer, global credit assignment mechanism that back propagates errors through the NN while operating exclusively within the binary domain? To the best of our knowledge, we introduce the fully binary error Backpropagation (BP) algorithm capable of effectively training BNNs without relying on FP gradients. The algorithm, called Binary Error Propagation (BEP) hereafter, establishes a binary analog of the standard BP chain rule, where error signals represented binary vectors - are computed at the output and propagated backward through each layer of the NN. To ensure learning stability, BEP employs integer-valus mitigate catastrophic forgetting (Kirkpatrick et al.] 2017). Crucially, the entire forward and backward passes rely solely on efficient XNOR, Popcount, and increment/decremeent operations. In summary, this work makes the following contributions:
hidden weights that provide synaptic inertia and
We formalize fully binary BP algorithm for BNNs that propagates binary-valued error signals end-to-end, establishing a discrete analog of the gradient- based BP chain rule.
We demonstrate that BEP successfully train both MLP and RNN architectures, overcoming the limitations of prior local and gradient-based learning methods.
The remainder of this paper is organized as follows. Section 2 reviews related work. Section formalizes the proposed BEP algorithm. Section presents experimental results on binary MLP and RNN architectures. Finally, Section draws conclusions and outlines future research directions.
2 RELATED LITERATURE
The predominant paradigm for training BNNs is QAT (Courbariaux et al. 2015 Hubara et al. 2016 Rastegari et al./ 2016). In this approach, models maintain latent full-precision parameters that are binarized during the forward pass, while gradients are computed with respect to the latent parameters using a surrogate gradient, typically a STE (Bengio et al. 2013]. The STE approximates the derivative of the non-differentiable sign function as an identity within a bounded region, enabling the use of standard BP. Numerous subsequent works have built upon this foundation, introducing improvements such learnable representations, enhanced architectures, and strategies to narrow the accuracy gap with full-precision models (Lin et al.) 2017 Liu et al. 2020: Tu et al., 2022, Schiavone et al. (003). While QAT has achieved strong empinical results, it remains fundamentally continuous optimization method applied to a discrete problem. Training relies on FP arithmetic, which prevents the full realization of BNN efficiency during learning and introduces discrepancy between training and inference dynamics (Yin et al., 2019). Recent work (Liu et al.) 20018) Bulat & Tzimiropoulos 2019 Vargas et al., 2024) has further refined QAT by incorporating residual connections and improved surrogate-gratt gradienn mechanisms to enhance gradient flow. In contrast, our approach diverges from this paradigm by eliminating the need for any real-valued parameters or surrogate gradients.
A distinct line of research frames BNN training as a purely binary optimization problem. Early work in statistical physics explored combinatorial optimization techniques for training single-layer perceptrons (Gardner 1988 Engel| 2001), while later studies investigated fully binary training via global heuristics such as simulated annealing (Kirkpatrick et al.) [1883] Hinton| 1990) and evolutionary strategies (Salimans et al. 2017 Such et al. [017;; Loshchilov & Hutter 2016). Although these methods avoid continuous relaxations, they explore the weight space via stochastic perturbations (e.g., random weight flips) and lack structured, layer-wist credit assignment comparable to BP, which limits their scalability and efficiency in deep settings. Our work addresses this limitation by developing multi- layer, tradient-free training method with deterministic error- pror-prropggttion rule.
In parallel, research at the intersection of statistical physics and computational neuroscience has developed efficient local learning rules for binary neurons. Algorithms such the Clipped Perceptron
2

---

Published as a conference paper at ICLR 2026
P
W
W1-1
(a) BEP applied binary MLP
(b) BEP applied to binary RNN
Figure I:
Information flow for a sample ya in an MLP and an RNN trained with BEP. Each model uses binary and a fixed classifier. The forward and backward passes are shown in gray and red.
with Reinforcement (CP+R) (Baldassi 2009) and related message-passing passeespssnnnggg approaches (Baldassi et al. 2007] 2015) introduce integer-valuee hidden variables to represent synaptic confidence, demonstrating that single binary units can learn effectively. Multi-layer extensions of these rules generate layer-wise local error signals using fixed random classifiers, enabling training of several binary layers but still lacking end-to-end propagation of task loss (Colombo et al.) 2025). This structural constraint notably precludes their application to relevant class of recurrent sequential architectures, such as RNNs. BEP overcomes these shortcomings by introducing a global credit assignment mechanism that propagates binary error signals end-to-end, bridging the gap between fully binary optimization and multi-layer training of deep architectures.
THE BINARY ERROR PROPAGATION ALGORITHM
In this section, we formalize the BEP algorithm for training multi-layer BNNs using exclusively binary operations. We consider a supervised learning setting defined by classification task and corresponding dataset X """ p==
of size N, where x" R are input patterns of dimension Ko. and e [l...... C are their corresponding labels, with C denoting the number of classes. Specifically, Section 3.1 introduces the BNN architecture and forward-pass dynamics. The binary backward pass for error propagation is described Section 3.2 The weight-update rule is defined in
Section 3.3 3 while the analogy with standard full-precision BP is presented in Section [.4] Finally, demonstrates the application of the proposed BEP algorithm RNN architectures.
Section 3.5
3.1
BNN ARCHITECTURE AND FORWARD PROPAGATION
The NN under consideration consists of two main components: trainable binary backbone that extracts features, and a fixed task-specific output layer that maps these features to the final prediction, shown in Figure Given mini-batch x [_..,x"... ....] of bs input samples x" where E (1..... bs), we first obtain their binary representations a = bin(x") ) {+1]%o The binarization function be realized via median thresholding for images thermometer encoding for
tabular data (Bacellar et al. 2024a). The resulting binary batch ag is then fed to the binary backbone.
The backbone comprises stack of L fully-connected binary layers. For each laver E 11.... L). its state is defined by two matrices, following prior work on binary synapses (Baldassi 2009). The first is the matrix of hidden discrete weights H, E KK These integer-valued weights encode the synaptic inertia of each connection, providing mechanism for stable learning and mitigating catastrophic forgetting (Kirkpatrick et al.) (017). Although formulated as unbounded integers, in practice H, is constrained to the finite range of B-bit signed integer, 1- 2 - 1]. The second is the matrix of visible binary weights W; = sign(H;) (=1]ki*K... representing the effective weights used during the forward pass. Here, Ki denotes the number of neurons layer I, and the sign function is applied component- component-wise, returning 1 for negative inputs and +1 otherwise. For each layer I, the pre-activations and activations are computed as zi Wia and aj = sign(z;). respectively. The output the BNN backbone (i.c., the activations ar.) is then fed the task-specific output layer to produce the final predictions ý.
Trainable BNN Backbone.
3

---

Published as a conference paper at ICLR 2026
Task-Specific Output Layer. To make predictions, the binary features at of each sample j must be mapped to the space of target variables y". For regression tasks, this layer acts as a binary decoder {+1)*² RD, where D is the output dimensionality. For C'-class classification problem, the goal is to produce a logit vector 5 E TRC whose largest component corresponds to the predicted class. While different mappings and accuracy. In this work, we use linear layer represented by fixed, randomly initialized matrix P [=1]0xK,, This corresponds to associating a prototype vector pe (the e-th row of P) with each class (1....,c) C} (see Appendix C for the generation method). Providing the backbone with a stable and geometrically optimal set of target activations the class prototypes - simplifies credit assignment during training and consistently yielded the best performance in our experiments. Although is possible to train this classifier, we found empirically that keeping it fixed more practical and effective. The final logits are computed as ý = P ag, and the predicted class for each sample j is given by 8" = arg max. 8
be used, the choice of output layer influences both learning
3.2 BINARY ERROR BACKPROPAGATION
The BEP learning rule is error-driven. backward update initiated for each sample j when the logit y associated with the ground-truth class ct is not sufficiently larger than the others. Specifically, an update is triggered if
OF
max < <rKL, Cffor
(1)
where r E (0, 1] is a user-specified margin hyperparameter and Ky, is the size of the last hidden layer (also representing the maximum possible logit value). Higher values of r enforce a larger gap between the correct-class logit and all others, thereby encouraging more robust classification.
For clarity, describe the backward pass for a single training sample although in practice it is applied to all elements of a mini-batch. When a sample i meets the update criterion in Eq [ BEP initiates the backward phase adjust the hidden integer weights H2 so that the final output better aligns with the correct target. This is accomplished by defining, for each layer I, a binary desired activation vector a," that is propagated from the output layer L back to the first layer.
Desired Activation at the Last Layer. The backward pass begins at the final layer L of the BNN backbone. The first step is to determine the desired activation vector a,," which the ideal binary vector that maximizes the logit for the correct class et, Since the logit is scalar product between activations and the class prototypes from the fixed classifier P, the desired activations can be found analytically. They correspond to the prototype vector p for the correct class:
L := arg max (a. p = =p"
(2)
ac aE(il)
Backpropagation of Desired Activations. For each layer < L. the desired activation vector a obtained by propapprpggg the error signal a from the subsequent layer / 1+ 1. The goal is to determine activations at" that maximize alignment with desired activations a
arg max (att)) sign( 141 a)).
aE(=1]%
Since this is a combinatorial search over 2 configurations, BEP solves relaxed problem by removing the non-linear sign function and instead maximizing the alignment with the pre-activations:
arg max (aft1 W1+1a).
ae (1
This relaxed objective increases the magnitude of the pre-activations Zi+1 while aligning their signs with -- A global optimum is unnecessary, as the goal is to steer weight updates in the appropriate direction. As shown in Lemma I the solution this relaxed problem can be derived analytically.
Backward Gating. In addition, we introduce gating mechanism to regulate the back propparaag signal. This gate is applied at layer / + 1 to emphasize learning on neurons most amenable to change, ie., those with pre-activations 2 Wt+1a near the decision threshold of O. Neurons with
4

---

Published as a conference paper at ICLR 2026
large-magg mageemmag pre-activations (nearly saturated responses) excluded from the backward pass of sample μs. This is realized through a binary gating vector Bf+1
(gf+ = =10 [0,
{1 1.
if 1 S LuK; otherwise
(5)
where 1, is the i-th pre-activation layer 1 + 1 and E [0, [] is a tunable threshold. More formally, for generic gating vector g, we define the gated scalar product (a, aa) := Eigaa,f This modifies the relaxed optimization problem of Eq. to:
arg max (a Wi+1a) 8:+1 aE{#1]%)
This gating function effectively filters the binary error signal, allowing to pass only through neurons close to their activation boundary. This ensures that weight updates are driven by the parts of the BNN most susceptible to flipping their activations. The solution to Eq. 6 is given by the following lemma (proved in Appendix A).
Lemma 1 (Desired activations). Consider a binary vector E [=1]K,, a binary matrix W e [ti]KKxK,, and a gating vector g [0,1]Kk, Problem arg (a)x) (b, unique solution: a" sign (WT (g ② b)).
(b, Wa)g has the
Combining the base case from Eq. 2 with L yields the recursive expression for computing the desired activation vector at any layer l:
p
=f i L'
+ al
sign (W 1+1 (Bt+) o
where O denotes the element wise product. This formulation forms the cornerstone of BEP, estab- lishing a fully binary chain rule for propagating binary error signals throughout the BNN.
3.3 WEIGHT UPDATE MECHANISM
Once the target activation vector a has been determined for layer 1, the corresponding hidden weights H, are updated. This update strengthens the association between the incoming activation vector = and the desired output pattern a," "," For every sample that triggers an update, we first compute a candidate weight-update matrix from the desired binary weights, following a procedure similar to that in prior work (Baldassi et al. using the same strategy employed to obtain a we maximize (a,", Wia with respect to W,
2007| Baldassi] 2009 Colombo et al., (2025). In partieular,
The solution gives the desired direction of change for the integer weight [
AH sign (a," (a) = "af"""
E (+1) [+1]KixKi1
8
Eq. 8 is the natural matrix generalization of the classical supervised Hebbian perceptron rule (Rosen- blatt 1958) and of the Clipped Perceptron (CP) and CP+Reinforcement (CP+R) variants (Baldassi et al. 2007: Baldassi 2009). When K; 1, the update AH, reduces exactly to the CP update, which adds (or subtracts) the input vector a to the synaptic stability variables whenever the desired output a," is +1 (or 1)). In the multi-layer setting, Eq. g applies this outer-product mechanism independently to each hidden neuron, with the desired activations a," "M serving as binary targets.
This potential update is then filtered by a binary mask M E {0, [0,1]KixMe KiXKi that selects which weights to modify. As shown in Appendix B the resulting update is locally optimal with respect to the desired activations a," Following the sparse update strategy of (Colombo et al. 2025). the mask is constructed by partitioning the neurons of layer 1 into subgroups and, within each group, updating only the incoming weights of the wrong perceptron deemed easiest to correct based on its pre-activation. This mechanism promotes more uniform distribution of updates, leading more stable training and improved generalization. If M denotes the set of mini-batch indices that trigger an update, we first compute each per- sample update AH using the same pre-update hidden weights H₁, and then apply the aggregated update as:
H t H +
Σ (M O AHH).
PEM
It follows from treating each of W, independent optimization and swapping a and W in Lemma
5

---

Published as a conference paper at ICLR 2026
The binary mask Mf implements a sparse winner-takes-updar rule within each neuron group: only the least stable unit, i.e., the neuron with the smallest signed stability a," Ht,j among the misclassified ones, is updated. This mechanism bounds the number of synapses modified per pattern and prevents over-reinforcing neurons that already classify the sample with high confidence, effectivvly acting as discrete, data-dependent form of learning-rate control. Additional analysis of mask density and its effect on convergence and generalization is provided in Appendix D.5 Finally, a reinforcement step from the CP+R rule stochastically strengthens existing memory trajectories of each integer weight h E H, via the update h < h + 2h) This occurs with probability pr V 2/(T Ki), where the reinforcement probability p, is rescaled each epoch by E This mechanism reinforces weights more frequently when the model is uncertain and balances its effect across layers of different sizes.
While (Colombo et al., 2025) employs fixed group size 7, corresponding to a constant number of updates per epoch, BEP uses a scheduled group size. Let I = {d E N : Kr} denote the ordered list of positive divisors of K₁. The algorithm begins with user-defined initial group size "o. E Ií and increases it adaptively based on accuracy. Whenever the generalization error stagnates for user-defined number of epochs, the group size is increased to the next larger divisor T+-1.1 = min {d E T d > Vr.) ). If N/, is already the largest divisor of K2 (the last element of Tt), it remains fixed. Because only one perceptron per group is updated, enlarging the groups gradually reduces the number of weight changes at each step, resulting in increasingly sparse updates. Empirically, this mechanism leads to more stable optimization process in the later stages of training.
3.4 ANALOGY TO GRADIENT-BASEL BACKPROPAGATION
The BEP procedure can be viewed as a binary reformulation of the classical BP computational graph (Rumelhart et al. 1986). It preserves the global flow of information while substituting real- valued operations with binary, bit-wise counterparts. In standard BP, the weight update for layer / follows gradient-descent form:
W W 1) 61 a
where 17 is the learning rate and o, denotes the back propaagte error signal for layer L. This rellevalued signal is computed recursively by the chain rule:
Vaa C (WT ( 842) o Oz)),
=f ifl L
6,
(10)
Each component of BEP mirrors a counterpart in mudient-bbssed BP. The binary desired activation at" serves the binary analog of the error signal. Back- projecttoj it through W is structurally equivalent to the error propagation step in the continuous case. The binary gating function plays the role of the activation derivative a' o((.), blocking error transmission through saturated neurons a d'(-) approaches zero for large inputs. Finally, the sparse mask M, fulfills the role of the learning rate: whereas 77 controls the magnitude of an update, the mask governs its sparsity, These correspondences show that BEP constitutes a coherent end-to-end binary analog of traditional gradient-based BP.
3.5 APPLYING BEP-TT TO RECURRENT ARCHITECTURES
One of the main strengths of BEP that its global error- propppropagttion strategy directly extends to RNNs, a class of models that require explicit temporal credit assignment and is generally intractable for local learning rules. By unrolling the NN through time, we obtain binary counterpart of the BP Through Time (BPTT) algorithm, which we call BEP-TT. Demonstrating its effectiveness provides key validation of BEP, showing that a global, end-to-end binary error signal can train recurrent architectures. Accordingly, we employ a binary RNN for many-to-one sequence classification, as shown in Figure lb Given an input sequence x" = (x....., xx) stte-to-sstat weights Hgs E 2 K, X, input-to-state weights H.s E 2 K, X,,, and state-to-output weights H.y E Z k., along with their binary counterparts W ss. W ass and W sy-
of length T. the RNN is defined by
Forward Pass Through Time. At each time step E 1,..., T, the state vector s E (+1) is computed from the previous state s/_ and the current binarized input af E [=1) k= according to the recurrence s/¹ sign( W af + W W..)"-n). After the final time step T, the state s is mapped to an output activation s E [=1] K2 via sh = sign( ( ). This activation is then passed to the fixed classifier P to produce the prediction y² which is used to check the update criterion from Eq.

---

Published as a conference paper at ICLR 2026
Binary Backpropagation Through Time. When sequence x" satisfies the update criterion in T the BEP-TT procedure is performed. This mechanism parallels BEP's operation on feedforward NNs but operates over the unrolled temporal structure, propagating target state vector s." backward from time step = 7 to t = 1. First, the desired output state is set to the correct class prototype, / p= which provides the error signal for the Mate-to-output layer. Using Eq. 7 this signal is
then back- projeceeeeeted to obtaín the desired state at the last time step, sy sign sy (g) @ s7")).
For preceding time steps E {T - , }]. the desired state is propagated recursively:
= sign (W.. Ns (g @ s241))
Here, g and 8i+1 are the backward gates computed from the pre-activations z and - This re- cursion propagates the binary error signal through the sequence, enabling temporal credit assignment.
Accumulated Updates for Shared Weights. Because RNNs share weights across time, the matrices Hys and H.s must accumulate updates over all time steps for all samples μa M that triggered an update. These aggregated updates are then applied to the integer-valued hidden weights using the corresponding binary masks M and M as described in Eq. Eq.Bland Eq.
T Σ (s) (*
HGs <- H + LEM M & (s)"((a))
H.s +- H.S +
Σ PEM
M 88
NEM
These updates are followed by the reinforcement step described in the previous section. The binary masks M and M shared across time steps to preserve weight tying, ensuring masking commutes with temporal aggregation. Allowing masks to vary with time would violate this property. effectively training different copies of layer is updated analogously to the feedforward case described in Section 3.2
the same parameter at each time index. The final state-to-output
Hey Hay
Σ (M O o
PEM
This BEP-TT formulation yields a fully binary training procedure for binary RNNs. Its ability to perform temporal credit assignment stems from the global error propagation mechanism of BEP, key advantage over local learning rules.
EXPERIMENTAL EVALUATION
In this section, empirically evaluate the proposed BEP algorithm. The experiments have two main objectives: (i) to benchmark BEP against the SotA method for binary training of MLPs (Colombo) al.] 2025),, and (ii) to demonstrate the generality of BEP's global credit assignment by applying to binary RNNs, setting for which local learning methods are unsuited. The Python code used in our experiments available
public repository Section 4. describes the experimental setup. Section 4.2 evaluates evaalluatss BEP binary MLPs against both the SotA approach and QAT-based methods that rely on full-precision BP. Section 4.3 tests BEP on binary RNNs, contrasting its performance with QAT. Finally, Section explores the effect of the gating hyperparameter on training dynamics.
4.1 EXPERIMENTAL SETUP
Our evaluation spans a diverse set of benchmarks. For the feedforward MLPs, we adopt the same datasets (Colombo et al. 2025): Random Prototypes, FaShionMNIST (iao et al. 2017). CIFAR- 10 (Krizhevsky et al. 2009), and Imagenette (Howard) 2019). The Random Prototypes dataset comprises 20,000 training and 3,000 test samples with input dimension K, = 1000 and a flip probability = 0.46. FashionMNIST contains 50,000 training and 10,000 test grayscale images of size 28 x 28. For CIFAR-10 and Imagenette, in (Colombo et al. 2025). Both are 10-class datasets; CIFAR-10 has 50,000 training and 10,000 test samples, while Imagenette has 10,000 training and 1.963 test samples. To evaluate recurrent models, we use Sequential MNIST (S-MNIST) (Deng) 2012) and 30 real-world time-series classification tasks from the UCR Archive (Dau et al., 2019), which cover a wide range of sample sizes, feature dimensions, and class counts. In all experiments, we utilize a fixed output classifier P generated via the binary equiangular frame method detailed in Appendix
use the
Neex-ee-de features as
thtps://gttuu..co///a Recheech-- Research-Laah/BFT
7

---

Published as a conference paper at ICLR 2026
5143
143
183
3143
2.83
711
700
741
133
60c
(a) Random Prototypes
(b) MNIST
(c) CIFARIO
(d) Imagenette
Figure 2: Test accuracy as function of the number of parameters on Random Prototypes, Fashion- MNIST, CIFAR10, and Imagenette. Results compare BEP with both the SotA approach (Colombo)
al. 2025 and QAT-based methods for binary MLPs with L, 2 and I. hidden layers.
4.2 PERFORMANCE ON BINARY MULTI-LAYER PERCEPTRONS
Our first experiment evaluates BEP against the current SotA approach for binary training of MLPs (Colombo et al.) 2025). Specifically, we considered binary MLPs with two and three hidden layers, trained using both BEP and the SotA local learning rule. The tuned hyperparameters, across = 50 epochs, include the robustness parameter reinforcement probability pr, initial group size To. and gating threshold v. For a comprehensive benchmark, we also trained the same architectures using the QAT implementation from the Larq framework (Geiger& & Team 2020). In this setup, the forward pass employs binarized weights, while the backward pass updates latent FP parameters with the Adam optimizer (Kingma & Ba 2014). To ensure fair comparison, the main QAT baseline does not use batch normalization, resulting in a fully binary model at inference. For completeness, we also report the performance of QAT with batch normalization as reference point.
Figure 2 reports test accuracy, averaged over 5 runs, on four datasets as a function of the parameter count. The results show that both fully binary approaches BEP and the SotA local rule signifi- cantly outperform the comparable QAT baseline across all configurations. When comparing binary methods, BEP consistently surpasses the SotA approach, achieving improvements of up to +6.89% +1.22% +3.7--377000 70%, and +2.85% on Random Prototypes, FashionMNIST, CIFAR-10, and Imagenette, respectively, at the smallest parameter configurations. As model size increases, the performance gap narrows, with BEP matching the local credit assignment rule but falling slightly behind in one high-parametee setting (CIFAR-IO, 10, 3 layers). These findings underscore the importance of global error propagation for effective credit assignment in binary MLPs, while also showing that BEP outperforms both local learning rules and standard QAT baselines without relying on FP gradients.
4.3
VALIDATION ON BINARY RECURRENT NEURAL NETWORKS
Our second experiment tests the BEP algorithm on binary RNNs using its time-unrolled variant BEP-TT described in Section 3.5 The goal is to demonstrate that a global, end-to-end error signal can effectively train recurrent models, a task typically intractable for purely local learning rules. Specifically, we evaluate BEP-TT on many- seany-to-ome to-one sequence classification tasks across 30 datasets from the UCR Time Series Archive (Dau et al. 2019) using only the last window of each time series. All RNN results use 3-fold cross-valddation and 3 independent runs with the same hyperparameters: robustness r = 0.5, reinforcement probability p. 0.5, initial group size Vo.1 = 15, gating threshold 0.05, hidden and output layer sizes K. K 1035, training epochs = 50, and batch size bs = N/10. As a baseline, we trained binary RNNs with the same architecture using the QAT implementation from the Larq framework (Geiger & Team 2020). As in the previous experiment, no batch or layer normalization was applied so that the models remain fully binary at inference time. For completeness, report the performance of QAT with batch normalization reference point.
8

---

Published as a conference paper at ICLR 2026
Table 1:
Test accuracy on 30 UCR datasets. The results compare our BEP algorithm with the
DAT-based training from the Larq framework (Geiger & Team 2020) for binary RNN architectures.
UCR Datasct
BEP
QAT w/o batchnomm
QAT batchnorm
UCR Datasct
BEP
QAT w/o batchnorm
QAT hatchnorm
AricularyWonRRe.
$1.28 + 2.99
51.94 3.61
77.62 + 3,80
Jopanese Vowels
95.47 1.24
84,06 + 2.92
96.25 + 1.01
Cricket
86.85
1 6.88
86.11
Melbeurne Pedestrron
73.03
2.35
90.91
DistaPOA
79.78 2.65
73.84
2.51
76.94
3.27
MoteStrain
78.62 2.29
74,42
1.70
79.82 1.81
ECG5009
91.40 0.53
88.20 : 1,08
92.71 0.74
SeR
74.25 2.07
67.97 1 1.80
77.36 1.89
ECGFiveDeys
81.79
2.71
95.25
1.45
PEMS-ST
86.13
60.69 +: 4.55
85.83
ERing
82.44 1 4,90
71.67 3.89
79.11
1 44993
PenDigits
97.13 0.21
66.99
1.44
99.00
0.21
Earthquakes
4 2.94
79.83 + 2.83
77.01
2.96
ProstmalPOAgeet Group
82,64 3,08
77.24 3.64
82.20 2.27
Epilepsy2
92.45
88.82
0.58
93.72
0.42
Proximal PO POO
78.26
72.76 2,89
75.68 2.66
FreezerRegula
78.21 1.17
75.26
1.56
85.92
1.06
Proximal PhalanX7W
79.83
74.26
75.59 3.03
Smoll
18.16
1.50
SmoonkSuhsgice
49.00
58.00
90.33
Gam CamPtAge AeAe
86.18 2.27
81.08 2 4.53
84.11 1 2.55
SomyAlBORobottin
75.68
64.14
80.68 3.04
GunP Oodloooeeeerrooom
94.01 +: 1.57
2.911
93.79
1.26
Mon.A/BORoboSSure
82.04
73.10 1.71
84.59 2.62
Insect GreeeEEGReuulrTrrui
99.14
0.711
99.04
StarLightCurver
81.96.10.00
InseriE GSmaliTraia
99.37
0.56
96.62 + 2.43
98.
0.83
Tiselar
0.47
64.52 0.998
84.81 0.25
94.65 + 1.22
83.18
96.23
1,01
Wider
9.69
95.25 + 0.64
97.93
0.24
For all datasets and models, inputs were binarized with a distributive thermometer encoder (Bacellar al. 2024a), followed by fixed +1 expansion layer projecting the dimension to Ko 1035, A hyperparameter search was conducted to tune the sequence window length, defined as the number of timesteps included in the temporal window, and the number of thermometer bits.
Table D presents the test accuracy across the considered UCR tasks. On every dataset, BEP-TT consistently outperforms the comparable QAT baseline in training fully binary RNNs, achieving an average test accuracy improvement of + +10.57% These results validate the proposed binary error propagation mechanism beyond feedforward MLPs. Moreover, because BEP relies on bitwise operations even during training. it substantially reduces memory and computational costs compared to QAT, which requires full-precision Adam updates, as discussed Section 4.3
4.4 THE ROLE OF THE GATING THRESHOLD
A crucial element to the generalization performance of the BEP algorithm is its gating mechanism, which modulates the backward error signal as described in Section 3.2 In this third experiment, we examine how varying the threshold y affects the validation accuracy of a binary RNN on the S-MNIST dataset. The following hyperparameters are used: robustness r 0.5, reinforcement probability P. = 0.5, initial group size 'uu = 15, layer sizes K2 Kg 1875, training epochs 50, and batch size bs = 100. Corresponding results for binary MLPs are provided in Appendix D.I
Figures 3a and 3b which show the accuracy averaged over 3 runs as function of window length and gating threshold v, respectively, highlight the crucial role of this mechanism. Its effect becomes more pronounced as the window length grows (corresponding to deeper backward steps through time), emphasizing the importance of focusing updates on neurons whose pre-activations lie near the decision boundary. The results reveal an optimum v: very low or very high thresholds degrade performance, whereas intermediate values (around 10 10-2) consistently yield the highest accuracy across different temporal depths. By filtering out saturated neurons from the error signal, the gate ensures that weight updates focus on parts of the BNN most susceptible to flipping their activations.
4.5 DISCUSSION AND LIMITATIONS
key advantage of BEP lies in its ability to perform training using only bitwise computations. Table 2 summarizes the memory footprint and computational complexity of BEP compared with standard QAT. Relative to QAT-based methods, our approach achieves 2x reduction in memory usage for hidden weights and a 32x reduction for error signals and weight updates. Moreover, Adam-based QAT requires an additional 64 bits per parameter to store its first- and second-order moment estimates, further increasing its memory demand. From computational standpoint, we adopt the hardware-level metric of equivalent Boolean gates (Colombo et al. 2025) to compare the intrinsic cost of arithmetic operations. All operations in BEP can be implemented using primitives such as XNOR, Popcount, and increment/dccremen operations, which require at most (11N) Boolean gates for --bit operands (Bacellar al. 2024b). In contrast, IEEE-754 single-precision

---

Published as a conference paper at ICLR 2026
Table 2: Comparison of memory footprint and computational complexity between QAT (with Adam) and BEP. Memory is reported in bits per element, and complexity denotes the order of magnitude of equivalent Boolean-gan -agtee operations per element (Colombo al. 2025). f This estimate assumes the absence of batch normalization and full-precision scaling factors commonly used in QAT.
Memory (Bits)
Complexity (Boolean gates)
Method
Weights
Activations
Errors / Gradients
Forward
Backward
Update
QAT (Adam)
32 (FP32)
1
32 + 64 (Moments)
101
104
104
BEP
16 (Int16)
1
~ 10
10
10
95.0
92.0
15.0
830
150
220
71.
150
820
y 1,00
1.01
610
610
510
130
55.
510
Window langt
Freacthratin threshole
(a) Accuracy as function of window length
(b) Accuracy function of gating threshold v
Figure 3: Validation accuracy of a binary RNN trained with BEP on the S-MNIST dataset for different
values of the gating threshold hyperparameter and window length WL.
FP additions and multiplications (876 2019). which underpin Adam-based QAT, are estimated to require on the order of 10t Boolean gates each (Luo & Sun 2024). This comparison highlights the substantial computational advantage of BEP, which reduces the Boolean-gatt cost by approximately three orders of magnitude relative to FP32 QAT with Adam.
Despite its promising results, our approach has some limitations that suggest natural directions for future research. First, we focus primarily on binary MLPs and RNNs. Extending BEP to convolutional or transformer-style models requires a full binary design for these architectures, including handling binary convolutions, filter-level masking, and additional adaptations to support weight sharing. spatial structure, and multi-head mechanisms. Second, our experiments are restricted to classification tasks. Although BEP naturally supports arbitrary binary output vectors and could, in principle, applied to multi-labe llbel prediction, binary segmentation, more general binary-vector regression, such tasks necessitate additional design choices regarding the output encoding. Third, we evaluate BEP on mid-scale datasets and mederate-depth NNs. Extending BEP to large-scale models (e.g., ImageNet- level CNNs (Bulat & Tzimiropoulos, [019)] Vargas et al. 2024)) requires substantial architectural adaptations to convolutional or transformer backbones. Moreover, binary-compatible normalization mechanisms may become necessary to ensure stable training in such settings.
5 CONCLUSION AND FUTURE WORK
In this paper, we introduced BEP, an algorithm for training multi-layer BNNs using exclusively binary computations. The central contribution the formulation of principled, binary analog of the BP algorithm. By defining a recursive rule for propagating binary-valued error signals and updating integer-valued metaplastic weights, BEP bridges the gap between the global credit assignment of gradient-based learning and the computational efficiency of bitwise operations. Theoretically. this work shows that effective end-to-end learning multi-layer binary NNs possible without relying on continuous gradients, opening new avenues for analyzing discrete optimization in DL. Practically, BEP enables efficient training using only XNOR, Popcount, and incremen//decremeen operations, making it well-suited for constrained settings such as TLn (Capogrosso al. 2024 Pavan et al.)
2024).
DL with homomorphic encryption (Falcetta & Roveri 2022 Colombo
al. 2024), and neuromorphic systems (Indiveri & Liu 2015: Yamazaki et al. 2022). Future work
includes extending BEP to convolutional architectures, refining adaptive strategies for the gating
threshold v, exploring learnable masking mechanisms, and developing formal convergence analysis.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This paper is supported by Dhiria S.r.l. and by PNRR-PE-AI FAIR project funded by the NextGener- ation EU program.
REFERENCES
leee standard for floating-poln arithmetic. IEEE Std 754-2019 (Revision of IEEE 754-2008), pp- 8484 2019. doi: 10.1 (E110//EEESTD.2099987768
Alan Tendler Leibel Bacellar, Zachary Susskind, Mauricio Breternitz Jr, Eugene John, Lizy Kurian John, Priscila Machado Vieira Lima, and Felipe M.G. França. Differentiable weightless neu- ral networks. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning. volume 235 of Proceedings of Machine Learning Research, pp. 2277-2295. PMLR, 21-27 Jul 2024a. URL SttEp:://pzoceedinnee wwwiinngss..r.r..m.mmmm prw.s..rees//23 bacellar24a.. html
Alan TL Bacellar, Zachary Susskind, Mauricio Breternitz Jr, Eugene John, Lizy John, Priscila Lima, and Felipe MG França. Differentiable weightless neural networks. arXiv preprint
1XXi::4410.11112 2024b.
Carlo Baldassi. Generalization learning in perceptron with binary synapses. Journal of Statistical Physics, 136(5):902-916, Sep 2009. ISSN 1572-9613. doi: 10. 0011007//10955--09-98222-- URL
httpps:/ddo www.oorg.............oooo
Carlo Baldassi, Alfredo Braunstein, Nicolas Brunel, and Riccardo Zecchina. Efficient supervised learning in networks with binary synapses. Proceedings of the National Academy of Sciences, 104 109111007-110844 2007. doi: (0.10773pnas.070003
0700324104 URL.http. /www pnw.sppass........ org/
doi/abs/ 10. 1073//111.ppppso wwtp:/aaa....00...00 0700324104
Carlo Baldassi, Alessandro Ingrosso, Carlo Lucibello, Luca Saglietti, and Riccardo Zecchina. Sub- dominant dense clusters allow for simple learning and high computational performance in neural networks with discrete synapses. Physical review letters, 115(12):128100 2015.
Yoshua Bengio, Nicolas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint
2013. 2013.
Adrian Bulat and Georgios Tzimiropoulos. Xnor-net++: Improved binary neural networks. ArXiv, abs/1909. 13863, 2019.
203593135
Luigi Capogrosso, Federico Cunico, Dong Seon Cheng, Franco Fummi, and Marco Cristani. A machine Gaaming-oriented survey on tiny machine learning. IEEE Access, 12:23406-2333 22426,,22 2024.
Luca Colombo, Alessandro Falcetta, and Manuel Roveri. Training encrypted neural networks encrypted data with fully homomorphic encryption. WAHC'24, pp. 64, 2024.
Luca Colombo, Fabrizio Pittorino, and Manuel Roveri. Training multi-layer binary neural networks with random local binary error signals. Machine Learning: Science and Technology, 6(3):035015,
2025.
Matthieu Courbariaux, Yoshua Bengio, and -aen-i David. Binaryconnect: Training deep neural networks with binary weights during propagations. In Advances in Neural Information Processing Systems (NeurIPS), pp. 3123-3131 2015.
Hoang Anh Dau, Anthony Bagnall, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, and Eamonn Keogh. The uer time series archive. IEEE/CAA Journal of Automatica Sinica, 1293 2019.
Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web). IEEE signal processing magazine, 11-1,,21-1 2012.
11

---

Published as a conference paper at ICLR 2026
Andreas Engel. Statistical mechanics of learning. Cambridge University Press, 2001.
Alessandro Falcetta and Manuel Roveri. Privacy-presse srvaac-preserving deep learning with homomorphic encryption: An introduction. IEEE Computational Intelligence Magazine, 17(3):14-25 25, 2022.
Elizabeth Gardner. The space of interactions in neural network models. Journal of physics A:
Mathematical and general, 21(1):257, 1988.
Lukas Geiger and Plumerai Team. Larq: An open-source library for training binarized neural networks. Journal of Open Source Software, 5(45):1746, 2020.
Geoffrey E Hinton. Connectionist learning procedures, In Machine learning, pp. 510-610. Elsevier, 1990.
Jeremy Howard. imagenette, 2019. URL Ott.ttpps///tttab..occo//aaaaaaa
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran EI- Ylniv,,v, and Yoshua Bengio. Binarized neural networks. In Advances in Neural Information Processing Systems (NeurIPS), pp. 4107-4115, 2016.
Giacomo Indiveri and Shih-Chii Liu. Memory and information processing in neuromorphic systems. Proceedings of the IEEE, 103(8):137--139)) 2015.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 14/2.6980 2014.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barge Barwinskw, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences (PNAS), (1((13):3521-3322 2017.
Scott Kirkpatrick, C Daniel Gelatt Jr. and Mario P Vecchi. Optimization by simulated annealing. science, 220(4598):671-6888 1983.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images, 2009.
Xiaofan Lin, Cong Zhao, and Wei Pan. Towards accurate binary convolutional neural network. Advances in neural information processing systems, 30, 2017.
Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng. Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm. In Proceedings of the European Conference Computer Vision (ECCV), September 2018
Zechun Liu, Wenhan Luo, Baoyuan Wu, Xin Yang, Wei Liu, and Kwang-Ting Cheng. Bi-real net: Binarizing deep network towards real-network performance. International Journal of Computer Vision, 128:202-219, 2020.
Ilya Loshchilov and Frank Hutter. Cma-es for hyperparameter optimization of deep neural networks. arXiv preprint 1rXiv:1604. (666607269, 2016.
Carlo Lucibello, Fabrizio Pittorino, Gabriele Perugini, and Riccardo Zecchina. Deep learning via message passing algorithms based on belief propagation. Machine Learning: Science and Technology, 3(3):035005, 2022.
Hongyin Luo and Wei Sun. Addition all you need for energy-efficient language models. arXiv preprint arXiv:2410.0090) 2024.
Vardan Papyan, X. Y. Han, and David L.. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences. 117(40): 24652 24663, 2020. doi: 10. 10.1073/pmms URL https: /www.wwwwoooooo www.:ppaaa...m org/doi/ abs/ 10. 101.0.107333pmmm 0at.aaass..11111111111
12

---

Published as a conference paper at ICLR 2026
Massimo Pavan, Eugeniu Ostrovan, Armando Caltabiano, and Manuel Roveri. Tybox: An automatic design and code generation toolbox for tinyml incremental on-device learning. ACM Transactions on Embedded Computing Systems, 23(3):1-27, 2024.
Haotong Qin, Ruihao Gong, Xianglong Liu, Xiao Bai, Jingkuan Song, and Nicu Sebe. Binary neural networks: A survey, Pattern Recognition, 105:107281 2020.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. XNOR-Net: ImageNet classification using binary convolutional neural networks. European Conference on Computer Vision (ECCV), pp. 525-542, 2016.
F. Rosenblatt. The perceptron: A probabilistiic model for information storage and organization in the brain. Psychological Review, (5(6):386 408, 1958. ISSN 0033-295X. doi: 10.1037/h0042519. URL http::t:p...doo.ccoooo
David E. Rumelhart, Geoffrey Hinton, and Ronald Williams. Learning representations by errors. Nature, 323(6688)::533-533, Oct 1986. ISSN 1476-4687. doi: 10.1038/
323533a0. URL http::ttp:///ttt //oi.. wttpa/..00011111/12333333333
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as scalable alternative to reinforcement learning, arXiv preprint arXiv: 2017.
Ratshih Sayed, Haytham Azmi, Heba Shawkey, Alaa Hussein Khalil, and Mohamed Refky. A systematic literature review on binary neural networks. IEEE Access, 11:27546-27578, 2023.
Riccardo Schiavone, Francesco Galati, and Maria A Zuluaga. Binary domain generalization for sparsifying binary neural networks. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 123-140. Springer, 2023.
Felipe Petroski Such, Vashisht Madhavan, Edoardo Conti, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Deep neuroevolution: Genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning. arXiv preprint arXiv: 1712v:7712..66667, 2017.
Zhijun Tu, Xinghao Chen, Pengju Ren, and Yunhe Wang. Adabin: Improving binary neural networks
with adaptive binary sets. In Computer Vision - ECCV 2022: 17th European Conference, Tel
Aviy, Israel, October 23-27, 2022, Proceedings, Purt XI, pp. 379-395, Berlin, Heidelberg, 2022. doi: 00009/99778--0012008---000 URL https
Springer- Verlag. ISBN 978
//doi. 10. 1007 /
Edwin Vargas, Claudia V. Correa, Carlos Hinojosa, and Henry Arguello. Biper: Binary neural networks using periodic function. In 2024 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 5684-5693, 2024. doi: (0.110//CPPR3333222220000
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv: 07747, 2017.
Kashu Yamazaki, Viet-Khoa Vo-Ho, Darshan Bulsara, and Ngan Le. Spiking neural networks and their applications: A review. Brain sciences, 12(7):863, 2022.
Penghang Yin, Jiancheng Lyu, Shuai Zhang, Stanley Osher, Yingyong Qi, and Jack Xin. Under- standing straight througgh estimator in training activation quantized neural nets, arXiv preprint arXiv:1903.05662, 2019,
13

---

Published as a conference paper at ICLR 2026
A ANALYSIS OF THE BACKWARD PASS
is possible to provide a justification for the core backward propagation rule. In particular, we show that the recursion for the desired activations is the exact optimizer of tractable linear surrogate of the otherwise intractable combinatorial credit-assigmmnt problem. This positions BEP as principled binary analog of BP rather than ad-hoc heuristie. At layer I, one would ideally choose the binary activation vector a/" that, when mapped through W4+1 and binarized, maximizes alignment with the upper-layer target 4 From Eq.
arg max (411 sign( (Wi-1a)). aE(11)%
This is a nonconvex combinatorial optimization over the hypercube with a discontinuous objective. In general, it is NP-hard by reduction from standard binary optimization problems. Therefore, do not attempt to solve exactly. Instead, we consider the linear surrogate obtained by dropping the nonlinearity inside the inner product. From Eq.
arg max (a W W1+1a). aE(=1)%;
Equivalently, the set may be relaxed to the hypercube |- 1Jki and the optimum remains at vertex. Proposition (BEP back- projecctojooon solves the linear surrogate exactly). Let = "41aa4i RK The set of maximizers of Eq. iis
a E sign(v) a (=1) a, sign(v)) if + O, a {+1} = (II)
ie., any coordinate-wise sign choice consistent with v. In particular, when no coordinate tie occurs, a sign( WT (41 a/41), which is exactly the BEP recursion (Eq. without gating).
Proof. By the adjoint identity Av) = (A (u, v).
Kr a) Σ Ujag.
(a * 14
* W441a) (W 1+1 1+
The objective is separable across coordinates the product set (+1] Ki, so it is maximized by choosing each a; to maximize 0;0,, i.c., a; =sign(e;) if 0; + 0 and any a; E { if = 0. L
Lemma (Convex relaxation has an integral optimum). The convex relaxation of Eq. with E [_1, Kk has the same optimal value, and the set of maximizers is
{ a E {-1,1k,)
(
clip
which reduces to Eq. at the vertices. Hence the linear surrogate is solved exactly at a binary point.
Proof. Maximizing linear function over a hypereube attains the optimum a vertex. Coordinate- wise, the same separability argument as above applies.
Including the gate. Recall the binary gate from Eq. 5 and write D := diag(g"" The BEP
recursion with gating from Eq. 7 replaces a by D a i.c., it solves the masked surrogate
arg max (Di a W1+1a) Kr
a E sign ( (i a(
(12)
Thus, the gate simply out saturated coordinates of the upper-layer target before back projerroecttion, directly mirroring the role of derivative clipping in STE-based BP.
Proposition] I and Lemma 2 show that the BEP backward rule is the analytical optimizer of well- posed linear objective approximating the intractable target-selection in Eq. The gate induces a diagonal mask in that objective, yielding the exact masked optimizer in Eq. 12 Practically, this explains why BEP focuses the learning signal on neurons near their decision boundary (unsaturated coordinates) and provides principled binary analog of gradient gating used by STE-based methods.
14

---

Published as a conference paper at ICLR 2026
A.1 PROOF OF LEMMA
Proof. The gated scalar product can be expressed as (b, b²) - (g + b, b') and by the adjoint
identity (b, Wa) = (W by a). Combining these two relations leads to
(b, Wa)g (W (g C b).
Denoting W (g @ b) as z of components in E {0), we reduce problem arg maxa (z, a). The
objective is separable across coordinates: (z, a) = E; 2, so maximized by choosing each o to maximize 2,04, i.e., ag sign(2:).
B LOCAL CORRECTNESS OF THE WEIGHT UPDATE
Beyond justifying the backward pass, we also show that the resulting weight update is beneficial in layer-local sense. The next lemma proves that, for any sample triggering an update, the modification to the hidden integers Hi is guaranteed to be corrective: it pushes the neuron stabilities in the direction of the desired activation and increases an anchored alignment potential by a fixed, known amount.
Lemma (Local update correctness on the stabilities). Fix a sample μ E M that triggers an update. Let I be layer and j neuron selected by the neuron-wiss mask (i.e., the j-th column). Denote the hidden weights before and after the update by H4 and Hi, respectively, and define the stabilities
" = (a hi.s
== (;
the neuron-wise update Aht a (and zero otherwise), then the alignment strictly increases by a fixed amount:
* 14 = ap ufi + +KK111
01 [1
Proof.
1 = (a his 20 a = uhy + 2a 1
Since a K+ ||a Ki-1. Multiplying by a (+1) yields the claim.
Remark (From stability to visible areavtivatioon. The forward pre-activations use visible weights Wi = sign(H(), hence 2[.j 1a WL.)) can change discontinuously when entries of h cross zero. Nevertheless, Lemma 3 immlies monotonic drift of each coordinate of bus toward the signed target a After T updates of neuron j. each entry has shifted by 2T in the correct direction. Consequently, once
H (1))((0) +1
T max
2
all entries align with stable under further updates the anchored desired activations.
and the visible pre-activations satisfy sign(2f;,) (2t;;) ai and remain
Lemma shows that each neuron update yields a strict quantifiable increase of an anchored alignment by 2K (a discrete analog of guaranteed descent step). Together with Remark [[] this ensures that repeated anchored updates drive the visible state toward the desired activation and stabilize it once sufficient integer margin accumulates. full convergence proof left for future work.
C GENERATING A FIXED BINARY CLASSIFIER VIA EQUIANGULAR FRAMES
As stated in Section 3 our empirical results show that BEP achieves its best performance when using fixed output classifier P whose class prototypes are geometrically well-separated. This approach is inspired by the concept of Equiangular Tight Frames (ETFs), which have been shown to emerge the final layers of deep NNs during phenomenon known as neural collapse (Papyan et al. 2020). While a simple, randomly generated classifier offers a baseline, optimizing the structure these prototypes significantly improves class separability. This appendix details our method for generating structured binary classifier by constructing a set prototype vectors that are maximally and uniformly distant from each other in the binary feature space.
15

---

Published as a conference paper at ICLR 2026
Neural collapse describes empirical phenomenon where, in the terminal phase of training, the last-layer feature representations for all samples of a given class collapse to a single point (their class mean). Furthermore, the set of these class-mean vectors, along with the final-layer classifier weights, form a simplex ETF. A simplex ETF is a geometric configuration of vectors that are maximally separated from one another, characterized by equal norms and constant, negative pairwise inner product. This structure is optimal for linear classification.
The classical ETF is defined in a real-valued vector space RD In BNNs, we are interested in a binary analog where the feature vectors and classifier weights lie on the hypercube {=1]². We define a Binary Equiangular Frame (BEF) as set of C' binary vectors (pi..... PPC). where p. E {+1),. that satisfy two properties. The first property is high pairwise separation, i.e. the inner product (p):P)) for i * j should be as small (i.e., as negative) as possible. In the binary domain, this is equivalent to maximizing the Hamming distance between any two vectors. The second property is equiangularity, i.e. the inner products for all distinct pairs (i.PP) should be approximately equal. Such frame, when used as the columns of the classifier matrix P. provides a set of target prototypes that are maximally and uniformly distant from each other in the binary feature space.
Finding an exact BEF is a hard combinatorial problem. However, we can generate a high-quality approximation using a binary optimization procedure. Given the desired number of classes C and feature dimension D, we seek to find the set of vectors {pe]e=) that minimizes the cost function
J((p.)) = (P:. + ((),)),
(13)
iCj
where Var(.) is the variance and a 2 0 balances the two objectives. The first term encourages all pairwise inner products to be negative, while the second pushes them toward common value.
We optimize this objective using a simple iterative local search algorithm, starting from a random initialization of the C' vectors } from [=1] For a fixed number of iterations: (i) randomly select a vector p; and a coordinate k; (ii) compute the change in cost AJ that would result from flipping the sign of the k-th element of p:: and (iii) if AJ < O, accept the flip. This greedy coordinate-wis. descent procedure rapidly converges to a local minimum of the cost function. The resulting set of vectors {p.} can then be used to construct the fixed classifier matrix P [P))...,Pc]
D ABLATION STUDIES
D.1 THE ROLE OF THE GATING THRESHOLD ON MLPS
In this section, we present an ablation study on the gating threshold v for binary MLPs, as shown in Figure Empirically, across layer sizes, optimal and non-trivial values of v emerge, with the effect becoming more pronounced as NN depth increases, consistent with the RNN ablation in Section 4.4
244
2011
344
Dark
onl hrseee,
2014
20:1
Mate 2011
(a) Random Prototypes
(b) MNSSI
(c) CIFAR1O
(d) Imagenette
Figure 4: Validation accuracy as a function of the gating threshold v on Random Prototypes, Fashion- MNIST, CIFAR10, and Imagenette for binary MLPs with L 3 and L = 5 hidden layers.
16

---

Published as a conference paper at ICLR 2026
D.2 THE ROBUSTNESS HYPERPARAMETER
In this section, we perform an ablation study on the robustness parameter r for binary MLPs, as shown in Figure Empirically, setting the margin in the range 0.5-0.75 consistently yields the best generalization accuracy across the considered tasks and architectures.
2100
740
24
6.3
6.3
303
C2
0.
X71
140
400
(a) Random Prototypes
(b) FasHionMNIST
(c) CIFARIO
(d) Imagenette
Figure Validation accuracy as a function of the robustness r on Random Prototypes. FISH. CIFARIO, and Imagenette for binary MLPs with L = 2 hidden layers.
D.3 THE REINFORCEMENT PROBABILITY HYPERPARAMETER p.
In this section, we perform an ablation study on the reinforcement probability p. for binary MLPs, as shown in 6 Empirically, setting the probability p, in the range 0.5-0.75 consistently yields the highest gencralization accuracy across the evaluated tasks and architectures, although its overall impact remains moderate. Nevertheless, compared to disabling reinforcement entirely, enabling provides a measurable improvement, with the effect being more marked in lower-capacity models.
4.1
026
(a) Random Prototypes
(b) MNSS
(c) CIFARIO
(d) Imagenette
Figure Validation accuracy as a function of the reinforcement probability p. on Random Prototypes, FashionMNIST,, CIFARIO, and Imagenette for binary MIPs with T. hidden layers.
D.4 ANALYSIS OF BINARIZATION FUNCTION
To examine the sensitivity of BEP the input binarization method, compare simple median thresholding against thermometer encoding on five UCR datasets using binary RNNs. As shown in Table 3 thermometer encoding significantly outperforms median thresholding time-series data, it preserves coarse magnitude information that is essential for these tasks. In contrast, for image data, median thresholding sufficient achieve SotA performance with BEP.
Table 3: Validation accuracy on five UCR datasets using different thermometer--ncooding bits for input binarization. Notably, using single bit corresponds to the meddiannthresooldingg method.
Thermoneterrenccnding bits
Dataset
10
20
50
100
AnticularyWordRe..
48.98 + 3,93
81.28
2.99
79.37
2.70
81.16
2.17
81.45 1.85
DistalPOAgeGroup
77.18 2.74
78.92 +
78.11 2.03
79.78 + 2.65
79.78
Demaandd
73.54 + 2.44
94.65 = 1.22
94.59 1.13
94.34 1.03
93.92
PenDigits
61.60 + 2.05
96.66 + 0.22
96.79 0.35
96.88 # 0.26
0
ProximalPOAgeCrron
77.52
82.31
+328
82.37 3.36
82.70 2.41
82.53 + 2.85
2.85
17

---

Published as a conference paper at ICLR 2026
D.5 ANALYSIS OF MASK DENSITY
To substantiate the claim that the sparsity mask fulfills a role analogous to the learning rate, we evaluate the impact of the group size "o. (which determines mask density) training dynamics for binary MLPs. Figure shows that smaller group sizes (sparser updates) lead to slower convergence, whereas larger group sizes (denser updates) accelerate initial learning but may introduce instability, This behavior mirrors the effect of varying the learning rate in madiient-based optimization.
bpcke
bobs
2-44
(a) Random Prototypes
(b) MNIST
(c) CIFARIO
(d) Imagenette
Figure 7: Training and validation accuracy curves over epochs for different values of the group size Too/ on Random Prototypes. FashionMNIST. CIFARIO, and Imagenette.
D.6 SCALING DEEPER NETWORKS
To assess whether BEP can propagate informative error signals through deep computational chains, we evaluate binary RNNs on the S-MNIST dataset while progressively increasing the backward horizon length (i.e., the number of backpropagated time steps). This setup effectively simulates deeper networks by extending the length of the backward computational graph while keeping the temporal window fixed. As shown in Figure 8 performance steadily improves as the backward horizon increases, indicating that BEP successfully propagates useful binary error information over many steps without suffering from excessive signal decay.
$6.0
19.0
60.0
Sackward steas thrugg time
Figure Validation accuracy of binary RNN trained with BEP on the S-MNIST dataset for different values of the backward horizon length (depth of backpropagation).
18


Reviews:
[{"review_id": "aTOPsNRCsw", "reviewer": "Reviewer_N7bj", "summary": "The work proposes a novel alternative to the backpropagation algorithm by reformulating it in the binary domain to reduce computational cost. Unlike Quantization-Aware Training, BEP performs both the forward and backward passes entirely in the binary domain, relying solely on bitwise operations. The paper provides theoretical foundations and experimental results that validate this reformulation of backpropagation.", "questions": "How does BEP scale computationally compared to QAT during training?\n\n\n1. How sensitive is BEP to initialization and hyperparameter selection across datasets?\n\n\n2. Does the algorithm have similar results across different tasks to classification, for instance, binary segmentation?\n\n\n3. How does BEP handle vanishing or exploiting gradients in comparison to the standard backpropagation algorithm?\n\n\n4. How does the algorithm work for larger spatial dimensions and complex feature datasets, such as the celebA dataset with the multilabel classification task?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Novelty: the paper introduces an alternative approach to regular backpropagation by reformulating the algorithm to work for binary weights.\n\n\nDifferent from other approaches that rely on real value parameters, like a straight-through estimator, this work focuses on the idea of computing forward and backward on the binary domain. \n\n\nThe work displays consistent improvements over the classification task on two datasets.", "weaknesses": "1. Lack of organization, it is difficult to read smoothly, as figures, such as Figure 2, are on page 7, and mentioned in page 8, same as Table 1, where it is mentioned in page 9, and it is on page 8.\n\n\n2. The authors claim computational efficiency and memory reduction, but ablation studies over flops, training, and inference time, as well as details over the computational equipment, are not stated.\n\n\n3. The authors propose several hyperparameters, even though in section 4.4 is an analysis. Further ablation studies should be conducted on the sensitivity of parameters like r and pr.\n\n\n4. Lack of training details hinders the reproducibility of the work, as it is not stated the epochs or the number of iterations used in the experiments. \n\n\n5. The update rule should be further clarified. Even though there are references to previous works, it should be explicitly stated where they come from Eq. (8)\n\n\n6. The binary mask from Equation 9 is not theoretically explained, and its purpose is not clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0Mt6HdT5zD", "reviewer": "Reviewer_hAH5", "summary": "The paper introduces Binary Error Propagation (BEP), a new training algorithm for binary neural networks that avoids surrogate gradients. Instead of real-valued backpropagation, BEP defines a discrete backward rule that propagates binary \"desired activations\" using only bitwise operations. The method updates integer hidden weights through sparse binary masks that act as an adaptive learning mechanism. BEP is evaluated on MLPs and RNNs across several datasets, showing higher accuracy than quantization-aware training (QAT) and the method proposed in Colombo et al. (2025). The authors claim that this is the first fully binary error Backpropagation (BP) algorithm capable of effectively training BNNs without relying on floating-point gradients.", "questions": "- Have you tested other binarization functions, and can you comment on how sensitive BEP is to this choice?\n- The margin parameter $r$ in Eq. (1) controls when updates are triggered. Have you analyzed how different $r$ values affect training stability or accuracy?\n- You state that the sparse mask $M_l$ plays a role similar to a learning rate. Could you provide additional analysis to support this interpretation?\n- Have you tested BEP on deeper binary MLPs? If not, could you comment on potential challenges in extending BEP to deeper networks?\n- Given that most prior work on BNNs targets convolutional architectures, can you elaborate on how BEP might extend to CNNs? \n- Did you test BEP on tasks other than classification? Might it work?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is clearly written and well organized, with notation that is consistent and easy to follow.\n- The presentation is concise and direct, and I found the mathematical derivations correct and well-grounded.\n- The method is novel and well motivated, and the experimental results are consistent with most of the claims.", "weaknesses": "- In Section 3.1, the authors obtain binary input representations using a fixed binarization method (median or thermometer encoding) but do not analyze how different binarization functions affect BEP's performance.\n\n- The margin parameter $r$ in Eq. (1) determines when binary updates are triggered, but its value and sensitivity are not analyzed. Since this parameter effectively affects the learning dynamics, the authors should provide an ablation to show how it influences performance.\n\n- The paper asserts that the sparsity mask $M_l$ fulfills the role of the learning rate in classical BP, yet this is only mentioned in the text. Providing empirical evidence would strengthen this claim.\n\n- Experiments are limited to two- and three-layer MLPs. It would be helpful to discuss whether BEP scales to deeper binary networks, as the discrete backward propagation might face stability issues analogous to vanishing gradients.\n\n- The paper should include a limitations section. While the authors briefly mention possible future directions, they should clearly articulate the current constraints of BEP.\n\nMinor comments:\n- The related work section would benefit from including recent works such as \"BiPer: Binary Neural Networks using a Periodic Function (CVPR 2024)\"\n- The small QAT (w/o batchnorm) plots embedded in each subfigure of Figure 2 are difficult to interpret. I recommend improving their presentation or visibility for better clarity.\n- In sections 4.3 and 4.4, the term \"window length\" is not clearly defined.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MKa4dUPKb1", "reviewer": "Reviewer_uRee", "summary": "This paper presents Binary Error Propagation (BEP), a new training algorithm for binary neural networks (BNNs) where both weights and activations are binary. The key innovation is a fully binary, bitwise analog of the backpropagation chain rule, allowing end-to-end propagation of binary error signals through both feedforward and recurrent architectures. The method is empirically measured against prior binary optimization methods and QAT on several tasks and model architectures, showing test accuracy improvements.", "questions": "1. Similar to Weakness 1: Can BEP be applied to deeper binary CNNs or transformer-style architectures without encountering collapse or stagnation? Have the authors attempted to apply BEP to modern vision or language models beyond the provided MLP/RNN settings?\n\n2. Batch normalization seems to contribute significantly to performance improvement. Have the authors considered adopting a binary analog or an alternative form of normalization within BEP?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The method is reasonable. And the formalization of a binary version of global credit assignment for BNNs is clear.\n2. The empirical results are broad, showing the advantage of BEP.", "weaknesses": "1. The experiments are not sufficient. Although the authors conduct experiments on both MLP and RNN models to demonstrate the effectiveness of their method, the experimental setup appears somewhat toy. Could the authors perform experiments on larger-scale networks (e.g., comparable to ResNet in size) and datasets (e.g., ImageNet-1k)? In addition, since the BNN field has been studied for a long time, could the authors compare BEP with other BNN methods (like [1-3]) beyond QAT to further validate its effectiveness?\n\n[1] Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithm\n\n[2] MeliusNet: Can Binary Neural Networks Achieve MobileNet-level Accuracy?\n\n[3] XNOR-Net++: Improved Binary Neural Networks\n\n2. Have the authors measured the actual training overhead, such as training time and memory usage, to demonstrate that BEP indeed achieves better training efficiency compared to the baseline?\n\n3. The paper format does not appear to follow the ICLR style. The pages seem wider and contain more content, which might be unfair to other submissions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)