Mind the Gap: Removing the Discretization Gap in Differentiable Logic Gate Networks
Authors: Shakir Yousefi
Andreas Plesner
Till Aczel
Roger Wattenhofer
Venue: NeurIPS 2025 poster ()
Keywords: Logic gate
logic gate network
efficient training
Gumbel noise
Primary area: deep_learning

Abstract: Modern neural networks exhibit state-of-the-art performance on many existing benchmarks, but their high computational requirements and energy usage cause researchers to explore more efficient solutions for real-world deployment.
    Differentiable logic gate networks (DLGNs) learns a large network of logic gates for efficient image classification. However, learning a network that can solve simple problems like CIFAR-10 or CIFAR-100 can take days to weeks to train. Even then, almost half of the neurons remains unused, causing a \emph{discretization gap}. This discretization gap hinders real-world deployment of DLGNs, as the performance drop between training and inference negatively impacts accuracy.
    We inject Gumbel noise with a straight-through estimator during training to significantly speed up training, improve neuron utilization, and decrease the discretization gap. 
    We theoretically show that this results from implicit Hessian regularization, which improves the convergence properties of DLGNs. We train networks $4.5 \times$ faster in wall-clock time, reduce the discretization gap by 98\%, and reduce the number of unused gates by 100\%.

Mind the Gap: Removing the Discretization Gap in Differentiable Logic Gate Networks
Shakir Yousefi ETH Zürich Switzerland syousef ieaasef@@ettinn @@ethz.ch
Andreas Plesner" ETH Zürich Switzerland aplesnerGethz.cl ch
Till Aczel" ETH Zürich Switzerland taczel@etiee ch
Roger Wattenhofer
ETH Zürich Switzerland
wattenhoofer@ete wwweeennofr@@etth.ccom ch
Abstract
Modern neural networks exhibit state-of-the-art performance on many existing benchmarks, but their high computational requirements and energy usage cause researchers to explore more efficient solutions for real-world deployment. Differ- entiable logic gate networks (DLGNss) learns large network of logic gates for efficient image classification. However, learning network that can solve simple problems like CIFAR-10 or CIFAR-100 can take days to weeks to train. Even then, almost half of the neurons remains unused, causing a discretization gap. This discretization gap hinders real-world deployment of DLGNs, the performance drop between training and inference negatively impacts accuracy. We inject Gum- bel noise with straight- striiht-through estimator during training to significantly speed up training, improve neuron utilization, and decrease the discretization gap. We theoretically show that this results from implicit Hessian regularization, which improves the convergence properties of DLGNs. We train networks 4.5x faster wall-clock time, reduce the discretization gap 98%, and reduce the number of unused gates by 100%. "Equal contribution.
Introduction
Deep neural networks have reached human-level per- formance across wide array of tasks [1] 2 However, these advances come at the cost of immense com- putational demands during training and inference, limiting their deployment in many real-world envi- ronments Kl This has sparked growing interest in designing models that retain competitive accuracy
Test Accuracy
0.4
GLGN (ours) (discrete)
0.2
GLGN (ours) (soft)
while being more efficient (711).
DL.N (discrete)
DL.G (soft)
At their core, all computations on digital hardware reduce to Boolean operations such AND, OR, and NOT. This motivates the question: Can we express and execute machine learning models directly in the native language of hardware namelyy logic gates?
500
1000
Iteration (-103)
Figure 1: CIFAR-10 test accuracy compar- ison. Solid and dashed lines show discrete and soft performance, respectively. Gumbel L.NNs (GLGN, red) demonstrate faster con- vergence and minimal discretization gap pared to DLGNs (DLGN, blue).
One approach is logic gate networks (LGNs), which replaces arithmetic computations with compositions of discrete logical operations, thereby enabling effi- cient inference. While inference with LGNs is effi- cient, training them poses significant challenges. To address this, differentiable logic gate networks (DL-
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Gumbel LGNs (ours)
Differentiable LGNs
(a)
(b)
Gumbel(O,
DaA Bicesto
Snow Leopard
Cell
(020 (00t))
Cap SX.
Detata (4)
(c)
(d)
Figure 2: Overview figure. (a) DLGNs: Leftmost shows the internal structure of a node. We parameterize each node by weighing the possible logic gates (shown below) and summing their output. This results in hard, brittle, loss landscape, slowing convergence and increasing the discretization gap. (b) Gumbel LGNs: during training, we inject Gumbel noise on the gate weights and select the logic gate with the highest weight. This results in smoother loss landscape and aligns the training with the network at inference. This results in faster training and reduces the discretization gap. (c) Structure of DLGN, and Gumbel LGNs. Each neuron receives two inputs. The nodes in the final layer are aggregated by summation, thus producing class likelihoods. (d) Results of using Gumbel LGNs instead of DLGNs. We achieve up to 4.5x faster convergence (in wall-clock time), 98% reduction discretization gap, and 100% reduction of unused neurons.
GNs) 7 9 introduce continuous relaxations of both logical operations and gate selections, allowing the of gradient-based optimization methods.
Despite their potential, we identify and propose solutions to two major challenges. (1) Discretization gap: DLGNs rely continuous relaxations during training, but final models must be discretized for inference. This mismatch may result in notable degradation, reducing model accuracy by 3% on the same data. (2) Slow convergence: While LGNs efficient at inference time, training DLGNs remains slow due to the reliance on differentiable relaxations, making DLGNs converge substantially slower than standard neural networks.
These challenges are interrelated. The gap arises because the final parameters, after training, must be discretized. Small parameter perturbations can significantly change performance if the loss landscape sharp [2]. A sharp loss landscape can also cause poor gradient signals, which impact the convergence speed, causing training to take much longer, while smooth loss landscape reduce the discretization gap and speed up convergence 13H16. Our central hypothesis that smoother loss landscapes make DLGN models more robust to discretization and facilitate faster and more stable training. Since the loss landscape is smoother, the gradient signal better, thus making the networks converge faster. At the same time, the improved gradient signal also causes more neurons to collapse, thus reducing the impact of discretization.
We propose Gumbel Logic Gate Networks (Gumbel LGNs), which inject Gumbel noise into the gate selection process using the Gumbel-Softmax trick. By injecting Gumbel noise into the gate selection process, introduce stochasticity that flattens the loss landscape, improving the optimization

---

process and reducing discretization sensitivity. Empirically, we find that Gumbel LGNs exhibit faster convergence and smaller discretization gaps than baseline DLGNs.
To further reduce the discretization gap, we explore a training strategy inspired by techniques from discretization-awar training and neural architecture search (NAS): using continuous relaxations only in the backward pass, while enforcing discrete gates in the forward pass. Although ST estimator may slightly slow convergence, substantially reduces the discretization gap. Moreover, it aligns training dynamics with inference-time behavior, and as it only influences training, there is no impact on inference.
To our knowledge, no prior work analyzes the discretization gap in DLGNs, and draw formal connections to the smoothness of the loss landscape. NAS and DARTS typically studies explored search spaces up to size 1018, while our experiments operate over vastly larger parameter spaces, exceeding 103,600,000, and show that NAS techniques scale this setting.
Our contributions are as follows:
Empirical validation. We design and execute experiments showing that Gumbel LGNs both speed up training and boost neuron utilization.
Theoretical analysis. We prove that injecting Gumbel noise into DLGNs smooths their loss landscape by regularizing the Hessian's 's trace, thereby reducing the discretization gap and accelerating convergence.
Practical algorithmic insight. We demonstrate that employing the straigh--through estima- tor further closes the discretization gap in practice.
2 Background
Logic Gate Networks Logic Gate Networks (LGNs) represent an entire network as a composition of discrete logic operations, such as AND, OR, and XOR. LGNs consist of several layers, where each layer contains n fixed logic gates, referred to as neurons, that work in parallel. n is the width of the layer. Each neuron in layer takes input the output of two (random) neurons in the previous layer. We use the GroupSum operation to get class scores as done by Petersen et al. 77 For classifying the input into & classes, the neurons in the final layer are grouped into disjoint groups with one group per class. The outputs of the neurons in group i are summed to give the likelihood of class i. Mathematically this is expressed as 8; र EjeG, aj. where G; is the i'th group, aj is the output of neuron j. and GG is a temperature parameter. We then predict the class with the highest score. For more details, see [7]]
Differentiable Training Directly searching for the best discrete gate assignments is infeasible due to the size of the search space, so Differentiable Logic Gate Networks (DLGNs) 7 introduce a continuous relaxation. Each of the 16 possible binary gates h; b) is replaced by continuous surrogate (e.g. AND(a, + a b). In addition, each neuron maintains logits E IR which are initialized using a Gaussian, distribution over gates, and the neuron's output is weighted sum of the 16 gates:
(O, 1)26. After softmax, these logits define probability
16 exp
fooft (a.
h:(a, 6).
exp
Ensuring each logic gate maps from continuous domain f 10, 1]2 + 10, 1]. This "soft" network can be trained end-to-end with gradient descent.
Discretization After training, DI.GNs discretized to LGNs by selecting the logic gate with the highest logit value, i.e., uses h4 = arg max, 2,. We denote DL.GN; evaluated in the differentiable setting (using Equation (T)) as soft and otherwise as discrete.
The Clumbel-Softmaal trick offers efficient and effective way to draw samples from categorical distribution with class probabilities æ E & [8-21]. Let ~ Gumbel(O, 1) distribution if (00 1) and g - log( u). We can then draw a sample from as the
Gumbel-Softmax
3

---

value for index i given by Equation 2
arg max + (ogz))) 9j ~ Gumbel(O, 1).
We can make the argmax operation continuous and differentiable with respect to the class probabilities Ty and generate k-dimensional sample vectors y R using the Softmax with temperature 7 as below:
Gumbel
exp((log + g:)/r) exp((logg + (
T. Σ
exp 2j
R.
(3)
9;)/7)²
Σ exp2y
Related Work
Efficient Neural Architectures A significant body of research has focused on designing neural models that maintain high performance while operating within limited computational budgets, e.g.. for deployment on edge devices (72 These light models use various methods such as lookup tables [28,
binary and quantized neural networks
and sparse neural networks 33 37
Of particular interest in this context are differentiable logic gate networks (DLGNs), which have demonstrated state-of-the-ar performance in image classification tasks 7 [9. well as in rule extraction from observed cellular automata dynamics [17]. Our proposed improvements target convergence behavior and are orthogonal the architectural innovations of the convolutional DLGN variant [9; hence, we expect them to be transferable without loss of generality. We refrain from comparisons to other efficient neural models, as such benchmarks were already comprehensively addressed in the works mentioned above by Petersen et al. 97.
Kim (38 extend DLGNs by modifying the neuron parameterization introduced in Petersen al. 77 Specifically, they normalize the logits and remove the temperature parameter т. In addition, they employ ttraight-through (ST) estimator [39) and injectGumbel noise with learnable scale parameter. However, Kim [38] do not analyze the discretization gap or study the convergence behavior of DLGNs. Moreover, they draw no formal connection between Gumbel noise and its role as an implicit Hessian regularizer or loss smoothener.
Differentiable Neural Architecture Search Neural Architecture Search (NAS) aims automate the selection of high- perforref model architectures from a large design space [4]. While early approaches were computationally expensive, subsequent efforts have focused improving Several works have addressed the issue train-test performance discrepancies by proposing sampling-based training 45 regularization techniques that bias architecture selection
efficiency 43 44
toward configurations with better generalization [46].
A seminal contribution in this domain is Differentiable Architecture Search (DARTS) by Liu et al. relaxation over discrete architectural choices, allowing end-
[47, which introduces a softmax- to-end optimization through gradient descent. This principle strongly resonates with the soft gate selection mechanism employed in DLGNs.
More recently. Chen and Hsich [66 reduced the discretization gap of DARTS by introducing Smooth DARTS, which uses weight perturbations through uniform noise or adversarial optimization. These were shown to bias the optimization toward solutions with flatter minima and lower Hessian norm. This technique often referred to as curvature regularization reduces sensitivity to sharp local optima and enhances generalization.
Differentiable LGNS as DARTS The works on DLGNs by Petersen al. 7 do explicitly draw connections to NAS, but the conceptual similarity is high. Both LGNs and DARTS use softmax- based weighting to choose between multiple candidate functions differentiable manner. A key distinction lies in the scale of the search space. Conventional NAS approaches typically explore search space sizes up to 10 48 50), while LGNs operate over exponentially larger spaces-16 166-61,0000 as 10262.382 for MNIST and s 103,699.08 696,05666666 for CIFAR-10. This scale is enabled by the simplicity of logic operations, which have no learnable parameters. Thus, DL GNs demonstrate the viability of DARTS at previously unexplored scales.

---

Conventional NAS frameworks often permit retraining phase after discretizing the architecture, thereby reducing the discretization gap. LGNs, in contrast, lack such flexibility, as their neurons contain no parameterized operations, and thus the gap persists.
Moreover, in DARTS and related approaches, this training approach favors operations, such residual connections [51]. Typically. we aim to avoid these residual connections, as they do not increase the models' expressive power [46]. In relation, Petersen et al. g finds that their convolutional method with residual initialization mainly converges to residual connections.
Sharpness-Aware Minimization A parallel line of research focuses on improving generalization by minimizing the sharpness of the loss landscape. Motivated by prior theoretical works on generalization and flat minima [1 52 53 Foret et [15] introduced Sharpness- Aware Minimization (SAM) in Equation 4 This technique explicitly seeks flat minima by optimizing the worst-case loss within perturbation neighborhood.
SAM (w) + A1/2112 where Lg (w - max Ls(w + t), min LS
SAM
(4)
1eppSp
where (s(w) is loss function over training set S of training samples evaluated for model parameters w. p E [o| is the p-norm used (usually p = 2) and p > 0 is a hyperparameter [1]]. Since its introduction, SAM has inspired numerous follow-up studies focused improving computational efficiency [54 59] as well as providing theoretical insights into its efficacy [60 63
We refer to Appendix D for detailed description of SAM.
4 Gumbel Logic Gate Networks
We introduce Gumbel Logic Gate Networks (Gumbel LGNs), which employ discrete sampling of logic gates via the Sambel-Softtax trick 20 21 with straight-through (ST) estimator (39]. While conventional DLGNs maintain a convex combination of gates throughout training and prune to hard selections at inference time, Gumbel LGNs resemble inference-time behavior directly during training by stochastically selecting individual gates per forward pass. We perturb the gate logits with Gumbel noise and seleet the most probable gate, following the argmax operation (cf. Equation (2). During backpropagation, the non-differentiable argmax is approximated using the Gumbel-Softuaal (Equation (3), enabling end-to-end training.
This approach is motivated by two observations: (1) Implicit smoothening via noise: injecting Gumbel noise during the forward pass introduces a form of stochastic smoothing, effectively averaging over local perturbations of the loss surface. As we show, this process approximates a curvature- curvatureepemnaiiing loss that favors flatter minima and smaller Hessian norm. addition, this is known to correlate with improved generalization [15 [6]. (2) Inference-time alignment: In DLGNs, the training objective misaligned with inference behavior, as training relies on weighted combinations of gates that are ultimately discarded. This discrepancy harms generalization. In contrast, Gumbel LGNs train under the same discrete selection mechanism, which is used at inference.
Training Gumbel L.NSs
As with DLGNS, model each neuron distribution over binary, relaxed logic gates S = {41, hy,...,, haa...,,hs,, where each gate h; : [0, 1|2 - [0, 1] operates on relaxed Boolean inputs. We associate each gate i with logit 2,, and the gate has weight ग. Gumbell from Equation (3) The output of the neuron with inputs (a, b) is then given below.
16 Σ
exp((log n; + 9:)TT) h:(a, b) Σ, exp((log (logg, + 9;)/T)
16
faolt (a, b)
Σ Gumbel he(a, b),
(5)
where 0 is temperature parameter controlling the sharpness of the distribution. As T 0. the distribution increasingly peaks around the maximum-login index (20). This relaxation enables end-to-end differentiability while encouraging the network commit discrete logie gates during training.
We employ straight-through (ST) estimator to bridge the discretization gap between the continuous relaxation used during training and the hard decisions required during inference, this formulation, each neuron selects single logic gate in the forward pass via hard (non-differentiaall) choice.
5

---

while gradients are estimated through soft relaxation in the backward pass. See Appendix for pseudo-code implementation of the training process. Coneretely, during the forward pass, we sample Gumbel noise g ~ Gumbee(O,, 1) l and compute:
Jz priserete (a, b) = h2 (a, 6)
(6)
using the gate hy with maximum perturbed logit. During the backward pass, we the soft Gumbel- Softmax relaxation (Fquation (5) to compute gradients, effectively treating the hard output as if were differentiable:
0 fdiscrate
ffzoft
dz
0z.
This ST estimator mechanism encourages the network to make discrete decisions and allows end-to- end optimization via backpropagation. See Figure 2 or Figure 11 in Appendix A for visualizations.
Implicit Gap Reduction via Gumbel Smoothing. We present theoretical result that supports the use of Gumbel perturbations during training. Consider a loss function L, logits E 16 and with i.i entries Øi Gumbel(0, 1). Adding Gumbel noise with T IR to the logits can be seen as Monte-Carlo sample of the objective (z);
((z) E [C[softmax((z +
(g)))))))))))))))))))))( g)/r)]
This can be interpreted as a form of stochastic smoothing. This gives us the following lemma:
Lemma 1 Lipschitz Hessian), and let E IR g Gumbel(0, 1)16. Consider
Let 16 + twice continuously differentiable (with
(z) = E [C(softmax((2 + g)/T]]
and set z/T and f(a) [(softmax(a)) We then get the expression below.
n²
J(z) = (s)ftmax(z//7) - 12r2
tr O0
Proof. See Appendix B
Intuitively, by injecting Gumbel noise during training, we encourage the optimizer to find parameters that are robust small perturbations. This results in flatter loss landscapes and reduces the sensitivity to parameter discretization when switching to inference mode [2] The expected loss scales with
2² Ir (r(Hz(Z/T)).
12,2
As the temperature 7 increases, the coefficient 1/7² decreases, reducing the degree of implicit smoothing. We illustrate this with two representative choices:
Small (e.g. 0.1) = 1/7² is large = large smoothing, flat minima.
Largg 7 (e.g. 2.0) = 1/72 small almost no smoothing.
As result, adjusting the temperature offers mechanism to control the strength of this eurvature- aware regularization, the convergence of the model, and to reduce the discretization gap implicitly.
Empirical Evaluations
Our empirical evaluations focus on CIFAR-10 and CIFAR-100. Due to constrained resources, we limit experiments by default to 48 GPU hours. To ensure fair comparison, we use the hyperparameters from [7] whenever possible rather than tuning the parameters, such as learning rate, ourselves. Appendix H.T contains all the default parameters.

---

Discretization Gap
GILGNN Trpth
BLG Depth
Test Accuracy
0.10
GLLCI:: Depet 10
Dept
DLANN
EILGNN Depeh 10
0.4
DLGN Depth
(ouns) (discrete)
GL
DLGN (discrose)
DI KIN (soft)
0.00
0.0
200
400
600
800
1000
1200
200
400
600
800
1000
1200
Iteration (-10²)
Iteration (- (-10²)
Figure 3: Performance of Gumbel LGNs and DLGNs on CIFAR-10. (Left) Test accuracy (width of 256k, depth of 12). (Right) Discretization gap for various depths. DLGNs experience larger gaps and slower reduction as the depth increases. In contrast, Gumbel LGNS have consistently low gaps and fast reduction as the network depth increases.
DLGN (discrete)
GL.N (oura (disceete) (ours) (soft)
Discretization Gup
Deyt
Test Accuracy
0.3
(soft)
0.04
G GieeeDpphh
GUGN Dayh 10
TX ET Teghi 8
BLGN Depth
GLG Doph
0.2
0.02
0.1
0.00
0.0
200
400
600
800
1000
1200
1000
1200
Iteration (-10²)
Iteration (-10")
Figure 4: Performance of Gumbel LGNs and DLGNs on CIFAR- 100. (Left) Test accuracy (width of 256k, depth of 12). (Right) Discretization gap for various depths. DLGNs experience larger gaps and slower reduction as the depth increases. In contrast, Gumbel LGNs have consistently low gaps and fast reduction the network depth increases.
Discretization Gap Figure 3 shows the test accuracy as a function training iteration for an LGN of depth 12 and width 256k CIFAR- 10; these are the default parameters unless stated otherwise. To quantify the discretization gap. we take the absolute difference between the discrctized and soft network accuracy shown the right in Figure 3 Gumbel LGN converges much faster than the DL.NN, with virtually no discretization gap. We also measure the runtime and see that Gumbel increases the runtime by roughly 5% per iteration (cf. Appendix K.. Combined with runtime results from Table 4 in Appendix K Gumbel LGN converges 4.75x faster in iterations, making Gumbel LGNs 4.5x faster in wall-clock time train. Note that the Differentiable LGN still improves after 48 hours.
We perform additional evaluation CIFAR-100 using the same settings. On Figure 4 that GLGNs already converge at around 400K iterations, whereas DLGNs do not converge. This is consistent with the results, observe for CIFAR-10.
Gap Scaling with Depth On the right of Figure various depths for DLGNs and Gumbel LGNs. As the model depth increases, the expressive power of the networks theoretically increases. The DLGNs experience bigger discretization gaps as the depth increases, while our Gumbel LGNs are stable across depths. Hence, Gumbel LGNs do not struggle to converge as the networks are made deeper. Both methods experience shift in when the maximum gap occurs. This is expected since increasing the model size usually delays when the accuracy plateaus, i.e., more iterations are needed converge.
the discretization gap for models of
Shallow Network We fix the depth of the network at and increase the width to 2048K. On Figure that. Interestingly, the accuracy of the soft DLGN converges within 50K iterations. However, the discretized version does not seem to improve with continued training. We note that the
For this, match Gumbel LGNs' discrete accuracy with DLGNs maximum discrete accuracy.
7

---

Test Accurucy
0.4
0.2
(diserete)
(ours) (diserete)
DLON (sof))
GLGN (soft)
0.0
Iteration (-10")
Figure 5: Test accuracy on CIFAR- 10 for a shallow, wide network (width 2048k, depth 6).
Discretization Cap
0.20
Test Accuracy
Gumbel +
0.4
Gumbel ST (discrete)
0.15
Gumbel Soft
Gumbel ST (soft)
0.10
0.2
Gumbel Soft (discrete)
0.05
Gumbel Solt (soft)
0.00
200
400
600
800
1000
1200
0
200
400
600
800
1000
1200
Iteration (-10²)
Iteration (-10²)
Figure 6: Straight-through (ST) estimator ablation (width of 256k, depth 12). The Gumbel LGNs uses hard gate choices in the forward pass shown in (2) called the ST estimator. On the left, we show the test accuracy over training iterations; on the right, we show the discretization gap. Gumbel LGNs with ST estimator converge slightly slower test accuracy, but the discretization gap is smaller.
final accuracy of GLGNs is higher than DLGNs. This suggests increasing the number of neurons through the width also contributes discretization gap.
Straight-Through Estimator Ablation To better understand the source of gap reduction achieved by the Gumbel-Softmax trick, we perform an ablation isolate the contribution of the ST estimator. As discussed previously, Gumbel-Softtax combines (i) ST estimation as seen in (2) and (ii) implicit smoothing via Gumbel noise. By disabling the ST path, we aim to identify whether gap reduction primarily stems from the discrete gradient approximation or the added stochasticity. The setup without ST corresponds to DLGNs with noisy logits and is denoted as Soft Gumbel.
In Figure 6 we see that imputing Gumbel noise alone impacts both convergence and discretization compared to DLGNs. However, we observe that including the ST estimator delays convergence for fixed r, but further reduces the discretization gap.
Curvature, Hessian, and Smoothness As we showed in Section 4 loss minimization for Gumbel LGNs implicitly reduces the curvature of minima by minimizing the trace of the Hessian. The impact of the trace compared the loss depends on the temperature parameter т. Thus, we perform an ablation study on the effect of r, estimate the trace of the Hessian, and visualize the loss landscape.
Ablation over 7-parameter We evaluate how varying the temperature 7 affects optimization dynamics. Recall that higher reduces the degree of implicit smoothing, potentially leading to sharper minima and slower convergence. our experiment, we test [0.01, 2.0). We show the results in Table [ and fig. 7 where we observe a goldilocks zone for the temperature; if T is large (> 1) or small (< 0.1), then the network converges much slower. However, higher temperatures such as seem to converge to slightly better solutions. Although the difference is minor, < 0.5% when T 0.25 goes to T 1.
Hessian Trace Approximation The Hessian scales quadratically with model size, so direct compu- tations are infeasible for our networks with millions of parameters. Still, we can use iterative methods to approximate the trace, etc. (64-68). We approximate the trace using Hutchinson's 's method with

---

Test Accuracy
0.1
(r(H)
-10
0.15
0.575
0.550
-20
220
r- 0.5
1200
1300
== 1.0
200
400
600
800
1000
250
500
750
1000
1250
Iteration (-10²)
Iteration (-10²)
Figure 7: Ablation over the temperature r for Gumbel LGNs. Left)) Estimated Hessian trace using Hutchinson's s method. The trace shrinks decreases, indicating fewer large positive eigenvalues, thus suggesting a flatter loss surface, which may reduce the risk of loss increases when discretizing parameters. Right) Test accuracy for the r-values. We as if is large (> 1) or small (< 0.1), then the network converges much slower. In the zoomed-in view, we plot the non-discretized view as dashed lines and see that these similar to the discretized values, i.e., the discretization gap is low for all. Except for = 0.01 and r = 2, there is only a small variation in the final accuracy, as seen in Table
goldilocks zone for the temperature,
Table 1: Maximum and final test accuracy for the tested values. The iterations column indicates the number of training iterations (-109) required to be within 1% of the maximum accuracy. We see that with medium value of 0.25 the network converges much faster than for high values 1.
0.01
0.05
0.10
0.15
0.20
0.25
0.50
1.00
2.00
Max accuracy
0.547
0.566
0.574
0.574
0.566
0.573
0.573
0.578
0.490
Final accuracy
0.546
0.564
0.570
0.571
0.563
0.568
0.572
0.575
0.480
Iterations (-10%)
972
602
632
518
472
440
530
918
1342
200 Rademacher random vectors, where each coordinate is independently sampled from {-1, +1} with equal probability [67]. Full experimental details are provided in Appendix [F]
As shown in Figure 7 decreasing the r-parameter reduces the estimated Hessian trace. This aligns with the theoretical insights from Section 4 lower T values place greater weight on trace reduction. The trace is negative, this is expected: stochastic gradient noise in overparameterized networks tends to systematically lower the expected Hessian trace, biasing solutions toward flatter regions of the loss landscape that may have negative trace values [19) Large negative eigenvalues often vanish, the trace remains influenced by many small eigenvalues, resulting in a negative overall trace.
Curvature Visualization To qualitatively assess the loss landscape curvature, we project the high-dimensional parameter space onto two-dimensional subspaces. Following Li et al. [72). we select random directions and interpolate the loss surface along these axes, providing insight into the optimization landscape's 's geometry around learned solutions. Visualization details are in Appendix Figure shows that Gumbel LGN has a visually smoother loss surface.
Entropy over Logic Gates Petersen et al. noted that neurons collapse single gates, but they did not investigate the extent of the collapse. We examine this through neuron entropy by sampling 100k newly initialized neurons and computing the 95% interval. We also estimate expected entropy theoretically (see Appendix[], giving us baseline distribution for neurons that have not learned. As neurons collapse, their entropy converges to O. Figure 9 shows that many early Differentiable LGN layers do not collapse, while Gumbel LGN neurons converge with entropies near 0, Defining unused gates those with entropy above the 2.5% -e%ceeeeee threshold, Gumbel LGNs have 0.00% and DLGNs have 49,8 % unused gates, representing 100.00% reduction by Gumbel.
6 Limitations
While Gumbel LGNs demonstrate significant improvements for deeper networks, limitations do remain. Our evaluation focuses primarily on CIFAR-10 and CIFAR- 100, with limited exploration of more complex datasets. The temperature parameter requires tuning balance convergence speed

---

Differentiable LGN
Gumbel LGN
Loss
Loss landscape of a Differentiable LGN. We see that the landscape is overall noisy. Right: Loss landscape of Gumbel LGN with T 1.0. We observe a much smoother loss landscape compared to the DLGNs.
Figure 8: Visualization of loss landscapes. Left:
Differentiable LGN
Gumbel LGN
10a
Tayer
Layer 2
10
Layer
Layer
Density
10a
Layer 5
Layer
10-1
Layer
Layer 8
Layer
10
Layer 10
Layer
10-3
Layer
2
2
Exported Entropy
Entropy
Entropy
Figure Entropy distribution for neurons in each layer for trained DLGNs and Gumbel LGNs models. The dashed black line indicates the expected entropy (ef. Appendix (T] of neurons before training, and the shaded region the 95% interval computed by sampling. We see that almost all the neurons in Gumbel LGNs have converged, while neurons in early layers of DLGNs still have high entropy.
and accuracy. Finally, our theoretical analysis connects Gumbel noise Hessian trace minimization under simplifications, but comprehensive theoretical treatment of the discretization gap remains open challenge. Moreover, comprehensive analysis of width scaling and the interplay between width and depth is left for future work.
7 Conclusion
We introduced Gumbel Logic Gate Networks (Gumbel L.Nss,, addressing two critical limitations of DI.NNs: slow convergence during training and large discretization gap. Our theoretical analysis shows that Gumbel noise during gate selection promotes flatter minima by implicitly minimizing the Hessian trace, reducing sensitivity to parameter discretization. Experiments on CIFAR-100 and CIFAR-I0 demonstrate that Gumbel LGNs converge up to 4.5 faster wall-clock time than DLGNs while reducing the discretization gap by 98% and achieving 100.0% improvement in neuron utilization. These advantages become more pronounced with depth, indicating favorable scaling properties. Our improvements dataset and architecture architecture-indeeedden independent, and several promising directions remain for future exploration, e.g. adaptive temperature scheduling.
10

---

References
[1] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. In 2015 IEEE International Conference on Computer Vision (ICCV), pages 1026-1034, December 2015. doi: 10.1109/ ICCV.2015.123.
[2] Andreas Plesner, Tobias Vontobel, and Roger Wattenhofer. Breaking recaptchav2. In 2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC), pages 1047-1056. IEEE, 2024.
[3] Song Han, Huizi Mao, and William J. Dally, Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding, February 2016.
[4] Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and Policy Considerations for Modern Deep Learning Research. Proceedings of the AAAI Conference on Artificial Intelligence, April 2020. ISSN 2374-3468. doi: 10. v v34009.441.3 7123.
[5] Neil C. Thompson, Kristjan Greenewald, Keeheon Lee, and Gabriel F, Manso. The Computa- tional Limits of Deep Learning, July 2022.
[6] Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang. Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang. Pradeep Dasigi, and Hannaneh Hajishirzi. Tulu 3: Pushing Frontiers in Open Language Model Post- Bost-Training, April 2025. URL nttp://arxiv .rg. abs/ 2411 15124 4xXiv:241..15124 [cs].
[7] Felix Petersen, Christian Borgelt, Hilde Kuehne, and Oliver Deussen. Deep Differentiable Logic Gate Networks. Advances in Neural Information Processing Systems, October 2022.
[8] Zachary Susskind, Aman Arora, Igor D. S. Miranda, Luis A. Q. Villon, Rafael F. Katopodis, Leandro S. de Araújo. Diego L. C. Dutra, Priscila M. V. Lima, Felipe M. G. França, Mauricio Breternitz, and Lizy K. John. Weightless Neural Networks for Efficient Edge Inference. In Proceedings of the International Conference on Parallel Architectures and Compilation Techniques, PACT '22, pages 279-290, New York, NY, USA, January 2023. Association for doi: 10.1 10.1144/3550009.35560 33569600...66......
Computing Machinery. ISBN
[9]
Felix Petersen, Hilde Kuehne, Christian Borgelt, Julian Welzel, and Stefano Ermon. Convo- lutional differentiable logic gate networks. In A. Globerson, L. Mackey, Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 121 121182-22 Curran Associates, Inc., 2024.
[10] Marta Andronic and George A. Constantinides. NeuraL.UT-AAsemblle Hardware- Awaree As- sembling of Sub-Neural Networks for Efficient LUT Inference. In 2025 IEEE 33rd Annual International Symposium Field- Peeedd-rogrummable Custom Computing Machines (FCCM), pages 208-216, May 2025. doi: 10. (1000///CCC2223.22...000
[11] Alan T. L. Bacellar, Zachary Susskind, Mauricio Breternitz Jr. Eugene John, Lizy K. John, Priscila M. V. Lima, and Felipe M. França. Differentiable Weightless Neural Networks, March 2025.
[12] Jacob Mitchell Springer, Sachin Goyal, Kaiyue Wen, Tanishq Kumar, Xiang Yue, Sadhika Malladi, Graham Neubig, and Aditi Raghunathan. Overtrained Language Models Are Harder to Fine-Tune, March 2025.
[13] Sepp Hochreiter and Jürgen Schmidhuber. Flat Minima. Neural Computation, 9(1):1-42, January 1997. ISSN 0899-7667. doi: 10. (9t./eec...999.999
[14] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On Large-Batch Batch Training for Deep Learning: Generalization Gap and Sharp Minima. In International Conference on Learning Representattonss February 2017.
11

---

[15] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. SharpnessAAwar Minimization for Efficiently Improving Generalization, April 2021.
[16] Xiangning Chen and Cho-Jui Hsieh. Stabilizing Differentiable Architecture Search via Perturbation-base Regularization, January 2021.
[17] Pietro Miotti, Eyvind Niklasson, Ettore Randazzo, and Alexander Mordvintsev. Differentiable Logic CA: From Game of Life Pattern Generation. https//ggoole--ressahhggiihhhhh iowsef. March 2025.
[18) E. J. Gumbel. Statistical Theory of Extreme Values and Some Practical Applications. Journal of the Royal Statistical Society. Series A (General), 118(1):106, 1955. ISSN 00359238. doi:
10.2307/2342529.
[19] Chris J. Maddison, Daniel Tarlow, and Tom Minka. A Sampling. In Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
[20] Eric Jang, Shixiang Gu, and Ben Poole. Categorical Reparameterization with Gumbel-Softmax. In International Conference on Learning Representations, February 2017.
[21] Chris Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous re- laxation of discrete random variables. In International Conference on Learning Representations (ICLR), 2017.
[22] Donghyun Park, Sculgi Kim, Yelin An, and Jac- Yoon Jung. LiReD: A Light-Weight Real-Time Fault Detection System for Edge Computing Using LSTM Recurrent Neural Networks. Sensors, 18(7):2110, June 2018 ISSN 1424-8220. doi: 10.3390/s18072110.
[23] Shiya Liu, Dong Sam Ha, Fangyang Shen, and Yang Yi. Efficient neural networks for edge devices. Computers Electrical Engineering, 92:107121, June 2021. ISSN 0045-7906. doi:
00.1016f.compeecccng200000 2021.107121.
[24] Mohammad Al-Quraan, Lina Mohjazi, Lina Bariah, Anthony Centeno, Ahmed Zoha, Kamran Arshad, Khaled Assaleh, Sami Muhaidat, Mérouane Debbah, and Muhammad Ali Imran. Edge- Native Intelligence for 6G Communications Driven by Federated Learning: A Survey of Trends and Challenges. IEEE Transactions on Emerging Topics in Computational Intelligence, 7(3): 957-979, June 2023. ISSN 2471-285X doi: 10. 109/TETC1.2023.222
[25] Rahul Mishra and Hari Prabhat Gupta. Designing and Training of Lightweight Neural Networks on Edge Devices Using Early Halting in Knowledge Distillation. IEEE Transactions on Mobile Computing, 236744666-4467 May 2024. ISSN 1558-0660. doi: 10. 10/00MM.22223.3990006
[26] Yusen Zhang, Yunchuan Qin, Yufeng Zhang, Xu Zhou, Songlei Jian, Yusong Tan, and Kenli Li. OnceNAS: Discovering efficient on-device inference neural networks for edge devices. Information Sciences, 669: 120667, May 2024. ISSN 00200255. doi: 10.10166j.ins... itt.:///iis.224.... 120567.
[27] Shahzaib Iqbal, Tariq M. Khan, Syed S. Naqvi, Asim Naveed, Muhammad Usman, Ha- roon Ahmed Khan, and Imran Razzak. LDMRes-Net: A Lightweight Neural Network for Efficient Medical Image Segmentation on 1oT and Edge Devices. IEEE Journal of Biomedical and Health Informatccs, 28(7):3860-388 3871,, July 2024. ISSN 2168-2208. doi: 10.1 10.1109//BHL. 00.1109/BHH1
[28] Satrajit Chatterjee. Learning and Memorization. In Proceedings of the 35th International Conference on Machine Learning, pages 755-763. PMLR, July 2018
[29] Yichi Zhang, Junhao Pan. Xinheng Liu, Hongzheng Chen. Deming Chen, and Zhiru Zhang. FracBNN: Accurate and FPGA-Efficien Binary Neural Networks with Fractional Activations. In The 2021 ACM/SIGDA International Symposium on Field-Programmadn Gate Arrays FPGA '21, pages 171-182, New York, NY, USA, February 2021. Association for Computing Machinery. ISBN 978-1-4503-82188 doi:
[30] Yunqiang Li, Silvia-Laura Pintea, and Jan C Van Gemert. Equal Bits: Enforcing Equally Dis- tributed Binary Network Weights. Proceedings of the AAAI Conference on Artificial Intelligence, 36(2):1491-14 1499, June 2022. ISSN 2374-3468, 2159-5399,
doi: 10.1609/aau 166.9/22....33...00033...
12

---

[31] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq; Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17325 2022.
[32] Chunyu Yuan and Sos S. Agaian. A comprehensive review of Binary Neural Network. Artificial Intelligence Review, 56( I 01112999-10013 November 2023. ISSN 0269-2821, 1573-7462. doi: 10.10077/10462-0033- 10464... 140460000066002-2044-
[33] Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research, 22(241):1-124, 2021.
[34] Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. approach for large language models. arXiv preprini arXiv:2306. 11995, 2023.
simple and effective pruning
[35] Yang He and Lingao Xiao. Structured pruning for deep convolutional neural networks: A survey, IEEE transactions pattern analysis and machine intelligence, 46(5):2900- 2919, 2023.
[36] Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pages (0323-10337, PMLR, 2023.
[37] Hongrong Cheng, Miao Zhang, and Javen Qinfeng Shi. A survey on deep neural network pruning:
Taxonomy, comparison, analysis, and recommendations. IEEE Transactions Pattern Analysis and Machine Intelligence, 2024.
[38] Youngsung Kim. Deep Stochastic Logic Gate Networks. IEEE Access, 11:122488-122501, 2023. ISSN 2169-3536. doi: 10.1 /TT/////CCCSS2223.33222222
[39] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation, August 2013.
[40] Barret Zoph and Quoc Le. Neural Architecture Search with Reinforcement Learning. In International Conference Learning Representations, February 2017.
[41] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. Journal of Machine Learning Research, 20(55):1-2) 2019.
[42] Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-yao Huang, Zhihui Li, Xiaojiang Chen, and Xin Wang. A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions. ACM Comput. Surv., 54(4):76:1-7633, May 2021. ISSN 0360-0300. doi: 10.1145/3447582.
[43] Xuanyi Dong and Yi Yang. Searching for a Robust Neural Architecture in Four GPU Hours. In 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 1761-1770, Long Beach, CA, USA, June 2019. IEEE, ISBN 978-1-7281-3293-8. doi: 10.1109/ CVPR.2019.00186.
[44] Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. SNAS: Stochastic Neural Architecture Search, April 2020.
[45] Jianlong Chang. xinbang zhang, Yiwen Guo. GAOFENG MENG, SHIMING XIANG, and Chun hong Pan. DATA: Differentiable ArchiTecture approximation. In H. Wallach, H. Larochelle, A. Beygelzimer, dAlché-Buc Buc, Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[46] Xiangxiang Chu, Tianbao Zhou, Bo Zhang, and Jixiang Li. Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search, July 2020.
[47] Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable Architecture Search, April 2019.
[48] Arber Zela, Julien Siems, and Frank Hutter. MAS-Bench-1Shotl) Benchmarking and Dissecting One-shot Neural Architecture Search. In International Conference on Learning Representations, September 2019.
13

---

[49] Renbo Tu, Nicholas Roberts, Misha Khodak, Junhong Shen, Frederic Sala, and Ameet Talwalkar. MAAS-Bench-366 Benchmarking Neural Architecture Search on Diverse Tasks. Advances in Neural Information Processing Systems, 35:12380-12394 December 2022.
[50] Krishna Teja Chitty- Venkata, Murali Emani, Venkatram Vishwanath, and Arun K. Somani. Neural Architecture Search Benchmarks: Insights and Survey. IEEE Access, 11:25217-25233 2023. ISSN 2169-3536. doi: 10.1 20B.:/CCCESS.223.3555555
[51] Yunjie Tian, Chang Liu, Lingxi Xie, Jianbin jiao, and Qixiang Ye. Discretization-aware architecture search. Pattern Recognition, 120:108186, December 2021. ISSN 0031-3203. doi: 10. wt.1016/j.aatcog222000 10000...1111001 10110088166 186.
[52] Gintare Karolina Dziugaite and Daniel M. Roy. Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data, October 2017.
[53] Yiding Jiang*, Behnam Neyshabur*, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic Generalization Measures and Where Find Them. In International Conference on Learning Representations, September 2019.
[54] Yong Liu, Siqi Mai, Xiangning Chen, Cho-Jui Hsieh, and Yang You. Towards Efficient and Scalable Sharpness-Aware Minimization. In 2022 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 12350- 12360, New Orleans, LA, USA, June 2022. IEEE. ISBN 978-1 1688--6654-6946-3,
doi: 10. P0.15099 2022.01204.
[55] Jiawei Du, Hanshu Yan, Jiashi Feng, Joey Tianyi Zhou, Liangli Zhen, Rick Siow Mong Goh, and Vincent Y. F. Tan. Efficient Sharpness-aware Minimization for Improved Training of Neural Networks, May 2022.
[56] Jiawei Du, Daquan Zhou, Jiashi Feng, Vincent Tan, and Joey Tianyi Zhou. Sharpness-aware training for free. In S. Koyejo, Mohamed, A. Agarwal, D. Belgrave, K. Cho, and Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages Curran Associates, Inc., 2022.
[57] Peng Mi, Li Shen, Tianhe Ren, Yiyi Zhou, Xiaoshuai Sun, Rongrong Ji, and Dacheng Tao. Make sharpness-aware minimization stronger: A sparsified perturbation approach. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 30950-30962. Curran Associates, Inc., 2022.
[58] Yong Liu, Siqi Mai, Minhao Cheng, Xiangning Chen, Cho-Jui Hsieh, and Yang You. Random sharpness-await minimization. In S. Koyejo, S. Mohamed, Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 24543-24556. Curran Associates, Inc., 2022.
[59] Maximilian Mueller, Tiffany Vlaar, David Rolnick, and Matthias Hein. Normalization Layers Are All That Charpness-Aware Minimization Needs, November 2023.
[60] Maksym Andriushchenko and Nicolas Flammarion. Towards Understanding Sharpness-Aware Minimization, June 2022.
[61] Pengfei Wang. Zhaoxiang Zhang, Zhen Lei, and Lei Zhang. Sharpness-Awan Gradient Match- ing for Domain Generalization. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3769-3778, Vancouver, BC, Canada, June 2023. IEEE. ISBN 979-8-3503 ----0300------... doi: 10.11099VVPS5222992200 2000522772.222.003367
[62] Kaiyue Wen, Tengyu Ma, and Zhiyuan Li. How Does Sharpness-Awar Minimization Minimize Sharpness?, January 2023.
[63] Tao Li, Pan Zhou, Zhengbao He, Xinwen Cheng, and Xiaolin Huang. Friendly Sharpness-Aware Minimization. In 2024 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5631-5640, Seattle, WA, USA, June 2024. IEFE, ISBN 979-8-3503-5300-6 doi: 10. 101100/CCPS3223..22240000
14

---

[64] R. V. Mises and H. Pollaczek Pollaczek-Geiringer Geiringer. Praktische Verfahren der Gleichungsauffösungg Section
1-4. ZAMM- Journal of Applied Mathematics and Mechanics / Zeitschrift für Angewandte Math- ematik und Mechanik, 9(1):88-777 17758,77,70 1929. ISSN 1521-4001. doi: 10. 10.100222amm. 199000000..9999909000000000
[65] R. V. Mises and H.
Geiringer. Praktische Verfahren der Cleichungsaufföuung Section 5-6. ZAMM Journal of Applied Mathematics and Mechanics Zeitschrift für Angewandte Mathematik und Mechanik, 9(2): ((2):152-166 164, 1929. ISSN 1521-4001. 19290090206.
doi: 10.1002/zamm.
[66] Ilse CF Ipsen. Computing an eigenvector with inverse iteration. SIAM review, 39(2):554-291, 1997.
[67] Michael Hutchinson, A stochastic estimator of the trace of the influence matrix for Laplacian smoothing splines. Communications in PaatiticssSimullttio and Computation, 18(3):1059- 1076, 1989.
[68] Lloyd N Trefethen and David Bau. Numerical Linear Algebra. SIAM, 2022.
[69] Levent Sagun, Leon Bottou, and Yann LeCun. Eigenvalues of the hessian in deep learning: Singularity and beyond. arXiv preprint arXiv:1611.07476 2016.
[70] Levent Sagun, Utku Evci, V Ugur Guney, Yann Dauphin, and Leon Bottou. Empirical analysis of the hessian of over-parametrized neural networks. arXiv preprint 1.Xiv:170.0044544 2017.
[71] Mingwei Wei and David J Schwab. How noise affects the hessian spectrum in overparameterizee neural networks. arXiv preprint arXiv:1910.000 2019.
[72] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018 URL https neurips. pdff]
2018/file/441bUU htt.:///.......30000000000070060
[73] David A. McAllester. PAC-Bayesian model averaging. In Proceedings of the Twelfth Annual Conference on Computational Learning Theory, pages 164-170, Santa Cruz California USA, July 1999. ACM. ISBN 978-1-58113--67777 doi: 10. 145/337400.307433.
[74] Barak A. Pearlmutter. Fast exact multiplication by the hessian. Neural Computation, 6(1): 147-160, 1994.
[75] Haim Avron and Sivan Toledo. Randomized algorithms for estimating the trace of an implicit symmetric positive semi-definite matrix. Journal of the ACM (JACM), 58(2):1-34 2011.
[76] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the Loss Landscape of Neural Nets. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018
[77] L. Bottou, C. Cortes, J.S. Denker, H. Drucker, I. Guyon, L.D. Jackel, Y. LeCun, U.A. Muller, E. Sackinger, P. Simard, and V. Vapnik. Comparison of classifier methods: A case study in handwritten digit recognition. In Proceedings of the /2th IAPR International Conference on Pattern Recognition, Vol. 3- Conference C: Signal Processing (Cat. No. No.94CH3340-5), volume pages 77-82 vol.2, October 1994. doi:
[78] Y. LeCun, I.. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278- 22242 November 1998. ISSN 1 1558-2256. doi: 10.1 109/5.726791
[79] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images, 2009.
[80] Gregory Cohen, Saeed Afshar, Jonathan Tapson, and André van Schaik. EMNIST: An extension of MNIST to handwritten letters, March 2017.
[81] Han Xiao, Kashif Rasul, and Roland Vollgraf. MAsh:: A Novel Image Dataset for Benchmarking Machine Learning Algorithms, September 2017.
15

---

[82] Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep Learning for Classical Japanese Literature, November 2018.
[83] Chhavi Yadav and Léon Bottou. Cold Case: The Lost MNIST Digits, November 2019.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
17

---

Justification:
Proof in Appendix B
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See, for instance, Appendices G and H.1
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification: CIFAR-10 is an open-source dataset and we provide the codebase.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://niis.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/ddeeeeee Cooeeeoodeesssseeeeee //nips.com / cassiicnncccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
See Appendix H.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
We were not able rerun the experiments under multiple seeds due compute constraints.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
19

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://aaarag.cccpa/a/////aaaaaa
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
We conduct foundational research.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: To the best of our knowledge, there are no risks posed by the release of our code.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We only used open-sourced models and data.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.paaersstt
package should be provided. For popular datasets, paperswi thcode. cow/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide instructions for the codebase in the zip file.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
22

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
23

---

A
Logic Gate Network Visualizations
Snow Leopard
0
1
0
1
1
1
1
0
Cat
1
Figure 10: A diagram showing a standard logic gate network (LGN). Each logic gate receives two inputs from the previous layer. The image first binarized before being passed into the network, and the output neurons are grouped, and each neuron in group votes whether the image belongs to the group/class.
Snow sopand
1
0
1
1
1
1
0
1
1
0
Cat
0
1
y Cat(p)
Figure 11: Forward pass through a Gumbel LGN. The top panel shows neurons producing class scores. Bottom-left: categorical distribution Cat(p) over relaxed logic gates, parameterized by learnable weights z E 16 Bottom-right (zoom-in): internal view of one neuron. The signal passes through single selected relaxed logic gate (colors indicate which gate is chosen).
B Gumbel Smoothing
The proof of Lemma depends on the translation invariance of the softmax.
Lemma 2 -nniattiion-nnaaiannneeee of Softmax), Consider logits E Rd,, then adding any constant e E IR to z, z + e, does not change the output of the softmax. Concretely, for any logit zı we have
softmax 2; + c) = softmax(z;)
Proof. Denote of ;= 2, + then writing the output of our softmax yields

---

el
softmax(2) (2))
Σ, E 0%
e
Eg e
:
el ee.ef
e E
e Σ, .t=:
=oftmax(22) (2))
Σ, et
Restating Lemma] !
Lemma 3 Goumbel-Smoothing] Let L 16 + R be twice continuously differentiable (with
Lipschitz Hessian), and let z E 16 g (0mbel(O, 1)16. Consider (zz)
((z = E [C(softmax( g)/r)]
and set z/T and f(a) L (sofmmax(a)) then
x²
((z = [(softmax(z/T)) + 1272 tr(H)(a)) + O
Proof. Rewriting (zz in terms of gives us
J(z) = E { (a a +
Consider second-order Taylor expansion of f around
f - f(a) V
T g 1 g
(+)
Hy(a) (e) + 011el1//7)
+
T 2
7
Taking expectations, and recalling that E Masch-Mascherooi constant. we get
[g:] Var(g:) = =²/66, where ~ 0.57721 is the
2² (H))(())) + + 6
J(z) = f(a) +
1
221 H Ha 1
27
which follows from E Eig, Var(g:) + [[g]] = 2//6 + E.g 22 for i # j.
1 11 + x2
E E[gg
=
I
6
and the following trace-lemma
(a(g) = (a) TI) 21 Hy (a)1 1 + =²
tr())(()))
Since the softmax is vanslation-invariant in its input a, we have vf(a) and (a) so all terms depending on drop, finally giving us
?² tr (H
(zz) L(softmax(z/7)) 12.2
Hence, minimizing our stochastic loss implicitly smoothens the curvature by minimizing the trace of the Hessian.
25

---

Differentiable LGN
Gumbel LGN
0
0.20
Class Index
0.15
3
7/4 = B
D)
A
AVB
-(A
A B
((AB
AAA
-(4 mD R
-(4 D)
ASB
A
(Av
(A&
A B
A B
(AA.B)
-M4
Figure 13: Gate distribution for each class in the CIFAR-10 classifiers.
The gates are from the last
layer before the groupsum is applied.
C Distribution over Logic Gates
logic gates in the final Gate Distribution by Layer We look Figure 12 at the distribution network. Our first observation is that the distributions layers 1 to 11 than for DLGNs. At the same time, we see a sharp transition for DLGNs after layer This could match the results in Figure 9 indicating that neurons in DLGNs struggle to converge in all but the final layers.
far more uniform for Gumbel LGNs in
The "1" gate can be seen as a bias towards specific classes. Notably, Gumbel LGN primarily uses this gate type the last layer, so we analyze this further.
Differentiable LGN
Gumbel LGN
Layer Index
0.05
1A => 8]
-A4 = D)
AvB
-)
-8
-[A88]
AAB
1W +- 8)
AvB
(. -Av
A B
A-B B
Figure 12: Gate distribution for the gates split by layer for Differentiable LGN and a Gumbel LGN. Interestingly, the distribution far more uniform for the Gumbel LGN in layers 11 than for the Differentiable LGN. In addition, the Gumbel LGN primarily has "1" gates in the final layer, which can be seen as a constant bias towards certain classes. We analyze this further Appendix C
Gate Distribution by Class We see in Figure 13 the gate distribution for each 10 classes in the last layer. Interestingly, the distributions between Softmax and Gumbel are quite different, but the classes are nearly identical. Since almost all classes have several "I" gates, pruning these would be possible as softmax is translation invariant.
26

---

D Extended Sharpness-Aware Minimization
A parallel line of research focuses improving generalization by minimizing the sharpness of the loss landscape. Motivated by prior theoretical works on generalization and flat minima [4] (3),, Foret et al. [1]] introduced Sharpness-Awan flat minima by optimizing the worst-case loss within a perturbation neighborhood. Let Ls (w) be loss function over a training set S of samples from distribution 9 evaluated for model parameters w.
Minimization (SAM). This technique explicitly seeks
min L.S
T.A.M (w) - max Ls( + +), T.SAM (w) + 21 where T.S
where p E [1, oo is the p-norm used (usually p = 2) and p > 0 hyperparameter [15]. This optimization objective arises from the PAC-Bayesian generalization bound shown in Equation (
[3]. and Foret et al. [13] use it to show Equation (9). Below, n = [S), A is the number of model parameters, 9 and 2 are the prior and posterior distributions of the model parameters, respectively.
Lastly, the equations hold with probability
KL 2 9) + log A
Ew 24, SEw 2 [[s(w))+
V
(8)
2(n - 1)
) 1
2
k: log
1
|| 3 p²
1+
log(n) k
+ 4log " Ö(1)
(w(w)= < max Ls(w + c)+
(9)
lel
Since its introduction, SAM has inspired numerous follow-up studies focused on improving computa- tional efficiency [54 59 as well as providing theoretical insights into its efficacy 66
E
Training GLGNS
Algorithm Training STE-GLGNs
while not converged do
Sample Gumbel noise g ~ Gumbel(0,) 1)
Compute soft sample a' softmax (log + g//
Compute hard sample à = one_hot (arg max a²)
Forward pass through logic gate network using á (with stop_grad(a - a') + d') Compute loss L
Backpropagate and update a and other parameters end while
Estimating Hessian of Loss
Computing the full Hessian of the loss function is not computationally feasible due to the quadratic scaling with the number of parameters. Instead, we use scalable stochastic methods to estimate the trace of the Hessian, which provides useful curvature information. Specifically, focus on the trace as our Lemma directly minimizes the trace of the Hessian.
Hessian- Vesttor Products In order to estimate the trace, we will be using Hessian-vector products on the form Hv, where H 2 [()) is the Hessian of the loss L with respect model parameters 0, and v Rd arbitrary vector. While explicitly forming H would require (dd2) memory and computation, such products can be computed efficiently using reverse-mode automatic differentiation (also known Pearlmutter's 's trick) in ((d) time and memory (74.
Given a scalar-valued function [(0), the Hessian-vector product He defined as:
d
He 2 C())
VC Ev)
de This formulation allows for efficient computation using automatic differentiation frameworks, without explicitly constructing the Hessian matrix.
27

---

Trace Estimation via Hutchinson's Method To estimate the trace of the Hessian, we employ Hutchinson's stochastic trace estimator 66. which approximates the trace of matrix as
tr(
H
M
where each E Rd is random vector with zero- cero-mean, uari-variance, i.id. entries. The choice of distribution for 2,'s 's affects the variance of our estimator. While both Gaussian and Rademacher distributions satisfy this, Rademacher vectors (each entry sampled from {-1, +1} with equal proba- bility) lead to lower-variance estimator. This is formally shown in [75]. This makes especially well-suited for estimating curvature efficiently in high-dimensional models.
We use m 200 Rademacher vectors to produce a stable estimate. Each Hessian vector product Ha, is computed efficiently using reverse-mode automatic differentiation without explicitly forming Hessian matrix.
-eestor
Choice of Evaluation Points Since both trace and eigenvalue estimators are noisy and computa- tionally expensive, we evaluate them only at selected points during training, Specifically, we choose points that correspond to monotonically increasing accuracy on the test set.
G Loss Surface Visualization
Visualizing the geometry of the loss landscape provides insights into the optimization dynamics during training and the discretization gap. We follow the same procedure in Li. et al 2018 (76), which constructs a two-dimensional slice of the high-dimensional loss surface by perturbing model weights in random directions.
Methodology Let 0 E Rd be a parameter vector of a trained model. The goal is to evaluate the loss L(0") over grid of points
(' 3) = = + ad, + Ad2
where di and di are orthogonal directions with 3 E R. We choose o, 3 such that we probe the model in unit circle, i.e. 1-
Direction Sampling The direction vectors dj and d2 are generated follows: Each direction is drawn using a Gaussian distribution d; ~ (0, 1)d. We then normalize d, d//l|d.]l. such that each perturbation has the same overall scale.
Orthogonalization To ensure that d1, d2 span meaningful plane, we apply Gram-Schmidt orthog- onalization:
d1 + dg (d1, d2) This ensures that span independent, meaningful axes for visualization. This allwos to visualize the curvature in three-dimensions.
d1
(d1, di)
H Experimental Configuration
H.1 Hyperparameters
Gap Scaling with Depth we use the same hyperparameters for this experiment for DLGNs and Gumbel LGNs. Specifically. we fix the width of the network to 256K neurons and train the network over the depths {6, 8, 10, 12). We optimize the network using Adam with learning rate of 0.01. Furthermore, the batch size is set to 128 and the final parameter in the GroupSum is set to 1 1/0.01. This mirrors the original experimental setup Petersen al. 7 for CIFAR-10. Finally, we fix the
1 parameter for the Gumbel noise in the Gumbel LGNs.
28

---

A. Default CIFAR-10 Training
Optimizer
Adam
Learning rate
0.01
Batch size
128
Depth
12 (unless varied)
Width
256 neurons
GroupSum scale
1/0.01
B. Gap-Scaling (Depth Ablation)
Depths tested
{6. 8, 10, 12) 256k
Width
Other settings
same as (A)
C. Ablatio Studies
Straight-Througe vs Soft
12pth=12, width=256k, tau=1.0
Temperature sweep
tau (0.01,0.05,4) 111,,,06...1.111500225555...0.10
D. Hessian Estimation
Trace estimator
Hutchinson, m=200 Rademacher vectors
Top-eigenvalue estimator
Power iteration, 200 iterations
Evaluation points
checkpoints monotonic test accuracy
Table 2: All hyperparameter settings, grouped by experiment.
Ablation Studies For both the straight-through and 7-parameter ablations, the deepest model from the gap scaling experiment (depth 12, width 256K) to evaluate each effect in a stress-tested regime. This allows to isolate the effect of either variant.
H.2 Implementation
Our code extends the official PyTorch Difflogic library by Felix Petersen, i.e., the refer- ence implementation provided alongside the Differentiable LGN paper [7]. During the forward pass we replace the standard torch. mn. functional. laaiionanlssotaa... softmax with hard Gumbel-Softmaa torch. nn. functionn ionll www.eiiinalggubell......e wwggggelussfftmmme thereby enabling discrete sampling while maintain- ing end-to-end differentiability.
I Softmax DiffLogie Performance on MNIST-like Baselines
For all datasets, use the same model and experiment configs The models have six layers and width of 64k. See for the results.
Besides the classic MNIST [77 78), CIFAR-10, and CIFAR- 100 datasets 79]. also evaluate EMNIST (balanced and letters) 88), FashionMNiST 81). KMNIST 82). and QMNIST [3). These black and white images, in MNIST, but with other more classes. We refer the reader to the original papers for details and examples.
Thuss the CIFAR numbers not representative of the optimal performance.
29

---

Table 3: The table shows the performance of DLGNs on many datasets. The key takeaway that the discretization gap for MNIST-like datasets is minimal. The discretization gap is the difference between the discrete and soft performance. The numbers are averaged over five runs.
Train
Test
Accuracy
Accuracy
Dataset
Soft
Discrete
Disc. gap
Soft
Discrete
Disc. gap
CIFAR-10
79
100.0
100.0 %
0.0
52.04 %
50.72
1.31 %
100
79
83.44 %
80.39 %
3.05 %
23.86 %
23.1 %
0.76 %
EMNIST balanced
95.52
94.87 %
0.65
84.57
84.28
0.29 %
EMNIST letters 80
98.69 %
98.3 %
0.38 %
91.43 %
91.04 %
0.38 %
FashionMNIST
99.02
98.17 %
0.85
90.37 %
90.0
0.36 %
KMNIST 82
100.0 %
100.0 %
0.0 %
97.14 %
97.0 %
0.14 %
MNIST 777
100.0
100.0 %
0.0
98.33 %
98.16
0.17 %
QMNIST 83
100.0 %
100.0 %
0.0 %
98.33 %
98.17 %
0.16 %
J Expected Entropy
Lemma 4. The expected entropy newly initialized neuron in a Differentiable LGN is ss log 16 RJ 2.27.
Proof. A neuron has n 16 gates that it makes a choice over and gate has (i.i.d.) logit 2; N(0, 1) and probability p. exp where C Σー= exp 2y. For the rest of the proof, we assume the number gates n is not fixed, and show that the expected entropy converges log 2.
The expected entropy of p = (p1,D2...) is
cxp 22
exp 2j
B(H(p) = E
log
C
2i exp
= E logg Cl - E
C
1=1
27 exp 23
= E
log
exp Zj
2ド=1 exp 2j
1=1
i=1
Here, we have as n + 00:
exexx
1 E -i=1 exp
E[2; exp zi] Elexp 2y]
=
j=1 exp 2j
exp Zj
Using Elexp 21] ee and El(2 exp 21] = Ve, the above gives us that as n -> oo:
n
24 expxxx 24
E
log
exp 2j
===1 exp 2j
1
E[2, exp 24]
log n + E
log
exp Zz
E(exp 2y]
7
B 22 exp zi]
1
log + log E
Elexp zy]
log 2
1 log
E Elexp 2y]
We only need to plug in 16 to get the last part.
30

---

Table Iterations per hour and the relative change from DLGNs Gumbel LGNs.
Iterations per hour Softmax
Gumbel
Change
Depth 6
38708
38375
-0.86%
Depth
36292
34000
-6.31%
Depth
32792
31167
-4.96%
Depth 12
29417
27583
-6.23%
Mean
-4.59%
K Runtime
In Table 4 the number of iterations per hour Softmax and Gumbel LGNs completed while training models for the results in Figure 3 We also calculate the relative difference between the two. Gumbel slightly slower due to the noise sampling; however, as it converges much faster terms of iterations, the net effect is that it converges 4.5 times faster in wall-clock time.
L Computational Resources
The experiments were done on an internal cluster with RTX 3090s and RTX 2080 Tis. In total, we have logged 1284 GPU hours for the experiments and testing. significant part of the compute was spent on exploration.
31


Reviews:
[{"review_id": "ahv2YUxU6k", "reviewer": "Reviewer_n2V7", "summary": "This paper introduces Gumbel Logic Gate Networks (Gumbel LGNs) to solve two major issues with existing Differentiable Logic Gate Networks (DLGNs): slow training convergence and a significant \"discretization gap\" where performance drops after training. By injecting Gumbel noise during the training process, the authors create a smoother optimization landscape. This new method trains 4.5 times faster, reduces the discretization gap by 98%, and eliminates unused logic gates.", "questions": "(1) What is the estimated energy efficiency like? \\\n(2) In addition to the energy efficiency on edge devices, how much does this work advance the practical availability of LGN?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths:**\\\n(1) The research area of logic gate networks deserves more attention. Analogous to human brain using neurons as basic components, using logic gates as basic components in digital artificial neural networks is a very reasonable research direction. \\\n(2) The motivation of this work is clear. Hard, brittle loss landscape in existing DLGNs and the inefficiency of neuron usage in inference are significant problems to solve. \\\n(3) The results demonstrate the effectiveness of Gumble LGN, by being 4.5 times faster than existing DLGNs. \\\n\\\n**Weaknesses:**\\\n(1) The techniques proposed should be more closely bridged to their advantages in practical tasks. This work may have greater impact if some comparisons with standard neural networks can be provided to demonstrate the advantages of Gumble LGN. For instance, with discretization gap and unused neurons minimized, what is the estimated energy efficiency like compared to baselines and traditional NN approaches? Since energy efficiency is pivotal in edge device deployment.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and effort in helping to improve the paper. We will address the weaknesses and questions raised in the review.\n\n**Advantages of GLGNs over NNs.**\n\nIn both DLGNs and GLGNs, all gates remain in the final LGN architecture and are active during inference. Thus, the benefits of logic-gate-based computation (e.g., energy efficiency) apply equally to LGNs induced by either method.\nWe refer to estimates from Petersen et al. [2], where a CLGN deployed on a Xilinx XC7Z045 FPGA performs inference on CIFAR‑10 in 9 ns per image achieving 60.38% test accuracy. Assuming a peak power consumption of 15 W, this corresponds to approximately 135 nJ per image. Although the CLGN architecture differs from our models, it shares the same neuron parameterization as DLGNs, so we expect a Gumbel-parameterized CLGN to have comparable inference-time energy characteristics.\n\n**On Advancing the Practical Availability of LGNs**\n\nAs the reviewer notes, faster DLGN training directly advances the practical availability of LGNs. Our experiments also show that the discretization gap scales with architecture size and the number of logic gates. For more complex tasks, where larger and more expressive networks are needed, Gumbel LGNs makes scaling LGNs more feasible with less prohibitive training costs."}, {"review_id": "jCpyzu4G3v", "reviewer": "Reviewer_zLS1", "summary": "This work applies some ideas from other fields to Differentiable Logic Gate Networks (DLGNs), significantly improving their training speed and performance.\nUsually, the training of DLGNs involves keeping a distribution over the possible logical gates and employing the associated weighted sum of the gates' outputs as the output of the node.\nThe core idea of this work is to inject Gumbel noise into the logit values underlying the distribution over the gates and to select the output of the gate with the highest value as the output of the node (no weighted sum). In the backward pass, the non-differentiable \"winner-takes-all\" operation is approximated using a softmax function.\nThe authors provide some theoretical justification for the approach and provide experiments that indicate striking improvements over DLGNs.", "questions": "In Lemma 1, if we were to replace the $16$ by a general $d \\in \\mathbb{N}$, what would the thesis look like?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 2, "strengths_and_weaknesses": "This work advances the field of DLGNs significantly.\nThe technique is simple and clean and the results seem impressive.\nIn fact, I believe the authors could better advertise the improvement in robustness to depth increase as this is an important aspect of their contribution but seems only to be mentioned around line 223.\nPresentation is sufficiently clear, and the paper is well-structured with thoughtful guidance to abstracts when suitable.\nThe theoretical results, though not particularly deep, are a nice addition and provide some justification for the approach.\nThe rationale behind the experiments is well communicated and the results are clearly presented.\n\nMoving to the weaknesses, the main one seems to be the limited scope of the experiments.\nI would go harder on this if the authors had not stated it so clearly as a limitation (the honesty is appreciated).\nI also understand that those experiments are quite expensive to run.\nStill, the lack of error bars is a major issue, as it makes it hard to assess the significance of the results, which could kill the paper if the improvements were not so pronounced.\n\nMoreover, the claim that unused gates are reduced by 100% can give the wrong impression.\nPersonally, I did not expect to find much later in the text (around line 273) that this claim is based on defining \"unused gates\" as those with entropy above the 2.5%-percentile of the entropy distribution.\nThis threshold appears somewhat arbitrary; if there is a principled reason for choosing this rule, the authors should clarify and justify it in the main text.\nOtherwise, the claim should be reconsidered or reworded to avoid potential misunderstanding.\n\nFinally, the use of a fixed dimensionality (16) in Lemma 1 is unusual.\nI do understand that this happens to be the number of gates considered in the implementation of LGNs, but I fail to see a reason to limit generality.\nUnderstanding how the result varies with the number of gates adds information on its own.\n\nBelow I list some minor points that could help the authors improve the work:\n- Although not bad, the introduction of concepts is not particularly good.\nI feel like I only got it without much effort because I already had contact with the subject.\nFor instance, the quick overview of LGNs provided can be improved significantly without taking much more space, if any.\nA good example of how to achieve this is already in the references of the paper: [25].\nSimply adding an explicit recommendation to that source for an overview of LGNs would already be a meaningful improvement.\n- The mathematical typesetting looks a bit amateurish.\nTypos are not uncommon and the use of language, which is quite solid elsewhere, becomes clumsy around mathematical expressions.\nFor some examples, almost all colons in the text are used incorrectly, the statement of Lemma 1 reads a bit odd, and Algorithm 1 is hard to parse since it lacks a proper description of the inputs and their domains (variables need types!).\n- More references should be provided.\nNot that Yoshua Bengio needs the credit for the straight-through estimator, but it's good practice nonetheless.\nThere are many other examples of claims that could (and should) be easily backed by references.\n- The third equality in the proof of Lemma 2 shouldn't be there.\n\n\nOverall, I liked the work and I believe it's a clear accept.", "was_revised": true, "final_justification": "My understanding is that the decisive point here is the novelty relative to *Kim (2023)*.\nI maintain my emphatic recommendation of acceptance because:\n1. I'm confident the authors can and will fix the blunder of missing *Kim (2023)* in the related works.\n2. The results improve the SoTA massively, especially in one of the most notorious and crucial shortcomings of DLGN: their very long convergence times (and they do that while improving accuracy).\n3. Relative to *Kim (2023)*:\n   - The authors put the reduction of \"discretization gap\" at the centre of their work, analysing it extensively and *quantitatively*, while *Kim (2023)* only hints at it *qualitatively*.\n   - *Kim (2023)* does not study the convergence times, so the massive improvements detected in this work, if present in *Kim (2023)*, would remain unnoticed, limiting the impact of *Kim (2023)* (in this specific regard, that's a good paper).\n   - The authors provide their source code, while *Kim (2023)* does not, further amplifying the reproducibility and impact of the work.\n\nI intentionally didn't mention the differences between the methods since **even if the technique proposed was exactly the same as that from Kim (2023),** the points above make me confident that the paper would still be a significant contribution and should absolutely be published.\nPut differently, even if this paper were a mere \"We investigated *Kim (2023)* from a different angle and found out that it works extremely well\", it would still be a great contribution with a meaningful impact in the community.\nThis is where I diverge from Reviewer FVvS as they mention that \"it would be better to not only mention it in the related works but also benchmark against it to show the difference in performance, if any\"; I agree that it would be better, but I don't think the this work has no place without this reframe (despite of this divergence, I do appreciate the reviewer's inputs, which are quite constructive).\n\nFor those reasons, I, honestly, do not understand Reviewer iYnB's position, which doesn't appear to acknowledge any significance in this contribution.\nThe reviewer brings many valuable points that would surely improve the paper, but when it comes to motivating the rejection of the work, I don't think the criteria are reasonable.\nIn fact, the reviewer's opposition is so strong that it gives the impression that they believe not only that the paper should not be accepted for NeurIPS, but that no version of it should be accepted anywhere, given the intersection with *Kim (2023)*.\nI truly believe that the contribution is so significant that, even in this case, a preprint version of the paper would likely garner many citations, as its insights would still establish a new reference point for the community.", "rebuttal": "We thank the reviewer for their time and effort in providing valuable feedback to improve the paper. \n\n**Claim on the number of unused gates**\n\nWe agree that the terminology “unused gates” can be easily misinterpreted, as the neurons/gates are still processing the input during training and post-discretization. We use the 2.5% as a heuristic based on a standard two-sided 95% confidence, i.e. neurons whose entropy remain high and statistically similar to initialization. We will clarify this in the revised draft. Perhaps a term like “dead” or “inactive” neuron would better convey this idea.\n\n**Extension of Lemma 1 to general $d \\in \\mathbb{N}$**\n\nLemma 1 holds for arbitrary $d \\in \\mathbb{N}$. Indeed, it would also apply to the general Gumbel-Softmax setting beyond DLGNs. We agree that stating the lemma in its full generality makes more sense - we will change this in the draft.\n\n**Introduction to LGNs**\n\nWe thank the reviewer for pointing this out. We agree that the current introduction could be made more accessible, especially for readers unfamiliar with LGNs. In response, we will revise the introductory paragraph to clarify the LGN setup and improve readability without increasing length. Additionally, we will explicitly reference [25] as a recommended resource for readers seeking a more in-depth overview of LGNs. We appreciate the suggestion, as it meaningfully improves the paper’s approachability.\n\n**Mathematical typesetting** \n\nWe will carefully revise the mathematical typesetting, addressing the noted issues (colons, Lemma 1 phrasing, and Algorithm 1’s input descriptions). We will ensure all variables are properly typed and all mathematical expressions are clearly presented in the revised draft.\n\n**On References and Citations**\n\nWe thank the reviewer for this suggestion. We agree that citing foundational works (e.g., Bengio et al. for the straight‑through estimator, Adam, and other standard methods) will make the paper more polished and complete. We will add these references in the revised draft.\n\n\n**Lemma 2 typo**\n\nWe apologise for this mistake and will correct it.\n\nOnce again, we would like to thank the reviewer for a thorough and thoughtful review of our paper!"}, {"review_id": "G9mHeWp8rV", "reviewer": "Reviewer_iYnB", "summary": "This paper addresses the discretization gap between training and testing of differentiable logic gate networks by adding Gumbel noise to the softmax function during logic gate selection training. The perturbation-based regularization smooths the loss landscape, resulting in more stable gradient signals and faster convergence. By using the Gumbel-Softmax trick, the selection of logic gates can be effectively synchronized between the training and inference phases, thereby reducing the discretization gap.", "questions": "1. To isolate the effect of perturbations on the loss function, it is necessary to observe LGNs based on L(softmax(z/τ)) across different τ values. In Lemma 1, τ (tau) is not completely separated, so effect of tau seems mixed in the final loss. In figure 6, can different tau values  show different loss landscapes?\n2. As shown in Figure 4, despite a larger discretization gap, faster convergence of test accuracy is observed. Does this result match with the statement in lines 49–50 (\"Since the loss landscape is smoother, the gradient signal is better, thus making the networks converge faster.\")? Is the choice of backward method (or gradient propagation) more critical for convergence speed than the discretization gap itself? Also, it needs to clarify what Gumbel Soft (discrete) and Gumbel Soft (soft) are.\n3. Connecting to the above point, is the reduction of the discretization gap from the use of Straight Through Estimation (STE) or Gumbel Noise (which causes flat curvature)? In the Abstract: \"We inject Gumbel noise with a straight-through estimator during training to significantly speed up training,… and decrease the discretization gap.\" This is ambiguous.  \n4. Although smaller τ (temperature) leads to faster convergence, the best maximum and final test accuracy is achieved when τ=1 (Table 1). Do these rapid training convergences sacrifice test accuracy due to the perturbations, or could there be another interpretation? With  regularization, does worse generalization performance makes sense? Is it only contributing to faster convergence toward a local optimum?", "limitations": "As noted by the authors, the models should be tested over more datasets.", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "- **Strengths**:\n    - The authors provide a theoretical analysis showing that Gumbel perturbations during training act as a form of Hessian trace regularization, derived using a second-order Taylor expansion of the Gumbel-Softmax function, somewhat inspired by [3,4]. Furthermore, the paper presents a comparison of the loss landscapes between LGN and GLGN.\n- **Weaknesses:**\n    1. Gumbel-Softmax-based differentiable logic gate networks have been explored in *Deep Stochastic LGN*, where the Gumbel-Softmax formulation is further modified.\n    2. In this paper, only a single model is tested (depth: 12, width: 256k), whereas the LGN paper evaluates multiple models with varying depths and widths. It would be more convincing to include comparisons with the model configurations used in Petersen et al.'s LGN paper.\n    3. The main contribution of this paper is Lemma 1 (Gumbel-Smoothing), which derives a trace of the Hessian term in the expectation of the Gumbel softmax function. The smoothness of the softmax-based class probability function (Eq. 4) depends on the trace of the Hessian term with a tau (temperature) related weight (in front of the Hessian trace) and also a temperature (tau) inserted into the softmax function. However, the temperature term in the softmax function is neither specified nor explored (in Figure 3).\n\n\n*Deep Stochastic LGN*:\nY. Kim, \"Deep Stochastic Logic Gate Networks,\" in IEEE Access, vol. 11, pp. 122488-122501, 2023,", "was_revised": true, "final_justification": "This paper analyzes the effect of Gumbel-softmax for categorical sampling in logic gate networks, which has already been explored in existing work (Kim, 2023). The difference is that it shows discretization gap analysis and convergence speed along temperature values $\\tau$ in the Gumbel-softmax. As the loss function $\\mathcal{L}(\\text{softmax}(z/\\tau))$ in the objective function in Lemma 1 contains $\\tau$, the effect of $\\tau$ is not completely isolated. Therefore, the effect of $\\tau$ appears mixed in the shape of the final loss and convergence speed, even though the authors mainly explain it through Gumbel noise. The authors have failed to explain this clearly. Consequently, their experimental results are misleading and their conclusions are incomplete.\n\nBeyond the Gumbel-softmax setup and STE framework and their motivation in existing work, Hessian-based regularization using the Gumbel-Softmax function has already been explored in \"Stabilizing Differentiable Architecture Search via Perturbation-Based Regularization\", which further challenges the novelty of this paper's approach.\n\nThus, I maintain my initial assessment.", "rebuttal": "We thank the reviewer for their time and effort in helping to improve the paper. We address the reviewer’s concerns below:\n\n\n**Clarifying Model Evaluations**\n\nWe would like to clarify that our experiments already evaluate multiple configurations, not just a single model. Specifically, we sweep across four depths (6, 8, 10, 12) (Figure 3) and nine $\\tau$-values (0.01–2.0) (Figure 5) at a fixed width of 256k. These experiments form the basis for our analysis of depth scaling and $\\tau$’s role in loss smoothing and the discretization gap.\nWe will make this more explicit in the revised draft (Section “Gap Scaling with Depth”) to avoid any ambiguity.\n\n\n**Clarifying Terminology (DLGN vs. LGN)**\n\nSince our paper centers on the discretization gap, the distinction between Differentiable Logic Gate Networks (Differentiable LGNs, DLGNs), used during training, and fully discretized Logic Gate Networks (LGNs), used during inference, is critical. Some comments (e.g., regarding Lemma 1) appear to reference LGNs when DLGNs were intended. We will make this distinction clearer in the paper and correct mistakes.\n\n**Clarifying Novelty and Relation to Deep Stochastic LGN: Y. Kim, \"Deep Stochastic Logic Gate Networks\"**\n\nWe thank the reviewer for pointing out Y. Kim, \"Deep Stochastic Logic Gate Networks,\" in IEEE Access, vol. 11, pp. 122488-122501, 2023. We will add it to the related papers for completeness and highlight the differences to our paper. While both papers use variations of Gumbel-Softmax, our contributions differ substantially. Our work focuses on analyzing the discretization gap and its scaling behavior, framing DLGNs as a Neural Architecture Search problem. In particular, Kim (2023) does not study the gap or its scaling, nor does it analyze convergence speed or neuron utilization, all central to our theoretical and empirical results.\nBeyond scope, Kim (2023) adopts a different Gumbel-Softmax formulation and neuron parameterization that make its results incompatible with our analysis:\n1) it omits the $\\tau$-parameter (central to Lemma 1 and Fig. 5),\n2) normalizes logits (as shown in Kim (2023) this increases entropy over the induced distribution, which would worsen the discretization gap), and\n3) introduces a learnable Gumbel scaling parameter that alters the Hessian-trace term. These choices place Kim (2023) in a different training regime, so our theoretical and empirical results do not apply to their setup.\n\n**On Gumbel Noise and Regularization**\n\nKim (2023) suggests Gumbel noise prevents overfitting, drawing on Bishop’s result that training with noise approximates Tikhonov / L2-regularization. However, that derivation assumes zero-mean Gaussian noise, not asymmetric Gumbel noise. Moreover, our results (Fig. 3) show DLGNs continue improving with training as the discretization gap narrows, without the accuracy decline characteristic of overfitting. **This suggests that overfitting prevention is not the reason our model works.**\n\nWe hope this clarifies the distinctions between our work and Kim (2023). Below, we address the reviewer’s remaining questions.\n\n1) We believe the reviewer is referring to Differentiable Logic Gate Networks (DLGNs), as clarified in our terminology section, since LGNs denote the discretized (post-training) networks.\nIn Figure 6, we visualize the loss landscapes of DLGNs and GLGNs, trained on the setup from Figure 3 (left). We will update the caption to clarify that $\\tau = 1$ was used for the DLGN. Even at this baseline $\\tau$ value, GLGNs already exhibit a visibly smoother landscape, supporting our core claim about Gumbel-induced smoothing.\nLemma 1 formalizes this effect, showing that the expected trace of the Hessian scales like $1/\\tau^2$, meaning smaller $\\tau$ implies stronger implicit smoothing. This is also reflected empirically in Figure 5.\nWe agree that additional $\\tau$-specific loss visualizations could serve as a useful sanity check. However, we believe that the combination of the $\\tau$-sweep (Figure 5), the visual comparison (Figure 6), and our theoretical derivation (Lemma 1) already provides a comprehensive view of the relationship between $\\tau$, smoothing, and optimization performance.\n2) Figure 4 compares training modes in our STE ablation. As stated in lines 235–236, Gumbel Soft refers to a DLGN with Gumbel noise applied to logits or equivalently a GLGN trained without STE. The labels (soft) and (discrete) correspond to the softmaxed network and its fully discretized LGN version. We will clarify this in the figure caption.\nThis directly supports the statements in lines 49–50 and aligns with Lemma. Smoothing from Gumbel noise improves convergence. Notably, adding STE slows convergence, confirming that it helps reduce the discretization gap but is not the main driver of faster training.\n\n3) As shown in Figure 4, Gumbel noise alone significantly reduces the gap between soft and discrete performance, whereas adding STE to Gumbel noise effectively removes it entirely. We will revise the abstract accordingly to reflect this distinction more clearly.\n\n\n4) From our experiments it does seem to be the case that the models get caught in a local optimum when tau is small. And while the Gumbel noise does significantly help with convergence, GLGNs still have some of the same issues with vanishing gradients as observed in the original works by Petersen et al.\n\n\n**Additional Evaluations**\n\nWe agree that broader evaluation is important, and we have now added results on CIFAR-100 and a wider network to support generalization and scalability claims.\n\nOn CIFAR‑100 (depths 6–12, width 256k), we observe the same trend as Figure 3: the discretization gap grows with network depth for DLGNs, while GLGNs remain stable. For the largest model, the DLGN has not converged after 1M iterations, reaching a test accuracy/gap of 6.11% / 3.36% points, whereas the GLGN converges in ~400k iterations with 0.23% / 0.0099% points.\nCIFAR‑100’s complexity suggests that larger networks are required, where we expect the discretization gap to become even more significant for DLGNs, making GLGNs critical for scaling.\n\nFor a very wide CIFAR‑10 network (depth 6, width 2048k), the DLGN gap remains high (drop of 2.5% points of test accuracy at convergence) and does not shrink significantly over training, whereas GLGNs continue reducing the gap (drop of 0.188% points of test at convergence), consistent with our main results."}, {"review_id": "vA9ZMZ8Q4p", "reviewer": "Reviewer_NbF6", "summary": "This paper addresses the discretization gap and slow convergence in Differentiable Logic Gate Networks (DLGNs).\nThe authors adopt Gumbel noise with a straight-through estimator (STE) to train Logic Gate Networks (LGNs) more effectively, aligning training-time behavior with inference-time discretization.\nThey formally show that injecting Gumbel noise leads to implicit regularization of the Hessian trace, thereby flattening the loss landscape and improving robustness to discretization.\nEmpirically, the proposed Gumbel LGNs achieve a 4.5$\\times$ speedup in training, a 98% reduction in the discretization gap, and 100% neuron utilization on CIFAR-10, with favorable scaling to deeper architectures.", "questions": "- While the paper provides a theoretical explanation connecting Gumbel noise injection to Hessian trace regularization, it would be helpful to include a more explicit mathematical formulation quantifying how much flatter the resulting local minima become.\n\n- Logic Gate Networks often admit multiple mathematically equivalent but structurally different networks. In Convolutional Differentiable Logic Gate Networks Section A.3. [4], logic simplification was applied before counting the number of gates. Was a similar logic simplification process applied in this work? If so, how does the number of gates after simplification compare between the default training method and the Gumbel-based method?\n\n[4] Convolutional Differentiable Logic Gate Networks, Petersen et al., NeurIPS 2024.", "limitations": "The authors wrote limitations of this work in the section 6.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths\n- The paper presents strong empirical results, demonstrating significant improvements in training speed, discretization gap reduction, and neuron utilization.\n- It provides an explicit theoretical justification connecting Gumbel noise injection to implicit Hessian trace regularization, which clarifies why this technique is helpful.\n- The authors conduct a systematic analysis of the discretization gap, including ablations over temperature, Hessian trace estimation, and neuron entropy.\n\n### Weaknesses\n- The use of Gumbel noise is a well-established approach in the NAS literature [1,2,3]. Since LGNs can be viewed as an instance of NAS, the application of Gumbel in this context is not novel in itself.\n\n[1] SNAS: stochastic neural architecture search, Xie et al., ICLR 2019.\n[2] Searching for A Robust Neural Architecture in Four GPU Hours, Dong et al., CVPR 2019.\n[3] DATA: Differentiable ArchiTecture Approximation, Chang et al., NeurIPS 2019.", "was_revised": true, "final_justification": "The paper presents solid theoretical and empirical contributions, particularly in analyzing the effect of Gumbel noise on the loss landscape and demonstrating strong performance improvements. However, the framing of DLGNs as a NAS problem is not novel, and the use of Gumbel-based methods has been well-studied in the NAS literature. While the paper is technically sound, its originality is limited. I therefore maintain a score of 4 (Borderline accept).", "rebuttal": "We thank the reviewer for their time and effort in helping to improve the paper. We will address the weaknesses and questions raised in the review.\n\n**Use of Gumbel in NAS.**\n\nThe reviewer is correct that these and other papers in the NAS literature have used Gumbel noise before. We will make this clearer in the related work where we already included [1,2,3]. We would also argue that:\n\n1) To the best of our knowledge, framing DLGNs as NAS is itself a novel and new insight. We hope future LGN work will be inspired by this, and further utilize advancements from NAS literature.\n\n2) Lemma 1 generalizes beyond DLGNs, and would also theoretically justify why [1,2,3] would have a smoother loss landscape and discretization gap, which are not addressed. This would position [1,2,3] against the broader NAS / Smoothing DARTS literature, e.g. Chen and Hsieh:  Stabilizing Differentiable Architecture Search via Perturbation-based Regularization, 2021 and Zela et. al: Understanding and Robustifying Differentiable Architecture Search, 2020.\n\n3) The scale of the search spaces are not comparable. Thus, our work demonstrates that the methods have potential to scale to much larger search spaces.\n\n\n**Quantification of the flatness of the minima**\n\nWe thank the reviewer for this thoughtful suggestion. We agree that further formalizing how Gumbel noise affects the curvature of local minima is an interesting direction. However, we believe this is challenging theoretically without assuming strong properties (e.g., convexity, L-Lipschitz gradients, or toy quadratic settings) that would not hold in our setting.\n\nOur current analysis (Lemma 1) already shows that Gumbel smoothing reduces the expected trace of the Hessian, and Figures 5–6 empirically reflect this effect. A deeper mathematical treatment of curvature under noise may indeed be more suitable for follow-up work in a broader setting.\n\n\n**On Logic Simplification and Gate Counting**\n\nWe did not apply the post‑training logic simplification step described in Appendix A.3 of [4], as the synthesis tool used there is not publicly available. Our results report *raw* gate counts for GLGNs and DLGNs so that differences between the default and Gumbel‑based methods directly reflect neuron \nutilization without additional processing.\n\n\n**Additional Evaluations**\n\nTo account for limited evaluation, we have now added results on CIFAR-100 and a wider network to support generalization and scalability claims.\n\nOn CIFAR‑100 (depths 6–12, width 256k), we observe the same trend as Figure 3: the discretization gap grows with network depth for DLGNs, while GLGNs remain stable. For the largest model, the DLGN has not converged after 1M iterations, reaching a test accuracy/gap of 6.11% / 3.36% points, whereas the GLGN converges in ~400k iterations with 0.23% / 0.0099% points.\nCIFAR‑100’s complexity suggests that larger networks are required, where we expect the discretization gap to become even more significant for DLGNs, making GLGNs critical for scaling.\n\nFor a very wide CIFAR‑10 network (depth 6, width 2048k), the DLGN gap remains high (drop of 2.5% points of test accuracy at convergence) and does not shrink significantly over training, whereas GLGNs continue reducing the gap (drop of 0.188% points of test at convergence), consistent with our main results."}, {"review_id": "u6HxZ0t7ox", "reviewer": "Reviewer_FVvS", "summary": "The paper first identifies two important limitations of the current LGNs: slow convergence during training (a simple network can take days to weeks to train) and a large discretization gap between training and inference (i.e., lots of unused neurons). The paper thus proposes to inject Gumbel noise into the gate selection process using the Gumbel-Softmax trick and shows significantly faster convergence time and fewer unused gates.", "questions": "The paper studies Logic Gate Networks (LGNs) because they \"enable efficient inference\". While the paper mainly compares different LGNs on convergence times, is there a comparison on inference times? Because the method can \"reduce the number of unused gates by 100%\", will it be significantly slower for inference?", "limitations": "Although it is stated that the method is \"dataset and architecture-independent\", the main model is a network with 12 layers and a width of 256k, and there is no implementation for convolutional architectures and on more complex datasets, which is crucial for real-world applications.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "The paper focuses on Logic Gate Networks (LGNs), an interesting emergent and novel research direction which might be important for the future of DNNs both as a novel neural architecture and as a way for efficient inference. However, the main idea, that using the Gumbel-Softmax for the optimization of LGNs, has already been proposed previously in a recent publication, \"Deep Stochastic Logic Gate Networks\" (https://ieeexplore.ieee.org/document/10301592), with the same idea, networks, and datasets, decreasing the novelty of the paper.", "was_revised": true, "final_justification": "This paper focuses on an often overlooked but important direction, LGNs. However, the findings are not that groundbreaking, given that similar ideas have been published by Kim (2023).", "rebuttal": "We thank the reviewer for their time and effort in helping to improve the paper. We will address the weaknesses and questions raised in the review. \n\n**Clarifying Experiments and Evaluation Scope**\n\nWe would like to clarify that our experiments already evaluate multiple model configurations, not just a single 12-layer, 256k-width network. Specifically, we sweep across four depths (6, 8, 10, 12) (Figure 3) and nine $\\tau$-values (0.01–2.0) (Figure 5) at a fixed width of 256k. These sweeps form the basis for our analysis of depth scaling, $\\tau$-driven smoothing, and their effect on the discretization gap. We will make this more explicit in the revised draft (Section “Gap Scaling with Depth”) to avoid any ambiguity.\n\n**Clarifying Novelty Relative to Y. Kim, \"Deep Stochastic Logic Gate Networks\"**\n\nWe thank the reviewer for pointing out Y. Kim, \"Deep Stochastic Logic Gate Networks,\" in IEEE Access, vol. 11, pp. 122488-122501, 2023 and will add it to the related works section for completeness. Our paper focuses on analyzing the discretization gap in Differentiable Logic Gate Networks (DLGNs) by framing the problem as Neural Architecture Search and studying how the gap scales, how $\\tau$-driven smoothing impacts optimization, and how these dynamics affect pruning and final performance. These aspects are not addressed in Kim (2023), which primarily explores empirical Gumbel-Softmax modifications to improve performance without analyzing gap scaling or optimization dynamics.\nAdditionally, Kim (2023) adopts a different neuron parametrization and design that make its regime fundamentally distinct from ours:\n1) It omits the $\\tau$ -parameter (central to Lemma 1 and Fig. 5 in our paper),\n2) Importantly, normalizes logits. In Kim (2023) this increases entropy in a way that would actually exacerbate the discretization gap, and\n3) Introduces a learnable Gumbel scaling parameter that alters the Hessian-trace term, complicating any direct comparison to our analysis.\n\nBecause of these differences in focus and model regime, our theoretical and empirical analysis addresses a distinct research question.\n\n**On Gumbel Noise and Regularization**\n\nKim (2023) interprets Gumbel noise as preventing overfitting via Bishop’s result that training with noise approximates Tikhonov/L2-regularization. This derivation assumes zero-mean Gaussian noise, not asymmetric Gumbel noise. Moreover, our results (Fig. 3) show DLGNs continue improving with additional training as the discretization gap narrows, without the accuracy decline typical of overfitting. This indicates that overfitting prevention is **not the primary reason our method works.**\n\n\n**Unused Gates**\n\nRegarding “unused gates”: in both DLGNs and GLGNs, all gates remain in the final LGN architecture and process signals. Many of the neurons in DLGNs maintain high entropy close to initialization or change only slightly by convergence, meaning they are not meaningfully utilized. The final architecture keeps the same gate count, and we expect inference time between the discretized LGNs to match between DLGNs and GLGNs.\n\n**Additional Evaluations**\n\nWe agree that broader evaluation is important, and we have now added results on CIFAR-100 and a wider network to support generalization and scalability claims.\n\nOn CIFAR‑100 (depths 6–12, width 256k), we observe the same trend as Figure 3: the discretization gap grows with network depth for DLGNs, while GLGNs remain stable. For the largest model, the DLGN has not converged after 1M iterations, reaching a test accuracy/gap of 6.11% / 3.36% points, whereas the GLGN converges in ~400k iterations with 0.23% / 0.0099% points.\nCIFAR‑100’s complexity suggests that larger networks are required, where we expect the discretization gap to become even more significant for DLGNs, making GLGNs critical for scaling.\n\nFor a very wide CIFAR‑10 network (depth 6, width 2048k), the DLGN gap remains high (drop of 2.5% points of test accuracy at convergence) and does not shrink significantly over training, whereas GLGNs continue reducing the gap (drop of 0.188% points of test at convergence), consistent with our main results."}]

Author rebuttal:

Meta-review: This paper tackles the critical issues of slow convergence and the "discretization gap" in Differentiable Logic Gate Networks (DLGNs), where performance drops when moving from continuous training relaxations to discrete inference-time models. The authors propose injecting Gumbel noise via a straight-through estimator, providing a novel theoretical analysis that links this approach to implicit Hessian regularization and a smoother loss landscape. The resulting empirical gains are substantial, demonstrating a 4.5x training speedup and a 98% reduction in the gap. 

The main point of contention during review was the novelty relative to Kim (2023), which also applied a Gumbel-Softmax approach to LGNs. However, based on the discussion, I believe that this paper's contributions are distinct and significant due to its explicit focus on analyzing and solving the discretization gap, its novel theoretical framing, and its impressive empirical results that dramatically improve the practicality of LGNs. Other concerns that were raised were during the discussion were the limited experimental scope. The authors ran additional experiments on CIFAR-100 and on a wider network to address this.

This paper was discussed in detail with the senior AC, and we decided to accept the paper. Note to the authors: please fully integrate the discussion of Kim (2023) into the final manuscript as promised. Furthermore, this final version should also include the additional empirical results provided during the rebuttal and consider revising its claims about the novelty of framing DLGNs as a NAS problem to accurately situate the work within the existing literature.
Decision: Accept (poster)