Title: Mind the Gap: Removing the Discretization Gap in Differentiable Logic Gate Networks
Venue: NeurIPS 2025
Accepted: yes
Keywords: Logic gate

logic gate network

efficient training

Gumbel noise
Abstract: Modern neural networks exhibit state-of-the-art performance on many existing benchmarks, but their high computational requirements and energy usage cause researchers to explore more efficient solutions for real-world deployment.
    Differentiable logic gate networks (DLGNs) learns a large network of logic gates for efficient image classification. However, learning a network that can solve simple problems like CIFAR-10 or CIFAR-100 can take days to weeks to train. Even then, almost half of the neurons remains unused, causing a \emph{discretization gap}. This discretization gap hinders real-world deployment of DLGNs, as the performance drop between training and inference negatively impacts accuracy.
    We inject Gumbel noise with a straight-through estimator during training to significantly speed up training, improve neuron utilization, and decrease the discretization gap. 
    We theoretically show that this results from implicit Hessian regularization, which improves the convergence properties of DLGNs. We train networks $4.5 \times$ faster in wall-clock time, reduce the discretization gap by 98\%, and reduce the number of unused gates by 100\%.


Peer reviews:
comments: SUMMARY:
The paper first identifies two important limitations of the current LGNs: slow convergence during training (a simple network can take days to weeks to train) and a large discretization gap between training and inference (i.e., lots of unused neurons). The paper thus proposes to inject Gumbel noise into the gate selection process using the Gumbel-Softmax trick and shows significantly faster convergence time and fewer unused gates.

STRENGTHS AND WEAKNESSES:
The paper focuses on Logic Gate Networks (LGNs), an interesting emergent and novel research direction which might be important for the future of DNNs both as a novel neural architecture and as a way for efficient inference. However, the main idea, that using the Gumbel-Softmax for the optimization of LGNs, has already been proposed previously in a recent publication, "Deep Stochastic Logic Gate Networks" (https://ieeexplore.ieee.org/document/10301592), with the same idea, networks, and datasets, decreasing the novelty of the paper.

QUESTIONS:
The paper studies Logic Gate Networks (LGNs) because they "enable efficient inference". While the paper mainly compares different LGNs on convergence times, is there a comparison on inference times? Because the method can "reduce the number of unused gates by 100%", will it be significantly slower for inference?

LIMITATIONS:
Although it is stated that the method is "dataset and architecture-independent", the main model is a network with 12 layers and a width of 256k, and there is no implementation for convolutional architectures and on more complex datasets, which is crucial for real-world applications.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-06-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the discretization gap and slow convergence in Differentiable Logic Gate Networks (DLGNs).\nThe authors adopt Gumbel noise with a straight-through estimator (STE) to train Logic Gate Networks (LGNs) more effectively, aligning training-time behavior with inference-time discretization.\nThey formally show that injecting Gumbel noise leads to implicit regularization of the Hessian trace, thereby flattening the loss landscape and improving robustness to discretization.\nEmpirically, the proposed Gumbel LGNs achieve a 4.5$\\times$ speedup in training, a 98% reduction in the discretization gap, and 100% neuron utilization on CIFAR-10, with favorable scaling to deeper architectures.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- The paper presents strong empirical results, demonstrating significant improvements in training speed, discretization gap reduction, and neuron utilization.\n- It provides an explicit theoretical justification connecting Gumbel noise injection to implicit Hessian trace regularization, which clarifies why this technique is helpful.\n- The authors conduct a systematic analysis of the discretization gap, including ablations over temperature, Hessian trace estimation, and neuron entropy.\n\n### Weaknesses\n- The use of Gumbel noise is a well-established approach in the NAS literature [1,2,3]. Since LGNs can be viewed as an instance of NAS, the application of Gumbel in this context is not novel in itself.\n\n[1] SNAS: stochastic neural architecture search, Xie et al., ICLR 2019.\n[2] Searching for A Robust Neural Architecture in Four GPU Hours, Dong et al., CVPR 2019.\n[3] DATA: Differentiable ArchiTecture Approximation, Chang et al., NeurIPS 2019.\n\nQUESTIONS:\n- While the paper provides a theoretical explanation connecting Gumbel noise injection to Hessian trace regularization, it would be helpful to include a more explicit mathematical formulation quantifying how much flatter the resulting local minima become.\n\n- Logic Gate Networks often admit multiple mathematically equivalent but structurally different networks. In Convolutional Differentiable Logic Gate Networks Section A.3. [4], logic simplification was applied before counting the number of gates. Was a similar logic simplification process applied in this work? If so, how does the number of gates after simplification compare between the default training method and the Gumbel-based method?\n\n[4] Convolutional Differentiable Logic Gate Networks, Petersen et al., NeurIPS 2024.\n\nLIMITATIONS:\nThe authors wrote limitations of this work in the section 6.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the discretization gap between training and testing of differentiable logic gate networks by adding Gumbel noise to the softmax function during logic gate selection training. The perturbation-based regularization smooths the loss landscape, resulting in more stable gradient signals and faster convergence. By using the Gumbel-Softmax trick, the selection of logic gates can be effectively synchronized between the training and inference phases, thereby reducing the discretization gap.\n\nSTRENGTHS AND WEAKNESSES:\n- **Strengths**:\n    - The authors provide a theoretical analysis showing that Gumbel perturbations during training act as a form of Hessian trace regularization, derived using a second-order Taylor expansion of the Gumbel-Softmax function, somewhat inspired by [3,4]. Furthermore, the paper presents a comparison of the loss landscapes between LGN and GLGN.\n- **Weaknesses:**\n    1. Gumbel-Softmax-based differentiable logic gate networks have been explored in *Deep Stochastic LGN*, where the Gumbel-Softmax formulation is further modified.\n    2. In this paper, only a single model is tested (depth: 12, width: 256k), whereas the LGN paper evaluates multiple models with varying depths and widths. It would be more convincing to include comparisons with the model configurations used in Petersen et al.'s LGN paper.\n    3. The main contribution of this paper is Lemma 1 (Gumbel-Smoothing), which derives a trace of the Hessian term in the expectation of the Gumbel softmax function. The smoothness of the softmax-based class probability function (Eq. 4) depends on the trace of the Hessian term with a tau (temperature) related weight (in front of the Hessian trace) and also a temperature (tau) inserted into the softmax function. However, the temperature term in the softmax function is neither specified nor explored (in Figure 3).\n\n\n*Deep Stochastic LGN*:\nY. Kim, \"Deep Stochastic Logic Gate Networks,\" in IEEE Access, vol. 11, pp. 122488-122501, 2023,\n\nQUESTIONS:\n1. To isolate the effect of perturbations on the loss function, it is necessary to observe LGNs based on L(softmax(z/τ)) across different τ values. In Lemma 1, τ (tau) is not completely separated, so effect of tau seems mixed in the final loss. In figure 6, can different tau values  show different loss landscapes?\n2. As shown in Figure 4, despite a larger discretization gap, faster convergence of test accuracy is observed. Does this result match with the statement in lines 49–50 (\"Since the loss landscape is smoother, the gradient signal is better, thus making the networks converge faster.\")? Is the choice of backward method (or gradient propagation) more critical for convergence speed than the discretization gap itself? Also, it needs to clarify what Gumbel Soft (discrete) and Gumbel Soft (soft) are.\n3. Connecting to the above point, is the reduction of the discretization gap from the use of Straight Through Estimation (STE) or Gumbel Noise (which causes flat curvature)? In the Abstract: \"We inject Gumbel noise with a straight-through estimator during training to significantly speed up training,… and decrease the discretization gap.\" This is ambiguous.  \n4. Although smaller τ (temperature) leads to faster convergence, the best maximum and final test accuracy is achieved when τ=1 (Table 1). Do these rapid training convergences sacrifice test accuracy due to the perturbations, or could there be another interpretation? With  regularization, does worse generalization performance makes sense? Is it only contributing to faster convergence toward a local optimum?\n\nLIMITATIONS:\nAs noted by the authors, the models should be tested over more datasets.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work applies some ideas from other fields to Differentiable Logic Gate Networks (DLGNs), significantly improving their training speed and performance.\nUsually, the training of DLGNs involves keeping a distribution over the possible logical gates and employing the associated weighted sum of the gates' outputs as the output of the node.\nThe core idea of this work is to inject Gumbel noise into the logit values underlying the distribution over the gates and to select the output of the gate with the highest value as the output of the node (no weighted sum). In the backward pass, the non-differentiable \"winner-takes-all\" operation is approximated using a softmax function.\nThe authors provide some theoretical justification for the approach and provide experiments that indicate striking improvements over DLGNs.\n\nSTRENGTHS AND WEAKNESSES:\nThis work advances the field of DLGNs significantly.\nThe technique is simple and clean and the results seem impressive.\nIn fact, I believe the authors could better advertise the improvement in robustness to depth increase as this is an important aspect of their contribution but seems only to be mentioned around line 223.\nPresentation is sufficiently clear, and the paper is well-structured with thoughtful guidance to abstracts when suitable.\nThe theoretical results, though not particularly deep, are a nice addition and provide some justification for the approach.\nThe rationale behind the experiments is well communicated and the results are clearly presented.\n\nMoving to the weaknesses, the main one seems to be the limited scope of the experiments.\nI would go harder on this if the authors had not stated it so clearly as a limitation (the honesty is appreciated).\nI also understand that those experiments are quite expensive to run.\nStill, the lack of error bars is a major issue, as it makes it hard to assess the significance of the results, which could kill the paper if the improvements were not so pronounced.\n\nMoreover, the claim that unused gates are reduced by 100% can give the wrong impression.\nPersonally, I did not expect to find much later in the text (around line 273) that this claim is based on defining \"unused gates\" as those with entropy above the 2.5%-percentile of the entropy distribution.\nThis threshold appears somewhat arbitrary; if there is a principled reason for choosing this rule, the authors should clarify and justify it in the main text.\nOtherwise, the claim should be reconsidered or reworded to avoid potential misunderstanding.\n\nFinally, the use of a fixed dimensionality (16) in Lemma 1 is unusual.\nI do understand that this happens to be the number of gates considered in the implementation of LGNs, but I fail to see a reason to limit generality.\nUnderstanding how the result varies with the number of gates adds information on its own.\n\nBelow I list some minor points that could help the authors improve the work:\n- Although not bad, the introduction of concepts is not particularly good.\nI feel like I only got it without much effort because I already had contact with the subject.\nFor instance, the quick overview of LGNs provided can be improved significantly without taking much more space, if any.\nA good example of how to achieve this is already in the references of the paper: [25].\nSimply adding an explicit recommendation to that source for an overview of LGNs would already be a meaningful improvement.\n- The mathematical typesetting looks a bit amateurish.\nTypos are not uncommon and the use of language, which is quite solid elsewhere, becomes clumsy around mathematical expressions.\nFor some examples, almost all colons in the text are used incorrectly, the statement of Lemma 1 reads a bit odd, and Algorithm 1 is hard to parse since it lacks a proper description of the inputs and their domains (variables need types!).\n- More references should be provided.\nNot that Yoshua Bengio needs the credit for the straight-through estimator, but it's good practice nonetheless.\nThere are many other examples of claims that could (and should) be easily backed by references.\n- The third equality in the proof of Lemma 2 shouldn't be there.\n\n\nOverall, I liked the work and I believe it's a clear accept.\n\nQUESTIONS:\nIn Lemma 1, if we were to replace the $16$ by a general $d \\in \\mathbb{N}$, what would the thesis look like?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 2, "DATE": "2025-07-01"}

comments: SUMMARY:
This paper introduces Gumbel Logic Gate Networks (Gumbel LGNs) to solve two major issues with existing Differentiable Logic Gate Networks (DLGNs): slow training convergence and a significant "discretization gap" where performance drops after training. By injecting Gumbel noise during the training process, the authors create a smoother optimization landscape. This new method trains 4.5 times faster, reduces the discretization gap by 98%, and eliminates unused logic gates.

STRENGTHS AND WEAKNESSES:
**Strengths:**\
(1) The research area of logic gate networks deserves more attention. Analogous to human brain using neurons as basic components, using logic gates as basic components in digital artificial neural networks is a very reasonable research direction. \
(2) The motivation of this work is clear. Hard, brittle loss landscape in existing DLGNs and the inefficiency of neuron usage in inference are significant problems to solve. \
(3) The results demonstrate the effectiveness of Gumble LGN, by being 4.5 times faster than existing DLGNs. \
\
**Weaknesses:**\
(1) The techniques proposed should be more closely bridged to their advantages in practical tasks. This work may have greater impact if some comparisons with standard neural networks can be provided to demonstrate the advantages of Gumble LGN. For instance, with discretization gap and unused neurons minimized, what is the estimated energy efficiency like compared to baselines and traditional NN approaches? Since energy efficiency is pivotal in edge device deployment.

QUESTIONS:
(1) What is the estimated energy efficiency like? \
(2) In addition to the energy efficiency on edge devices, how much does this work advance the practical availability of LGN?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-07-02