Title: G-Net: A Provably Easy Construction of High-Accuracy Random Binary Neural Networks
Venue: NeurIPS 2025
Accepted: yes
Keywords: Hyperdimensional Computing

Random Binary Neural Networks
Abstract: We propose a novel randomized algorithm for constructing binary neural networks with tunable accuracy. This approach is motivated by hyperdimensional computing (HDC), which is a brain-inspired paradigm that leverages high-dimensional vector representations, offering efficient hardware implementation and robustness to model corruptions. Unlike traditional low-precision methods that use quantization, we consider binary embeddings of data as points in the hypercube equipped with the Hamming distance. We propose a novel family of floating-point neural networks, G-Nets, which are general enough to mimic standard network layers. Each floating-point G-Net has a randomized binary embedding, an embedded hyperdimensional (EHD) G-Net, that retains the accuracy of its floating-point counterparts, with theoretical guarantees, due to the concentration of measure. Empirically, our binary models match convolutional neural network accuracies and outperform prior HDC models by large margins, for example, we achieve almost 30\% higher accuracy on CIFAR-10 compared to prior HDC models. G-Nets are a theoretically justified bridge between neural networks and randomized binary neural networks, opening a new direction for constructing robust binary/quantized deep learning models. Our implementation is available at \url{https://github.com/GNet2025/GNet}.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on hyperdimensional computing (HDC), where the data is first transformed to some very high dimensional binary vector, and then inference is performed via some simple linear method. This paper shows how a network of a specific (but still quite generic) form can be converted to binary (in some sense) network in the infinite limit and some convergence results are presented for finite widths. Experiments on some datasets, such as MNIST and CIFAR10 are presented. The central result of the paper is to train a floating point network and binarizing it later in a way nothing is lost (in the infinite limit; in finite width, we are losing something and the paper quantifies how much)\n\nThe main identity that is behind all the constructions is the Grothendieck identity (quantifying the probability for two vectors $u,v$ that they are on the same side of a hyperplane with normal vector $g$, $g$ is normally (or rademacher) distributed; the probability is some function of $<u,v>$ for unit vectors $u,v$). Then, by the law of large numbers, if we draw enough normal vectors $g_i$, we can recover $<u,v>$ from the binary indicators sign$(<g_i,u>)$. Then, one  can express the inner product $<u,v>$ between input and weight matrix using the inner product of the binary embedings (sign$(<g_i,u>)$ for i-th element of embedding of $u$) modulo some technicalities such as input normalization or non-linearity application.  This is then used for every layer sequentially.\n\nSTRENGTHS AND WEAKNESSES:\nThe technique itself is neat and I like the rigorous quantification of the approximation error. I am not aware of the technique being used in a similar context, but my knowledge of the field is arguably limited.\n\nOn the other hand, there seem to be limited motivation. The goal of this paper is a bit vague. It presents a way  of constructing (binary, in some sense) networks with some goals in mind (efficient computation), but the evaluation solely focuses on accuracy on some datasets and it is not clear if this method should be preferred over standard CNN, because no concrete scenario where standard CNN is worse (or not applicable) is not presented.\n\nI personally did not understand the HDC link as descibed in the paper. Specially, this paper does not propose a method falling to that framework (as described in the paper) in a nontrivial sense; i.e., there is no high dimensional representation with a linear classifier. Instead, to me this work falls in the binary (resp. integer) network paradigm, where with suitable design choices in the paper, the internal representations/operations are integral. Thus, I am not convinced the baselines are chosen well.\n\nTo summarize, the paper is entertaining, presents an interesting technique, but I find it hard to conclude more from the paper, because the task considered in the paper is not well specified and I fail to see a concrete scenario where I would consider using this technique. I think the paper would benefit from proper task definition and subsequent comparison with other methods on the task.\n\n#### typo\nI think fig2 contains wrong variable names at places. Say in the embedded layer. First, we compute embedding $y = sign(Gx)$, then we should multiply this with $sign(GW)$ (cf. eq (3)), but the \"linear transform by embedded weights\" multiplies this weight thing by $x$ (and not by $y$. This seem to be a consistent typo in the figure, where $x$ is often used instead of $y$.\n\nQUESTIONS:\nThe network architecture used in the paper is a bit weird (chosen so that it can be binarized) and the experiments were presented only on small datasets. Can the method me scaled up to (say) Imagenet? CIFAR100? If not, what is the bottleneck? Does the training get challenging, or is it the binarization?\n\nLIMITATIONS:\n-", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates a method for constructing highly accurate binary neural networks, motivated by hyper-dimensional computing (HDC). The key idea builds on the Grothendieck identity and random matrix theory, which imply that the composition of a linear transformation and the arcsin function can be approximately converted into a binary linear transformation by multiplying a large random Gaussian matrix to the continuous parameter and input vector. Based on this observation, the authors introduce G-Net, a (continuous) neural network with an activation involving the arcsin function. After training of G-Net, it can be converted into a binary neural network (EHD G-Net) by the above technique, enabling binary inference in an approximately lossless way. The experimental results show that the proposed method successfully constructs binary neural networks that outperform previous methods in HDC.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- (S1) The paper is written so clearly and easy to follow.\n- (S2) The paper demonstrates an excellent application of the Grothendieck identity and random matrix theory for binary neural networks. Although the continuous counterpart, G-Net, has a seemingly weird activation involving arcsin, they is easily learnable via backpropagation and SGD.\n- (S3) While the main motivation in this paper is HDC, the proposed method may have broader applications as a novel technique of lossless binarization for neural networks.\n\n**Weaknesses**\n\n- (W1) Even though the method is motivated by the potential application to efficient computing with HDC, the paper lacks an analysis of actual or idealized computational costs.\n  - Is the proposed approach truly efficient? For instance, one could trivially binarize a given DNN by viewing floating-point weights as bit sequences. How does the proposed method compare in terms of the number of parameters or computational costs against such naive binarization?\n- (W2) According to Definition 3.2, in each layer, the input vector should be binarized by multiplying random Gaussians as well as the parameter matrix. However, in Figure 2 (b), it appears that such binarization (called sign embedding or input embedding) is applied only in the first layer.\n  - I'm confused by this gap. If we employed the sign embedding only in the first layer following Figure 2 (b), the approximate consistency would be broken. Indeed, the proof of Theorem 4.3 recursively applies Theorem 4.1, and thus the sign embedding seems to be required in each intermediate layer.\n  - On the other hand, if we employ Definition 3.2 as is, I'm concerned that the sign embedding in intermediate layers may be incompatible with the HDC computation because it involves continuous operations in each layer.\n- (W3) The paper does not cite a lot of prior work in neural network binarization/quantization. Even if the main motivation is HDC, there are many such prior work having a similar purpose or related method to this paper, and thus should be appropriately discussed as related work.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces G-Net, a novel approach for constructing high-accuracy random binary neural networks. The method is inspired by hyperdimensional computing (HDC) and aims to bridge the gap between traditional neural networks and randomized binary neural networks. The core idea is to train a \"floating-point\" G-Net in the real-valued domain and then convert it into an Embedded Hyperdimensional (EHD) G-Net, which operates on binary data, without requiring further training in the binary space. This conversion is theoretically justified by Grothendieck's identity and concentration of measure, ensuring that the EHD G-Net retains the accuracy of its floating-point counterpart. The paper provides theoretical consistency guarantees for both Gaussian and Rademacher embeddings and demonstrates strong empirical performance, significantly outperforming prior HDC models on benchmark datasets like MNIST and CIFAR-10, while rivaling real-valued CNNs.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n- The paper provides rigorous theoretical guarantees for the accuracy preservation during the conversion from G-Net to EHD G-Net, leveraging Grothendieck's identity and concentration of measure. This is a significant contribution, as many prior HDC works lack such deep theoretical underpinnings.\n- The concept of training in the real domain and then performing an \"inexpensive binary encoding\" to achieve high-accuracy binary networks without training in the high-dimensional, constrained binary space is highly innovative and practical.\n- The experimental results clearly show that G-Nets outperform existing HDC models by substantial margins (e.g., almost 30% higher accuracy on CIFAR-10 compared to prior HDC models). They also achieve accuracies comparable to real-valued convolutional neural networks, which is a major step forward for HDC.\n- The motivation from HDC for efficient hardware implementation and the exploration of Rademacher embeddings (which are simpler to generate and apply than Gaussian vectors) highlight the practical relevance and potential for deployment on edge and low-energy devices.\n- The paper is well-organized, with a clear introduction of concepts, theoretical results, and experimental validation. \n- The authors provide a link to their GitHub repository, which significantly aids reproducibility and allows for further research and application of their method.\n\nWeakness:\n- G-Nets require -normalization of inputs and row-wise normalization of weight matrices. While the authors argue these are advantageous, they might limit flexibility or require specific preprocessing steps not explicitly detailed for all scenarios.\n- The network consistency analysis (Section 4.1) focuses on the base ASU G-Net case and defers comprehensive treatment for TASU and RASU layers to an extended presentation. While understandable for brevity, it leaves a theoretical gap in the main paper for the more practically relevant RASU/TASU architectures.\n- The $\\sqrt{n/p}$​ discrepancy term in Rademacher embedding that doesn't vanish with increasing N is acknowledged, but its practical implications or strategies to mitigate it could be discussed further, especially given the claim that Rademacher embeddings achieve \"comparably close performance.\"\n\nQUESTIONS:\n- Could the authors elaborate on the practical implications of the l2​-normalization and row-wise weight normalization constraints for real-world datasets that might not inherently adhere to these properties? Are there recommended preprocessing steps or training adjustments for such cases?\n- For practical deployment, when would a user prefer a RASU G-Net over a TASU G-Net?\n- Regarding the assumption $|w_i^T​x| \\geq l_{min} ​>0$ for TASU analysis (Theorem 4.2), how robust are the empirical results to violations of this assumption? Are there strategies to ensure this condition is met during training, or does it hold naturally in common scenarios?\n- The paper mentions that \"wide neural networks experience only minor changes in their weight distributions during training.\" Could the authors expand on how this property is leveraged or maintained to ensure the \"spread-out\" condition for applying Theorem 4.4 and Proposition 5.1?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new class of randomized binary neural networks, G-Nets, designed to operate in the hyperdimensional Hamming space with guaranteed accuracy. Unlike traditional quantization methods, G-Nets employ a random binary embedding that enables neural networks to be represented in binary form without a significant loss in performance. This embedding is derived from hyperdimensional computing (HDC), a paradigm inspired by brain-like computation using high-dimensional vector representations. The approach provides theoretical guarantees for performance and is shown to outperform prior HDC models, achieving near state-of-the-art accuracy with significantly lower computational resources. The authors empirically demonstrate the effectiveness of G-Nets on classification benchmarks such as CIFAR-10 and MNIST.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nNovel Approach: The introduction of randomized binary embeddings and the theoretical framework based on HDC for constructing high-accuracy binary neural networks is highly innovative. It opens up new possibilities for low-resource deep learning.\n\nTheoretical Rigor: The paper provides strong theoretical underpinnings, including proofs and consistency analysis, which justify the performance of the proposed method. The use of Grothendieck’s identity and concentration of measure guarantees the method's accuracy.\n\nEmpirical Results: The experimental results on MNIST, CIFAR-10, and human activity recognition clearly demonstrate the superiority of the proposed G-Nets over prior methods in HDC. The results are consistent and include appropriate error bars, making them reliable.\n\nReproducibility: The paper offers detailed information about experimental setups, making it easy for other researchers to reproduce the results. The code is publicly available on GitHub, which is a strong positive.\n\nPractical Relevance: The proposed G-Nets show significant potential for edge and low-energy computing, which makes the work practically valuable for resource-constrained devices.\n\nWeaknesses:\nLimited Task Evaluation: While the empirical results are compelling, they focus primarily on image classification tasks (MNIST, CIFAR-10). The generalizability of the approach to other domains, such as NLP or time-series forecasting, remains untested.\n\nHigh-dimensional Embeddings: The reliance on high-dimensional binary embeddings may become inefficient in some use cases, especially when the hardware constraints are not well-suited for high-dimensional spaces. More discussion on potential downsides of this high-dimensionality in practical applications would be beneficial.\n\nComputational Complexity: The theoretical analysis focuses on Gaussian embeddings, but the paper does not sufficiently address the computational overhead when applying these methods in real-world scenarios with larger networks or datasets. The complexity analysis of Rademacher embeddings is promising, but more clarity is needed on the trade-offs.\n\nEvaluation against More Baselines: The comparison to existing methods in the HDC domain is solid, but the paper could benefit from a comparison against more state-of-the-art deep learning models, including quantized neural networks and binary neural networks.\n\nQUESTIONS:\nTask Generalization: Could the authors explore the application of G-Nets to other domains such as NLP or time-series forecasting?\n\nDimensionality vs. Efficiency: While high-dimensional embeddings contribute to the accuracy of the method, how does the model perform on hardware with limited resources, such as microcontrollers or edge devices? Could the authors explore how to balance dimensionality and hardware constraints?\n\nComparison with Binary Neural Networks: How do G-Nets perform compared to existing binary neural network architectures, such as Bi-Real Net or XNOR-Net, in terms of both accuracy and computational efficiency?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-05"}