BDC-Occ: Binarized Deep Convolution Unit For Binarized Occupancy Network

## Abstract

Abstract Existing 3D occupancy networks demand significant hardware resources, hindering the deployment of edge devices.
Binarized Neural Networks (BNN) offer substantially reduced computational and memory requirements.
However, their performance decreases notably compared to full-precision networks.
Moreover, it is challenging to enhance the performance of binarized models by increasing the number of binarized convolutional layers, which limits their practicability for 3D occupancy prediction.
To bridge these gaps, we propose a novel binarized deep convolution (BDC) unit that effectively enhances performance while increasing the number of binarized convolutional layers.
Firstly, through theoretical analysis, we demonstrate that 1 × 1 1 1 1\times 1 1 × 1 binarized convolutions introduce minimal binarization errors.
Therefore, additional binarized convolutional layers are constrained to 1 × 1 1 1 1\times 1 1 × 1 in the BDC unit.
Secondly, we introduce the per-channel weight branch to mitigate the impact of binarization errors from unimportant channel features on the performance of binarized models, thereby improving performance while increasing the number of binarized convolutional layers.
Furthermore, we decompose the 3D occupancy network into four convolutional modules and utilize the proposed BDC unit to binarize these modules.
Our BDC-Occ model is created by applying the proposed BDC unit to binarize the existing 3D occupancy networks.
Comprehensive quantitative and qualitative experiments demonstrate that the proposed BDC-Occ is the state-of-the-art binarized 3D occupancy network algorithm.
Code and models are publicly available at https://github.com/zzk785089755/BDC .

## 1 Introduction

Recent advancements in 3D occupancy prediction tasks have significantly impacted the fields of robotics [; ; ] and autonomous driving [; ; ; ], emphasizing the importance of accurate perception and prediction of voxel occupancy and semantic label within 3D scenes.
However, occupancy prediction requires predicting dense voxels, which leads to substantial computational expenses [; ; ].
Moreover, the formidable performance of occupancy prediction models relies on increasing model size [].
These factors collectively hinder the deployment of high-performance occupancy prediction networks on edge devices.
For instance, Convolutional Neural Networks (CNN) [; ; ; ] possess hardware-friendly and easily deployable characteristics.
Moreover, CNN-based occupancy prediction networks [; ] exhibit outstanding performance, making them the primary choice for deployment on edge devices.
However, high-performance CNN-based occupancy networks [; ] often involve complex computations and numerous parameters.
Therefore, it is necessary to introduce model compression techniques [] to reduce the computational complexity and parameter count of CNN-based occupancy networks.

Research on neural network compression and acceleration encompasses four fundamental methods: quantization [], pruning [], knowledge distillation [], and lightweight network design [].
Among these methods, Binarized Neural Networks (BNN), which fall under the quantization category, quantize the weights and activations of CNN to only 1 bit, leading to significant reductions in memory and computational costs.
By quantizing both weights and activations to 1 bit, BNN [] can achieve a memory compression ratio of 32$\times$ and a computational reduction of 64$\times$ when implemented on Central Processing Units (CPU).
Furthermore, compared to full-precision models, BNN [] only requires logical operations such as XNOR and bit counting, making them more easily deployable on edge devices.

Recent studies, such as BBCU [] and BiSRNet [], have demonstrated the capability of binarizing complex models with promising performance in tasks such as image super-resolution [] and denoising [].
We try replacing each full-precision convolutional unit in the occupancy network with the binarized convolutional units proposed by these binarization algorithms.
Such a binarized model could achieve a respectable level of accuracy but still a notable performance gap compared to the full-precision model.
In full-precision models, it’s common sense that increasing convolutional layers can lead to performance improvements.
However, the binarized model did not exhibit a trend of performance improvement as the number of binarized convolutional layers increased.
Instead, there is a tendency for performance to decline, making it challenging for binarized models to improve performance by increasing the number of convolutional layers [].
Insufficient performance of binarized occupancy networks inevitably will have adverse effects on the perception of 3D space, thereby restricting the deployment of binarized models in autonomous vehicles.

Figure: (a) 3D Occupancy Prediction Result
Refer to caption: extracted/5623170/figure/occ.png

Therefore, addressing the issue of decreasing accuracy with increasing binarized convolutional layers is crucial for bridging the performance gap between binarized and full-precision models.
To address this issue, we propose a novel BNN-based method, namely Binarized Deep Convolution Occupancy (BDC-Occ) network for efficient and practical occupancy prediction, marking the first study of binarized 3D occupancy networks.
First, our theoretical analysis concludes that increasing the size of the binarized convolutional kernel also increases the binarization error.
To deepen the binarized model, we add $1\times 1$ binarized convolutions as extra layers to reduce binarization errors.
Secondly, we propose the per-channel weight branch, placing newly added convolution layers within this branch to further enhance binarized model performance.
Thirdly, based on the above analysis, we design the Binarized Deep Convolution (BDC) unit, which remarkably enhances binarized model performance while deepening the binarized convolution layers.
Finally, we decompose the 3D occupancy network into four fundamental modules and customize binarization using the BDC unit for each module.

The innovations and contributions of this paper are summarized as follows:

(i) We propose a novel BNN-based occupancy network named BDC-Occ, utilizing the Binarized Deep Convolution (BDC) unit.
To our knowledge, it is the first work to study the binarized occupancy network.

(ii) Based on theoretical analysis, the proposed BDC unit employs a $1\times 1$ binarized convolution layer to deepen the binarized model.
Additionally, we introduce the per-channel weight branch to mitigate the impact of the added binarized convolution layer on binarized model performance due to binarization errors.
The 3D occupancy network is decomposed into four fundamental modules, allowing for a customized design using the BDC unit.

(iii) We conduct extensive experiments on the Occ3D-nuScenes dataset, demonstrating significant performance improvements through deepening the binarized neural network.
Our method achieved state-of-the-art (SOTA) results, and the performance of our binarized model, BDC-Occ, is comparable to that of full-precision models.

## 2 Related Work

### 2.1 3D Occupancy Prediction

The 3D occupancy prediction task comprises two sub-tasks: predicting the geometric occupancy status for each voxel in 3D space and assigning corresponding semantic labels.
We can categorize mainstream 3D occupancy networks into two architectures: CNN architecture based on the LSS [; ; ; ; ; ; ] method and Transformer architecture based on the BEVFormer [; ; ; ; ; ; ] method.
Due to the deployment advantages of CNN models, this paper focuses on CNN-based 3D occupancy networks.
MonoScene [] is a pioneering work that utilizes a CNN framework to extract 2D features, which it then transforms into 3D representations.
BEVDet-Occ [] utilizes the LSS method to convert image features into BEV (Bird’s Eye View) features and employs BEV pooling techniques to accelerate model inference.
FlashOcc [] replaces 3D convolutions in BEVDet-Occ with 2D convolutions and occupancy logits derived from 3D convolutions with channel-to-height transformations of BEV-level features obtained through 2D convolutions.
SGN [] adopts a dense-sparse-dense design and proposes hybrid guidance and efficient voxel aggregation to enhance intra-class feature separation and accelerate the convergence of semantic diffusion.
InverseMatrixVT3D [] introduces a new method based on projection matrices to construct local 3D feature volumes and global BEV features.
Despite achieving impressive results, these CNN-based methods rely on powerful hardware with substantial computational and memory resources, which are impractical for edge devices.
How to develop 3D occupancy prediction networks for resource-constrained devices remains underexplored.
Our goal is to address this research gap.

### 2.2 Binarized Neural Network

BNN [; ; ; ; ; ; ; ; ; ] represents the most extreme form of model quantization, quantizing weights and activations to just 1 bit.
Due to its significant effectiveness in memory and computational compression, BNN [] finds wide application in both high-level vision and low-level vision.
For instance, Xia et al. [] designed a binarized convolutional unit, BBCU, for tasks such as image super-resolution, denoising, and reducing artifacts from JPEG compression.
Cai et al. [] devised a binarized convolutional unit, BiSR-Conv, capable of adjusting the density and distribution of representations for hyperspectral image (HSI) recovery.
However, the potential of BNN in 3D occupancy tasks remains unexplored.
Hence, this paper explores binarized 3D occupancy networks, aiming to maintain high performance while minimizing computational and parameter overhead.

## 3 Method

### 3.1 Base Model

The full-precision models to be binarized should be lightweight and easy to deploy on edge devices.
However, prior 3D occupancy network models based on CNNs [] or Transformers [; ] have high computational complexity or large model sizes.
Some of these works utilize complex operations such as deformable attention, which are challenging to binarize and deploy on edge devices.
Therefore, we redesign a simple, lightweight, and deployable baseline model without using complex computational operations.

BEVDet-Occ [] and FlashOcc [] demonstrate outstanding performance in 3D occupancy prediction tasks using only lightweight CNN architectures.
Inspired by these works, we adopt the network structure shown in Figure 2 as our full-precision baseline model.
It consists of an image encoder $\mathcal{E}_{2D}$, a view transformer module $\mathcal{T}$, a BEV encoder $\mathcal{E}_{BEV}$, and an occupancy head $\mathcal{H}$.
The occupancy prediction network is composed of these modules concatenated sequentially.
Assuming the input images are $\textbf{I}\in\mathbb{R}^{N_{view}\times 3\times H\times W}$, the occupancy prediction output $\textbf{O}\in\mathbb{R}^{X\times Y\times Z}$ can be formulated as

$$ $\textbf{O}=\mathcal{H}(\mathcal{E}_{BEV}(\mathcal{T}(\mathcal{E}_{2D}(\textbf{ I}))))$ (1) $$

where $H$ and $W$ represent the height and width of the input images, and $X$, $Y$, and $Z$ denote the length, width, and height of the 3D space, respectively, $N_{view}$ represents the number of multi-view cameras.
Please refer to the supplementary materials for a more detailed description of the base model.

Figure: Figure 2: CNN-based 3D Occupancy Network
Refer to caption: x1.png

### 3.2 Binarized Deep Convolution

Due to its outstanding performance and lightweight architecture, FlashOcc [] serves as the full-precision baseline model for the binarized model.
Its performance reaches 37.84 mIoU, which sets the upper performance bound for the binarized models.

Empirical evidence in full-precision models has shown that increasing network depth improves performance.
The characteristic of the binarized model allows for maintaining significantly low computational and memory usage even with increased network depth.
However, in previous research, Xia et al. [] observed that increasing the number of binarized convolutional layers within the binarized convolutional unit leads to a significant decrease in binarized model performance, the performance degradation issue with the increase in binarized convolutional layer depth within each unit restricts the further application of the binarized model.
To address this issue, we propose the Binarized Depth Convolution (BDC) unit, which aims to enhance the binarized model performance by deepening the layers of the binarized convolution unit rather than reducing performance.

Cai et al. [] proposed the binarized convolution unit BiSR-Conv, which can adjust the density and enable effective binarization of convolutional layers.
We utilize BiSR-Conv to binarize FlashOcc [], forming our initial version of BDC-V0, with its structure shown in Figure 3 (a).
Please refer to the supplementary materials for a more detailed description of the BDC-V0.
The model achieves a performance of 34.51 mIoU.

Theorem 1.
In the process of backpropagation, we denote the expected value of the element-wise absolute gradient error of the parameters w in the $l$-th binarized convolutional layer as $\mathbb{E}[\Delta\frac{\partial L}{\partial w_{mn}^{(l)}}]$.
The specific expression is as follows:

$$ $\mathbb{E}[\Delta\frac{\partial L}{\partial w_{mn}^{(l)}}]\approx 0.5354\cdot( \sum_{i}\sum_{j}\sum_{m^{\prime}=-(k//2)}^{k//2}\sum_{n^{\prime}=-(k//2)}^{k// 2}\mathbb{E}[|\frac{\partial\sigma(y_{(i+m^{\prime})(j+n^{\prime})}^{(l)})}{ \partial y_{ij}^{(l)}}\cdot w_{m^{\prime}n^{\prime}}^{(l+1)}\cdot\frac{ \partial L}{\partial y_{ij}^{(l+1)}}|])$ (2) $$

where $k$ is the binarized convolution kernel size, $\frac{\partial\sigma(y_{(i+m^{\prime})(j+n^{\prime})}^{(l)})}{\partial y_{ij}^
{(l)}}$ is the derivative of the activation function $\sigma(\cdot)$, $w_{m^{\prime}n^{\prime}}^{(l+1)}$ represents the weights of the binarized convolutional kernel in the next layer, and $\frac{\partial L}{\partial y_{ij}^{(l+1)}}$ is the element-wise gradient in the next layer.

Based on Theorem 1, using a $3\times 3$ convolutional kernel for binarized convolution leads to more binarization errors than a $1\times 1$ kernel.
Additionally, the model necessitates the presence of the first $3\times 3$ binarized convolution layer to maintain its capability for extracting local features.
Therefore, building upon the binarized convolution unit BDC-V0, we introduce a $1\times 1$ binarized convolution layer after the $3\times 3$ binarized convolution and before the residual connection, proposing BDC-V1 as shown in Figure 3(b).
By deepening the binarized convolution unit, BDC-V1 enhances its feature extraction capability, achieving a performance of 36.29 mIoU.

To further deepen the model, we add multiple $1\times 1$ binarized convolution layers to the binarized convolution unit named BDC-V2.
The structure of BDC-V2 is shown in Figure 3(c).
We define the added multi-layer binarized convolution as $\text{MulBiconv}_{N}$, comprising $N$ RPReLU activations and $1\times 1$ binarized convolution layers, which can be expressed as

$$ $\text{MulBiconv}_{N}(\cdot)=\text{Repeat}_{N}(\text{Biconv1}\times\text{1}( \text{RPReLU}(\cdot)))$ (3) $$

where $\text{Repeat}_{N}(f)$ denotes repeating $N$ times operation $f$.

When $N=1$, the performance drops to 35.88 mIoU; $N=2$, it drops further to 35.43 mIoU.
We observe a decreasing trend in network performance as the number of $1\times 1$ binarized convolution layers increases.
It occurs as the accumulated binarization errors increase with the addition of more binarized convolution layers within the unit.
The negative effect on binarized model performance outweighs the positive influence of increasing parameters, resulting in decreased binarized model performance.

In BNN, not every channel feature contributes to the effectiveness of the binarized model.
Insignificant channel features carry binarization errors that contaminate important channel features, leading to performance deterioration in the binarized model.
The Squeeze-and-Excitation (SE) module proposed by SENet [] explicitly models interdependencies among feature channels, effectively extracting highly significant features.

Therefore, based on the idea of SENet [], we propose BDC-V3, where we use the additional $1\times 1$ binarized convolution to learn per-channel weights rather than features themselves.
The structure of BDC-V3 is illustrated in Figure 3(d).
First, the output of the first $1\times 1$ binarized convolution, $\textbf{X}_{1}$, serves as the input to the per-channel weight branch, which includes global average pooling (AvgPool), multi-layer binarized convolution (MulBiconv), and Sigmoid.
The branch output $\textbf{Y}_{1}$ is obtained by multiplying it with $\textbf{X}_{1}$, expressed as

$$ $\textbf{Y}_{1}=\text{Sigmoid}(\text{MulBiconv}_{N}(\text{AvgPool}(\textbf{X}_{ 1})))\odot\textbf{X}_{1}$ (4) $$

where $\odot$ denotes element-wise multiplication.
When $N=1$, the performance improved to 36.89 mIoU; when $N=2$, the performance further increased to 37.20 mIoU, which is closer to 37.84 mIoU, the performance upper bound offered by the full-precision baseline model.
We chose BDC-V3 with $N=2$ as the final binarized convolutional unit, named BDConv.

Figure: Figure 3: The illustration of the improvement process of our BDC.
Refer to caption: x2.png

### 3.3 Binarized Convolution Module

Cai et al. [] demonstrated the necessity of maintaining consistency in input and output dimensions for binarized convolutional layers to ensure the propagation of full-precision residual information.
Consequently, specialized design considerations are necessary for each binarized convolution module.
We can decompose the CNN-based occupancy network into four types of convolution modules:

(1) Basic convolution module: Input $\textbf{X}\in\mathbb{R}^{C\times H\times W}$, output $\textbf{Y}\in\mathbb{R}^{C\times H\times W}$;

(2) Down-sampling convolution module: Input $\textbf{X}\in\mathbb{R}^{C\times H\times W}$, output $\textbf{Y}\in\mathbb{R}^{2C\times\frac{H}{2}\times\frac{W}{2}}$;

(3) Up-sampling convolution module: Input $\textbf{X}\in\mathbb{R}^{C\times H\times W}$, output $\textbf{Y}\in\mathbb{R}^{C\times 2H\times 2W}$;

(4) Channel reduction convolution module: Input $\textbf{X}\in\mathbb{R}^{C\times H\times W}$, output $\textbf{Y}\in\mathbb{R}^{\frac{C}{2}\times H\times W}$;

Figure: (a) Basic
Refer to caption: x3.png

We adopt a binarized design approach for these four convolution modules, leveraging methodologies from previous works [; ; ], as illustrated in Figure 4.
Figure 4 (a) illustrates the basic convolutional module, preserving both the size and the number of channels in the input feature map.
Figure 4 (b) depicts the downsample convolution module, reducing the size of the input feature map by half and doubling the number of channels.
Figure 4 (c) showcases the upsample convolution module, doubling the size of the input feature map while preserving the number of channels.
Finally, Figure 4 (d) presents the channel reduction convolution module, maintaining the size of the input feature map while halving the number of channels.

## 4 Experiment

### 4.1 Experimental Settings

Datasets.
We use the Occ3D-nuScenes dataset [], which comprises 28,130 samples for training and 6,019 samples for validation.

Evaluation Metrics.
We evaluate the Occ3D-nuScenes’ validation set using the mean Intersection over Union (mIoU) metric.
Similar to [], we compute the operations per second of BNN (OPs^b) as $\text{OPs}^{b}=\text{OPs}^{f}/64$ to measure the computational complexity, where OPs^f represents FLOPS.
To calculate the parameters of BNN, use the formula $\text{Parms}^{b}=\text{Parms}^{f}/32$, where the superscript $b$ and $f$ refer to the binarized and full-precision models, respectively.
To calculate the total computational and memory costs, sum OPs as OPs^b + OPs^f and Params as Params^b + Params^f.

Implementation Details.
For 3D occupancy prediction tasks, we employ FlashOcc [] as the baseline network.
We utilized ResNet50 [] as the image backbone, with an input size of $256\times 704$.
Default learning rate $1\times 10^{-4}$, AdamW [] optimizer, and weight decay of $1\times 10^{-2}$ were utilized.
The training lasted approximately 29 hours, utilizing 24 epochs on two NVIDIA 3090 GPUs, with a batch size of 2 per GPU.
Data augmentation strategies for the Occ3D-nuScenes dataset remained consistent with those of FlashOcc [].
Previous works, such as FlashOcc and BEVDet-Occ [], have demonstrated the effectiveness of camera visibility masks during training.
Therefore, we also employ camera visibility masks to enhance performance.
Following the settings of FlashOcc, we employ the pre-trained model from BEVDet [] for 3D object detection tasks as our pre-training model.

### 4.2 Main Results

**Table 1: Occupancy Prediction performance (mIoU$\uparrow$) on the Occ3D-nuScenes datasets. Best and second best performance among BNNs are in red and blue colors, respectively.**
| Methods | Params(M) | OPs(G) | $\blacksquare$ others | $\blacksquare$ barrier | $\blacksquare$ bicycle | $\blacksquare$ bus | $\blacksquare$ car | $\blacksquare$ const. veh. | $\blacksquare$ motorcycle | $\blacksquare$ pedestrian | $\blacksquare$ traffic cone | $\blacksquare$ trailer | $\blacksquare$ truck | $\blacksquare$ drive. suf. | $\blacksquare$ other flat | $\blacksquare$ sidewalk | $\blacksquare$ terrain | $\blacksquare$ manmade | $\blacksquare$ vegetation | mIoU |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| *CNN-based (32 bit)* |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| BEVDet-Occ [] | 29.02 | 241.76 | 8.22 | 44.21 | 10.34 | 42.08 | 49.63 | 23.37 | 17.41 | 21.49 | 19.70 | 31.33 | 37.09 | 80.13 | 37.37 | 50.41 | 54.29 | 45.56 | 39.59 | 36.01 |
| FlashOcc [] | 44.74 | 248.57 | 9.08 | 46.32 | 17.71 | 42.70 | 50.64 | 23.72 | 20.13 | 22.34 | 24.09 | 30.26 | 37.39 | 81.68 | 40.13 | 52.34 | 56.46 | 47.69 | 40.60 | 37.84 |
| *BNN-based (1 bit)* |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| ReaActNet-T [] | 26.80 | 129.74 | 7.55 | 38.87 | 16.64 | 35.78 | 44.27 | 20.34 | 15.53 | 16.16 | 18.70 | 24.42 | 33.59 | 73.64 | 29.05 | 39.80 | 41.27 | 39.31 | 34.00 | 31.29 |
| ReaActNet-B [] | 28.17 | 133.89 | 8.62 | 40.92 | 15.94 | 37.45 | 47.23 | 18.57 | 17.47 | 18.91 | 21.52 | 23.14 | 33.13 | 77.20 | 34.58 | 45.48 | 48.31 | 42.95 | 35.06 | 33.32 |
| BBCU-T [] | 26.79 | 129.69 | 6.24 | 38.16 | 14.33 | 31.95 | 43.18 | 20.57 | 16.50 | 17.39 | 13.45 | 22.26 | 32.51 | 75.69 | 32.97 | 42.46 | 48.50 | 41.68 | 35.75 | 31.39 |
| BBCU-B [] | 28.16 | 133.84 | 7.61 | 41.14 | 13.64 | 35.54 | 46.55 | 20.86 | 17.44 | 19.87 | 17.58 | 24.24 | 33.94 | 76.19 | 34.05 | 44.61 | 48.08 | 42.67 | 35.28 | 32.27 |
| BiMatting-T [] | 26.82 | 129.95 | 5.96 | 38.17 | 15.27 | 35.85 | 44.11 | 19.35 | 14.38 | 18.98 | 15.84 | 23.22 | 31.16 | 73.97 | 30.51 | 35.42 | 40.9 | 41.65 | 35.05 | 30.58 |
| BiMatting-B [] | 28.17 | 134.05 | 6.80 | 38.65 | 17.99 | 33.02 | 43.80 | 19.91 | 18.29 | 18.67 | 19.82 | 21.83 | 32.09 | 72.99 | 32.44 | 41.23 | 43.64 | 36.24 | 35.07 | 31.32 |
| BiSRNet-T [] | 26.79 | 129.70 | 8.38 | 41.06 | 16.76 | 33.94 | 46.11 | 18.96 | 19.10 | 17.90 | 16.94 | 23.70 | 35.14 | 76.86 | 35.68 | 46.77 | 50.39 | 41.41 | 34.78 | 33.17 |
| BiSRNet-B [] | 28.16 | 133.85 | 9.27 | 41.94 | 19.53 | 37.33 | 47.48 | 20.83 | 19.17 | 20.08 | 20.21 | 25.36 | 33.99 | 77.42 | 35.78 | 47.35 | 50.58 | 43.24 | 37.20 | 34.51 |
| BDC-T (Ours) | 26.82 | 129.90 | 9.22 | 44.81 | 17.56 | 40.02 | 49.94 | 24.84 | 18.63 | 21.32 | 23.27 | 32.46 | 36.43 | 81.03 | 38.74 | 51.40 | 55.55 | 47.15 | 40.76 | 37.24 |
| BDC-B (Ours) | 28.19 | 134.02 | 9.24 | 43.93 | 20.25 | 39.92 | 49.90 | 23.11 | 21.44 | 21.22 | 21.98 | 31.23 | 37.09 | 80.93 | 39.41 | 51.03 | 54.88 | 46.80 | 40.04 | 37.20 |

**Table 2: 3D Object Detection performance (mAP$\uparrow$, NDS$\uparrow$) on the nuScenes val set. Best performance among BNNs are in bold.**
| Methods | Params(M) | OPs(G) | mAP$\uparrow$ | NDS$\uparrow$ | mATE$\downarrow$ | mASE$\downarrow$ | mAOE$\downarrow$ | mAVE$\downarrow$ | mAAE$\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| *CNN-based (32 bit)* |  |  |  |  |  |  |  |  |  |
| BEVDet [] | 44.25 | 148.77 | 0.3836 | 0.4995 | 0.5815 | 0.2790 | 0.4750 | 0.3807 | 0.2067 |
| *BNN-based (1 bit)* |  |  |  |  |  |  |  |  |  |
| ReactNet-T [] | 26.53 | 101.30 | 0.3222 | 0.4358 | 0.6609 | 0.3057 | 0.6298 | 0.4468 | 0.2100 |
| BBCU-T [] | 26.51 | 101.24 | 0.3166 | 0.4046 | 0.6697 | 0.3137 | 0.7822 | 0.5461 | 0.2255 |
| BiMatting-T [] | 26.55 | 101.41 | 0.3356 | 0.4428 | 0.6358 | 0.2968 | 0.6527 | 0.4485 | 0.2159 |
| BiSRNet-T [] | 26.52 | 101.25 | 0.3431 | 0.4519 | 0.6633 | 0.2940 | 0.5777 | 0.4550 | 0.2061 |
| BDC-T | 26.54 | 101.36 | 0.3598 | 0.4686 | 0.6362 | 0.2882 | 0.5388 | 0.4468 | 0.2030 |

To ensure performance, we refrain from binarizing the image backbone in the image encoder.
This component contains pre-trained weights from image classification tasks, effectively facilitating model convergence and incorporating prior semantic information from images.
We binarize the BEV encoder and occupancy head as the base version (-B) for all binarized models.
We further binarize the image neck in the image encoder to obtain the tiny version (-T) based on the base version.

Table 1 presents the evaluation results of our method BDC on the validation set of Occ3D-nuScenes.
To validate the effectiveness of our proposed method BDC, we compare it with other state-of-the-art binarized models, including ReActNet [], BBCU [], BiMatting [], and BiSRNet [].
We also compare it with full-precision occupancy prediction networks based on CNN architectures, including BEVDet-Occ [] and FlashOcc [], where FlashOcc serves as the baseline network for all binarized models and represents the theoretical upper limit of binarized model performance.

Table 1 presents performance metrics (mIoU), parameter counts, and the number of operations for different methods.
Compared to other binarized methods, our BDC-T and BDC-B achieve the best or second-best results across almost all binarized models.
Specifically, BDC significantly improves performance without increasing parameter count or computational complexity.
Compared to the previous state-of-the-art method, BiSRNet-B, our BDC-T demonstrates superior performance in mIoU, exceeding it by 2.73 mIoU (+7.91%), while saving 2.95% of operations and 4.76% of parameters of BiSRNet-B.
Moreover, BDC-T achieves competitive results compared to the full-precision model FlashOcc,
using only 52.26% of operations and 59.95% of parameters, with a minimal performance loss of -0.6 mIoU (-1.59%) due to binarization errors.
Both BBCU and BiSRNet exhibit performance degradation issues when binarizing additional modules.
Compared to BDC-B, BDC-T performs slightly better when binarizing image neck modules.
It demonstrates the robustness of BDC to the binarized modules.

To validate the generalizability of the proposed BDC, we also conduct experiments on 3D object detection tasks using the nuScenes [] dataset.
Table 2 presents performance metrics for the 3D object detection task in nuScenes, where our approach, BDC, continues to demonstrate superior performance in both mAP and NDS, two crucial indicators.

### 4.3 Ablation Study

In all ablation studies, we binarize the BEV encoder and the occupancy head of the full-precision model.
This binarization setting is configured as the base version (-B) for all binarized models as described in Table 1.

**Table 3: Break-down ablation. Figure 3 illustrates the structure of various versions of the BDC.**
| Methods | mIoU | OPs (G) | Params (M) |
| --- | --- | --- | --- |
| BDC-V0 | 34.51 | 133.85 | 28.16 |
| BDC-V1 | 36.29 | 133.93 | 28.17 |
| BDC-V2 | 35.43 | 134.10 | 28.19 |
| BDC-V3 | 37.20 | 134.02 | 28.19 |

Figure: Figure 5: Ablation study of multi-layer binarized convolution (MulBiconv)
Refer to caption: extracted/5623170/figure/conv_layer.png

Multi-layer Binarized Convolution (MulBiconv) Ablation.
To explore the impact of the number of binarized convolutional layers in MulBiconv on binarized model performance, we binarize FlashOcc using both BDC-V2 and BDC-V3 while varying the number of binarized convolutional layers in MulBiconv ($N=0,1,2,3,4$).

The results are illustrated in Figure 5.
When $N=0$, the structure of BDC-V2 is identical to that of BDC-V1.
At this point, MulBiconv in BDC-V3 contains no learnable parameters with the per-channel weight branch.
As $N$ increases, we observe a gradual decline followed by fluctuations in the performance of BDC-V2.
In contrast, BDC-V3 initially shows performance improvement followed by fluctuations as $N$ increases.
Compared to BDC-V2, BDC-V3 exhibits a slight performance improvement.
When MulBiconv selects $N=2$, BDC-V3 achieves the best performance, reaching 37.20 mIoU.
At this point, the difference between the gain from model parameter reduction and the loss from binarization errors reaches its maximum, achieving the optimal trade-off.

Break-down Ablation.
We binarize FlashOcc using four variants of BDC, where BDC-v0 represents the binarized method BiSRNet.
Additionally, for BDC-V2 and BDC-V3 utilizing the multi-layer binarized convolution (MulBiconv), we set $N=2$.

The results are presented in Table 4, from which we can draw the following conclusions:
(1) Compared to BDC-V0, BDC-V1 achieves a significant gain of 1.78 mIoU (+5.16%) by adding only one $1\times 1$ binarized convolution layer to each binarized convolution unit.
Due to the nature of binarized convolutions, extra binarized convolution layers result in minimal changes to model parameters and computational complexity.
(2) By adding multi-layer binarized convolution (MulBiconv) to each binarized convolution unit in BDC-V2 compared to BDC-V1, we observe a substantial decrease in performance, along with slight increases in parameter count and computational complexity.
(3) Compared to BDC-V2, BDC-V3 exhibits a significant performance improvement of 1.77 mIoU.
Additionally, BDC-V3 gains an extra 0.91 mIoU over BDC-V1.
Placing additional binarized convolutional layers within the per-channel weight branch effectively enhances model performance.

Kernel Size Ablation.
To validate whether $3\times 3$ binarized convolutions incur more binarization errors than $1\times 1$ ones, potentially leading to performance degradation, we apply BDC-V1 and BDC-V2 ($N=1$) to FlashOcc.
We present the results in Table 4.
For BDC-V1, replacing the $1\times 1$ binarized convolution with consecutive $3\times 3$ binarized convolutions led to a decrease in performance from 36.29 mIoU to 33.01 mIoU, with no changes in parameter count or computational complexity.

Additionally, we validate the necessity of using a $3\times 3$ binarized convolution as the first convolution layer.
If replaced with a $1\times 1$ binarized convolution, the receptive field of the binarized convolution unit becomes limited, preventing the establishment of connections with neighboring pixel features, resulting in a decrease in performance from 36.29 mIoU to 35.32 mIoU.
Experiments conduct on BDC-V2 ($N=1$) also support the conclusion that consecutive $3\times 3$ binarized convolutions lead to binarization errors and affect binarized model performance.

### 4.4 Visualization

Figure: Figure 6: Visualization rensults on Occ3D-nuScenes validation set
Refer to caption: x7.png

We also present some qualitative results of BDC and BiSRNet on the Occ3D-nuScenes’ validation set.
As illustrated in Figure 6, BDC exhibits comprehensive predictions about the bus in the first and last rows.
In the second row, BDC successfully identifies all pedestrians, whereas BiSRNet [] overlooks some pedestrians in the scene.
Moreover, in the third row, BDC provides accurate predictions about curbs, whereas BiSRNet misclassifies them as drivable surfaces, potentially posing safety concerns.
Additionally, in the fourth row, BDC accurately reconstructs traffic lights in the scene, showcasing its robust capability in scene perception.

## 5 Conclusion

This paper introduces a binarized deep convolution (BDC) unit for binarizing 3D occupancy networks.
The BDC unit addresses the issue observed in BNN, where increasing the number of binarized convolutional layers leads to a decrease in model performance.
Theoretical analysis demonstrates that $1\times 1$ binarized convolutions introduce minimal binarization errors during training.
Therefore, within the BDC unit, apart from the initial binarized convolution being a $3\times 3$ kernel, all others are $1\times 1$ binarized convolutions.
Furthermore, BDC employs the per-channel weight branching approach to effectively mitigate the impact of binarization errors from unimportant channel features on the performance of binarized models, thereby enhancing performance while increasing the number of binarized convolutional layers.
Furthermore, extensive experiments validate the effectiveness of the BDC unit.
Our proposed method significantly outperforms existing state-of-the-art binarized convolution networks and closely approaches the performance of full-precision networks.

Limitation.
We have not tested our method for performance in Transformer architectures, which may limit its broader application.

## 6 Appendix

### 6.1 More Details About Base Model

Base model consists of an image encoder $\mathcal{E}_{2D}$, a view transformer module $\mathcal{T}$, a BEV encoder $\mathcal{E}_{BEV}$, and an occupancy head $\mathcal{H}$.
The occupancy prediction network is composed of these modules concatenated sequentially.
Assuming the input images are $\textbf{I}\in\mathbb{R}^{N_{view}\times 3\times H\times W}$, the occupancy prediction output $\textbf{O}\in\mathbb{R}^{X\times Y\times Z}$ can be formulated as

$$ $\textbf{O}=\mathcal{H}(\mathcal{E}_{BEV}(\mathcal{T}(\mathcal{E}_{2D}(\textbf{ I}))))$ (5) $$

where $H$ and $W$ represent the height and width of the input images, and $X$, $Y$, and $Z$ denote the length, width, and height of the 3D space, respectively, $N_{view}$ represents the number of multi-view cameras.

First, Multi-view images are sent to the image encoder $\mathcal{E}_{2D}$ to obtain 2D features $\textbf{f}_{2D}\in\mathbb{R}^{N_{\text{view}}\times C_{2D}\times H_{2D}\times W
_{2D}}$ and depth prediction $\textbf{f}_{\text{depth}}\in\mathbb{R}^{N_{\text{view}}\times N_{\text{depth}}
\times H_{2D}\times W_{2D}}$, where $C_{2D},H_{2D},W_{2D}$ denote the number of channels, height and width of 2D features, respectively.
$N_{depth}$ represents the number of depth bins in the depth prediction.

Subsequently, the image features $\textbf{f}_{2D}$ and depth prediction $\textbf{f}_{depth}$ are passed through the visual transformation module $\mathcal{T}$, which transforms them into primary BEV features $\textbf{f}_{T}\in\mathbb{R}^{C_{BEV}\times H_{BEV}\times W_{BEV}}$ using camera intrinsic and extrinsic projection matrices.
Here, $C_{BEV}$ represents the number of channels of BEV features, while $H_{BEV}$ and $W_{BEV}$ represent the length and width of the BEV space, respectively.
Since the voxel distribution obtained from the depth map through projection matrices is sparse, the representation capability of primary BEV features may be insufficient.
To this end, $\textbf{f}_{T}$ is passed through the BEV encoder $\mathcal{E}_{BEV3D}$ to obtain fine BEV features $\textbf{f}_{BEV}\in\mathbb{R}^{C_{BEV}\times H_{BEV}\times W_{BEV}}$ for further refinement.

Finally, the semantic prediction output logits $\textbf{O}_{logits}\in\mathbb{R}^{N_{class}\times X\times Y\times Z}$ come from the BEV features $\textbf{f}_{BEV}$ processed through the occupancy prediction head $\mathcal{H}$, where $N_{class}$ is the number of semantic classes in the dataset.
By taking the index corresponding to the maximum value of the logits, we can obtain the final occupancy prediction output O.

### 6.2 More Details About BDC-V0

We define BDC-V0 following the method proposed in BiSRNet [].
Both full-precision image features and Bird’s Eye View (BEV) features, represented as $\textbf{X}_{f}\in\mathbb{R}^{C\times H\times W}$, serve as input for the full-precision activations.

In 3D occupancy networks, features transform from dense 2D space to sparse 3D space and then back to dense 3D space, causing significant differences in feature distribution.
Each module has distinct densities and distributions.

To address the problem of significant differences in feature distribution, we follow the approach of BiSRNet, employing channel-wise feature redistribution:

$$ $\textbf{X}_{r}=k\cdot\textbf{X}_{f}+b$ (6) $$

Here, $\textbf{X}_{r}\in\mathbb{R}^{C\times H\times W}$ represents the activations after channel-wise feature redistribution, and $k,b\in\mathbb{R}^{C}$ are learnable parameters.
$k$ represents the learnable density of redistribution, while $b$ represents the learnable bias of redistribution.

Next, $\textbf{X}_{r}$ is passed through the Sign function to binarize it, yielding 1-bit binarized activations $\textbf{X}_{b}\in\mathbb{R}^{C\times H\times W}$, as follows:

$$ $x_{b}=\text{Sign}(x_{r})=\begin{cases}+1,&\text{if }x_{r}>0\\ -1,&\text{if }x_{r}\leq 0\end{cases}$ (7) $$

where $x_{r}\in\textbf{X}_{r}$, $x_{b}\in\textbf{X}_{b}$.

Since the Sign function is not differentiable, approximation functions are required to ensure successful backpropagation.
Common approximation functions include piecewise linear function $\text{Clip}(\cdot)$, piecewise quadratic function $\text{Quad}(\cdot)$, and hyperbolic tangent function $\text{Tanh}(\cdot)$. We use the hyperbolic tangent function as the approximation function, defined as:

$$ $x_{b}=\text{Tanh}(\alpha x_{r})=\frac{e^{\alpha x_{r}}-e^{-\alpha x_{r}}}{e^{ \alpha x_{r}}+e^{-\alpha x_{r}}}$ (8) $$

The Tanh function ensures gradients exist even when weights and activations exceed 1, allowing parameter updates downstream during backpropagation.

In the binarized convolutional layer, the 32-bit precision weights $\textbf{W}_{f}$ are binarized into 1-bit binarized weights $\textbf{W}_{b}$ according to the following formula:

$$ $w_{b}=\mathbb{E}_{w_{f}\in\textbf{W}_{f}}(|w_{f}|)\cdot\text{Sign}(w_{f})$ (9) $$

Here, $\mathbb{E}_{w_{f}\in\textbf{W}_{f}}(|w_{f}|)$ represents the average absolute value of the full-precision weights, which serves as a scaling factor to reduce the discrepancy between the binarized weights $\textbf{W}_{b}$ and the full-precision weights $\textbf{W}_{f}$.
Multiplying this value by $\text{Sign}(w_{f})=\pm 1$ yields element-wise binarized weights $w_{b}$.

Figure: Figure 7: The schematic diagram of binarized convolution [].
Refer to caption: x8.png

Subsequently, the binarized activation $\textbf{X}_{b}$ is convolved with the binarized weights $\textbf{W}_{b}$.
Binarized convolution can be accomplished purely through logical operations.
The schematic diagram of binarized convolution [] is illustrated in Figure 7, and the expression is as follows:

$$ $\textbf{Y}_{b}=\text{Biconv}(\textbf{X}_{b},\textbf{W}_{b})=\text{BitCount}( \text{XNOR}(\textbf{X}_{b},\textbf{W}_{b}))$ (10) $$

Here, $\textbf{Y}_{b}$ is the output of binarized convolution, Biconv denotes the binarized convolution layer, and BitCount and XNOR represent the bit count and logical XOR operations, respectively.
In BDC-V0, the convolutional kernel size is $3\times 3$.

For the activation function, we utilize RPReLU, whose expression is defined as follows:

$$ $\text{RPReLU}(y_{i})=\begin{cases}y_{i}-\gamma_{i}+\zeta_{i},&\text{if }y_{i}> \gamma_{i}\\ \beta_{i}\cdot(y_{i}-\gamma_{i})+\zeta_{i},&\text{if }y_{i}\leq\gamma_{i}\end{cases}$ (11) $$

Here, $y_{i}\in\mathbb{R}$ represents the $i$-th element value of $\textbf{Y}_{b}$, and $\beta_{i}$, $\gamma_{i}$, and $\zeta_{i}$ are learnable parameters for the $i$-th channel.

### 6.3 Proof of Theorem 1

Theorem 1.
In the process of backpropagation, we denote the expected value of the element-wise absolute gradient error of the parameters w in the $l$-th binarized convolutional layer as $\mathbb{E}[\Delta\frac{\partial L}{\partial w_{mn}^{(l)}}]$.
The specific expression is as follows:

$$ $\mathbb{E}[\Delta\frac{\partial L}{\partial w_{mn}^{(l)}}]\approx 0.5354\cdot( \sum_{i}\sum_{j}\sum_{m^{\prime}=-(k//2)}^{k//2}\sum_{n^{\prime}=-(k//2)}^{k// 2}\mathbb{E}[|\frac{\partial\sigma(y_{(i+m^{\prime})(j+n^{\prime})}^{(l)})}{ \partial y_{ij}^{(l)}}\cdot w_{m^{\prime}n^{\prime}}^{(l+1)}\cdot\frac{ \partial L}{\partial y_{ij}^{(l+1)}}|])$ (12) $$

where $k$ is the binarized convolution kernel size, $\frac{\partial\sigma(y_{(i+m^{\prime})(j+n^{\prime})}^{(l)})}{\partial y_{ij}^
{(l)}}$ is the derivative of the activation function $\sigma(\cdot)$, $w_{m^{\prime}n^{\prime}}^{(l+1)}$ represents the weights of the binarized convolutional kernel in the next layer, and $\frac{\partial L}{\partial y_{ij}^{(l+1)}}$ is the element-wise gradient in the next layer.

Proof.
We assume the element of the input of a binarized convolutional layer as $x_{ij}$, with a binarization error denoted as $\epsilon_{ij}$, the full-precision input before binarization as $\hat{x}_{ij}$, and the output of the binarized convolutional layer as $y_{ij}$.
Thus, we have:

$$ $x_{ij}=\hat{x}_{ij}+\epsilon_{ij}$ (13) $$

Since the full-precision input $\hat{x}_{ij}$ at the current layer is the output from the batch normalization layer in the previous layer, we can assume that the full-precision input $\hat{x}_{ij}$ follows a Gaussian distribution $\mathcal{N}(0,1)$.
Based on Equations (7) and (13), we can then derive the distribution of $\epsilon_{ij}$ as follows:

$$ $|\epsilon_{ij}|=|\hat{x}_{ij}-x_{ij}|=|\hat{x}_{ij}-\text{Sign}(\hat{x}_{ij})| =\begin{cases}|\hat{x}_{ij}-1|,&\text{if }\hat{x}_{ij}>0\\ |\hat{x}_{ij}+1|,&\text{if }\hat{x}_{ij}\leq 0\end{cases}$ (14) $$

Assuming the convolution kernel size $k$ is odd, for a $k\times k$ convolutional layer, the kernel weight $w_{mn}$, and the kernel bias is $b_{mn}$.
The forward propagation equation is given by:

$$ $y_{ij}=\sum_{m=-(k//2)}^{k//2}\sum_{n=-(k//2)}^{k//2}(x_{(i+m)(j+n)}\cdot w_{ mn}+b_{mn})$ (15) $$

Assuming that during backpropagation, the gradient at current layer $l$ is given by $\frac{\partial L}{\partial y_{ij}^{(l)}}$, we can use the chain rule to derive the gradient for a $k\times k$ convolutional layer as follows:

$$ $\displaystyle\frac{\partial L}{\partial w_{mn}^{(l)}}$ $\displaystyle=\sum_{i}\sum_{j}x_{(i+m)(j+n)}^{(l)}\frac{\partial L}{\partial y _{ij}^{(l)}}=\sum_{i}\sum_{j}(\hat{x}_{(i+m)(j+n)}^{(l)}+\epsilon_{(i+m)(j+n)} ^{(l)})\cdot\frac{\partial L}{\partial y_{ij}^{(l)}}$ (16) $$

Given that the output of the current layer $y_{ij}^{(l)}$ becomes the input of the next layer after passing through the activation function $\sigma(\cdot)$.
Based on Equation (15), we can derive:

$$ $\displaystyle y_{ij}^{(l+1)}$ $\displaystyle=\sum_{m^{\prime}=-(k//2)}^{k//2}\sum_{n^{\prime}=-(k//2)}^{k//2} \sigma(y_{(i+m^{\prime})(j+n^{\prime})}^{(l)})\cdot w_{m^{\prime}n^{\prime}}^{ (l+1)}+b_{m^{\prime}n^{\prime}}^{(l+1)}$ (17) $$

We can obtain the gradient relationship between $\frac{\partial L}{\partial y_{ij}^{(l)}}$ and $\frac{\partial L}{\partial y_{ij}^{(l+1)}}$:

$$ $\displaystyle\frac{\partial L}{\partial y_{ij}^{(l)}}$ $\displaystyle=\sum_{m^{\prime}=-(k//2)}^{k//2}\sum_{n^{\prime}=-(k//2)}^{k//2} \frac{\partial\sigma(y_{(i+m^{\prime})(j+n^{\prime})}^{(l)})}{\partial y_{ij}^ {(l)}}\cdot w_{m^{\prime}n^{\prime}}^{(l+1)}\cdot\frac{\partial L}{\partial y_ {ij}^{(l+1)}}$ (18) $$

By substituting Equation (18) into Equation (16), we can obtain:

$$ $\displaystyle\frac{\partial L}{\partial w_{mn}^{(l)}}$ $\displaystyle=\sum_{i}\sum_{j}\sum_{m^{\prime}}\sum_{n^{\prime}}(\hat{x}_{(i+m )(j+n)}^{(l)}+\epsilon_{(i+m)(j+n)}^{(l)})\cdot\frac{\partial\sigma(y_{(i+m^{ \prime})(j+n^{\prime})}^{(l)})}{\partial y_{ij}^{(l)}}\cdot w_{m^{\prime}n^{ \prime}}^{(l+1)}\cdot\frac{\partial L}{\partial y_{ij}^{(l+1)}}$ (19) $$

We can derive the additional gradient error $\Delta\frac{\partial L}{\partial w_{mn}^{(l)}}$ induced by the binarization error $\epsilon$ as follows:

$$ $\displaystyle\Delta\frac{\partial L}{\partial w_{mn}^{(l)}}$ $\displaystyle:=\text{EAE}(\frac{\partial L}{\partial w_{mn}^{(l)}}-\frac{ \partial L}{\partial w_{mn}^{(l)}}|_{\epsilon=0})$ (20) $\displaystyle=\sum_{i}\sum_{j}\sum_{m^{\prime}}\sum_{n^{\prime}}|\epsilon_{(i+ m)(j+n)}^{(l)}\cdot\frac{\partial\sigma(y_{(i+m^{\prime})(j+n^{\prime})}^{(l)} )}{\partial y_{ij}^{(l)}}\cdot w_{m^{\prime}n^{\prime}}^{(l+1)}\cdot\frac{ \partial L}{\partial y_{ij}^{(l+1)}}|$ $$

Here, EAE represents the element-wise absolute error.

By utilizing Equation (14), we can calculate the expected value of the absolute binarization error, denoted as $\mathbb{E}[|\epsilon_{ij}|]$:

$$ $\displaystyle\mathbb{E}[|\epsilon_{ij}|]$ $\displaystyle=\int_{0}^{\infty}|\hat{x}_{ij}-1|\frac{1}{\sqrt{2\pi}}e^{-\frac{ \hat{x}_{ij}^{2}}{2}}\,d\hat{x}_{ij}+\int_{-\infty}^{0}|\hat{x}_{ij}+1|\frac{1 }{\sqrt{2\pi}}e^{-\frac{\hat{x}_{ij}^{2}}{2}}\,d\hat{x}_{ij}$ (21) $\displaystyle=2(\int_{0}^{1}\frac{1-\hat{x}_{ij}}{\sqrt{2\pi}}e^{-\frac{\hat{x }_{ij}^{2}}{2}}\,d\hat{x}_{ij}-\int_{1}^{\infty}\frac{1-\hat{x}_{ij}}{\sqrt{2 \pi}}e^{-\frac{\hat{x}_{ij}^{2}}{2}}\,d\hat{x}_{ij})$ $$

The Gaussian error function, often abbreviated as "$erf(x)$" is defined as follows:

$$ $erf(x)=\frac{2}{\sqrt{\pi}}\int_{0}^{x}e^{-t^{2}}\,dt$ (22) $$

Based on the definition of the Gaussian error function and the use of the substitution rule, we can compute the integral as follows:

$$ $\displaystyle\int_{0}^{x}e^{-\frac{u^{2}}{2}}\,du$ $\displaystyle\xlongequal{u=\sqrt{2}t}\sqrt{2}\int_{0}^{\frac{x}{\sqrt{2}}}e^{- t^{2}}\,dt$ (23) $\displaystyle=\frac{\sqrt{\pi}}{\sqrt{2}}\frac{2}{\sqrt{\pi}}\int_{0}^{\frac{x }{\sqrt{2}}}e^{-t^{2}}\,dt$ $\displaystyle=\frac{\sqrt{\pi}}{\sqrt{2}}erf(\frac{x}{\sqrt{2}})$ $$

We can continue the computation of the integral further.

$$ $\displaystyle\int_{a}^{b}\frac{1-x}{\sqrt{2\pi}}e^{-\frac{x^{2}}{2}}\,dx$ $\displaystyle=\int_{a}^{b}\frac{1}{\sqrt{2\pi}}e^{-\frac{x^{2}}{2}}\,dx-\int_{ a}^{b}\frac{x}{\sqrt{2\pi}}e^{-\frac{x^{2}}{2}}\,dx$ (24) $\displaystyle=\frac{1}{\sqrt{2\pi}}(\int_{0}^{b}e^{-\frac{x^{2}}{2}}\,dx-\int_ {0}^{a}e^{-\frac{x^{2}}{2}}\,dx-e^{-\frac{a^{2}}{2}}+e^{-\frac{b^{2}}{2}})$ $\displaystyle=\frac{1}{\sqrt{2\pi}}[(\frac{\sqrt{\pi}}{\sqrt{2}}erf(\frac{b}{ \sqrt{2}})-\frac{\sqrt{\pi}}{\sqrt{2}}erf(\frac{a}{\sqrt{2}})-e^{-\frac{a^{2}} {2}}+e^{-\frac{b^{2}}{2}}]$ $$

Equation (21) can be written as follows:

$$ $\displaystyle\mathbb{E}[|\epsilon_{ij}|]$ $\displaystyle=\frac{2}{\sqrt{2\pi}}\{[\frac{\sqrt{\pi}}{\sqrt{2}}erf(\frac{1}{ \sqrt{2}})-\frac{\sqrt{\pi}}{\sqrt{2}}erf(\frac{0}{\sqrt{2}})-e^{-\frac{0}{2}} +e^{-\frac{1}{2}}]$ (25) $\displaystyle-[(\frac{\sqrt{\pi}}{\sqrt{2}}erf(\frac{\infty}{\sqrt{2}})-\frac{ \sqrt{\pi}}{\sqrt{2}}erf(\frac{1}{\sqrt{2}})-e^{-\frac{1}{2}}+e^{-\frac{\infty }{2}}]\}$ $\displaystyle\xlongequal{erf(0)=0,erf(\infty)=1}2[erf(\frac{1}{\sqrt{2}})- \frac{1}{2}-\frac{1}{\sqrt{2\pi}}+\frac{2}{\sqrt{2\pi e}}]\approx 0.5354$ $$

Therefore, based on Equations (20), the expected value of the additional gradient error $\mathbb{E}[\Delta\frac{\partial L}{\partial w_{mn}^{(l)}}]$ can be expressed as follows:

$$ $\displaystyle\mathbb{E}[\Delta\frac{\partial L}{\partial w_{mn}^{(l)}}]$ $\displaystyle=\sum_{i}\sum_{j}\sum_{m^{\prime}}\sum_{n^{\prime}}\mathbb{E}[| \epsilon_{(i+m)(j+n)}^{(l)}\cdot\frac{\partial\sigma(y_{(i+m^{\prime})(j+n^{ \prime})}^{(l)})}{\partial y_{ij}^{(l)}}\cdot w_{m^{\prime}n^{\prime}}^{(l+1)} \cdot\frac{\partial L}{\partial y_{ij}^{(l+1)}}|]$ (26) $$

Based on Equation (14), since the binarization error $\epsilon_{ij}^{(l)}$ depends solely on the input $x_{ij}^{(l)}$ and is independent of any other variables, $\epsilon_{ij}^{(l)}$ and other random variables in Equation (26) are mutually independent.
Therefore, it follows that:

$$ $\displaystyle\mathbb{E}[\Delta\frac{\partial L}{\partial w_{mn}^{(l)}}]$ $\displaystyle=\sum_{i}\sum_{j}\sum_{m^{\prime}}\sum_{n^{\prime}}\mathbb{E}[| \epsilon_{(i+m)(j+n)}^{(l)}|]\cdot\mathbb{E}[|\frac{\partial\sigma(y_{(i+m^{ \prime})(j+n^{\prime})}^{(l)})}{\partial y_{ij}^{(l)}}\cdot w_{m^{\prime}n^{ \prime}}^{(l+1)}\cdot\frac{\partial L}{\partial y_{ij}^{(l+1)}}|]$ (27) $\displaystyle\approx 0.5354\cdot(\sum_{i}\sum_{j}\sum_{m^{\prime}=-(k//2)}^{k/ /2}\sum_{n^{\prime}=-(k//2)}^{k//2}\mathbb{E}[|\frac{\partial\sigma(y_{(i+m^{ \prime})(j+n^{\prime})}^{(l)})}{\partial y_{ij}^{(l)}}\cdot w_{m^{\prime}n^{ \prime}}^{(l+1)}\cdot\frac{\partial L}{\partial y_{ij}^{(l+1)}}|])$ $$

From the above equations, it is evident that as the size $k$ of the convolutional kernel in the subsequent layer increases, the element-wise gradient error introduced during the binarization process also increases.
Consequently, in binarized convolutional units, the smaller the size of the convolutional kernel $k$, the smaller the binarization error introduced into the binarized model.

Therefore, we use $1\times 1$ binarized convolution as the new binarized convolution.

### 6.4 More Details About Experiments

#### 6.4.1 Result of Different Version of BDC

We tested the performance metrics of different versions of BDC on the Occ3d-nuScenes validation set.
Table 5 presents the results.
The configurations of BDC-B and BDC-T follow the settings outlined in Table 1.
We binarized all modules in the 3D occupancy network except for the view transformer, referring to this as the small version (-S).
These modules include the image encoder, the BEV encoder, and the occupancy head.

Compared to BDC-T, BDC-S additionally binarizes the image backbone in the image encoder.
The image backbone contains substantial pre-trained knowledge, and binarizing it hinders leveraging this pre-trained knowledge, which leads to a significant performance drop compared to BDC-T.
Compared to FlashOcc$\dagger$, which does not use pre-trained weights in the image backbone, the binarized version shows a significant performance decline.

Therefore, we recommend against binarizing the image backbone.

**Table 5: Comparison of the occupancy prediction performance across different versions of BDC. BDC-S binarizes all modules in the 3D occupancy network except for the view transformer. These modules include an image encoder, BEV encoder, and occupancy head. $\dagger$ stands for not using pre-trained weights from an image backbone.**
| Methods | Params(M) | OPs(G) | $\blacksquare$ others | $\blacksquare$ barrier | $\blacksquare$ bicycle | $\blacksquare$ bus | $\blacksquare$ car | $\blacksquare$ const. veh. | $\blacksquare$ motorcycle | $\blacksquare$ pedestrian | $\blacksquare$ traffic cone | $\blacksquare$ trailer | $\blacksquare$ truck | $\blacksquare$ drive. suf. | $\blacksquare$ other flat | $\blacksquare$ sidewalk | $\blacksquare$ terrain | $\blacksquare$ manmade | $\blacksquare$ vegetation | mIoU |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| *CNN-based (32 bit)* |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| FlashOcc [] | 44.74 | 248.57 | 9.08 | 46.32 | 17.71 | 42.70 | 50.64 | 23.72 | 20.13 | 22.34 | 24.09 | 30.26 | 37.39 | 81.68 | 40.13 | 52.34 | 56.46 | 47.69 | 40.60 | 37.84 |
| FlashOcc$\dagger$ [] | 44.74 | 248.57 | 6.10 | 35.78 | 0.50 | 26.97 | 42.39 | 11.16 | 7.13 | 10.99 | 10.68 | 20.95 | 24.35 | 80.60 | 40.02 | 50.44 | 55.11 | 44.67 | 38.85 | 29.81 |
| *BNN-based (1 bit)* |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| BDC-B | 28.19 | 134.02 | 9.24 | 43.93 | 20.25 | 39.92 | 49.90 | 23.11 | 21.44 | 21.22 | 21.98 | 31.23 | 37.09 | 80.93 | 39.41 | 51.03 | 54.88 | 46.80 | 40.04 | 37.20 |
| BDC-T | 26.82 | 129.90 | 9.22 | 44.81 | 17.56 | 40.02 | 49.94 | 24.84 | 18.63 | 21.32 | 23.27 | 32.46 | 36.43 | 81.03 | 38.74 | 51.40 | 55.55 | 47.15 | 40.76 | 37.24 |
| BDC-S | 3.51 | 45.30 | 3.13 | 24.25 | 6.02 | 22.21 | 36.23 | 7.29 | 5.78 | 14.11 | 14.04 | 4.86 | 22.99 | 68.21 | 14.29 | 33.52 | 36.76 | 33.20 | 30.63 | 22.21 |

#### 6.4.2 Operations and Parameters of Binarized Module of 3D Occupancy Network

**Table 6: Operations and parameters of binarized module of 3D occupancy network.**
|  | Binarized | Image<br>Backbone | Image<br>Neck | View<br>Transformer | BEV<br>Backbone | BEV<br>Neck | Occupancy<br>Head |
| --- | --- | --- | --- | --- | --- | --- | --- |
| OPs(G) |  | 88.785 | 1.377 | 0.165 | 17.724 | 102.989 | 34.755 |
| ✓ | 3.700 | 0.034 | - | 0.098 | 30.071 | 11.233 |  |
| Params(M) |  | 23.508 | 4.155 | 0.039 | 12.394 | 6.556 | 0.869 |
| ✓ | 0.162 | 0.016 | - | 0.036 | 2.975 | 0.282 |  |

In Table 6, we investigate the changes in computation (OPs) and parameters (Params) across different modules of the 3D occupancy network before and after binarization.
The image encoder consists of the image backbone and image neck, while the BEV encoder includes the BEV backbone and BEV neck.

We do not binarize the view transformer because its 32-bit full-precision parameters and computation are already sufficient.
Additionally, the view transformer relies on full-precision computation to precisely map 2D image features to 3D BEV features.

#### 6.4.3 More Visualization

Figure: Figure 8: More Visualization rensults on Occ3D-nuScenes validation set
Refer to caption: x9.png

In this section, we provide additional occupancy prediction results of BiSRNet [] and our BDC applied to Flashocc in Fig 8.
Compared to BiSRNet, BDC offers superior scene reconstruction capability and more accurate label prediction.

### 6.5 Broader Impacts

3D occupancy prediction stands as a core task in autonomous driving perception.
Leveraging occupancy grids effectively addresses real-world challenges such as long-tail datasets and target truncation, which 3D object detection algorithms may struggle to resolve.
Our approach, BDC-Occ, demonstrates superior efficiency and accuracy in predicting the occupancy status of voxels in 3D space compared to all existing state-of-the-art methods based on Binarized Neural Networks (BNNs), holding significant value for practical applications.
Thus far, 3D occupancy prediction technology has not yielded any adverse societal impacts.
Our proposed BDC-Occ likewise does not introduce any foreseeable negative social consequences.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Why are so many versions of the main block are needed?
soundness: 2
strengths: - It shows as main strength the theoretical insights that 1x1 binarized convolution is more robust to binarization and is thus used in to make the network deeper. Furthermore, it introduces an additional branch within the network to further refine the per-channel output of each layer based on this observation.
- The results indicate an improvement over other binarization methods in terms of IoU and mAP.
- Ablation studies are performed to show the impact of the various proposed changes.
summary: This paper presents a method for binarizing convolution operations in binary occupancy networks. It first analyzes the impact of binarization theoretically, and then proposes a mitigation approach based on binarized convolution unit that enhances performance. It provides results based on two benchmarks and comparisons with other networks and binarization methods.
weaknesses: - In terms comparing the FPS with FlashOCC, one of the main objectives of binarization is to provide FPS speedups, since the provided results are marginally improved at best it seems to make the proposed approach less necessary. Instead FPS should be compared with other binarization methods as well for fairness. 
- From the current writeup it is not clear how the proposed module can be plugged into other existing methods.
- I am unconvinced that since in my understanding the backbone is left as FP32 why binarization is necessary and other forms of quantization are not considered appropriate.
- It is not clear what aspects of the approach are specific to the occupancy task and what is generalizable to other tasks as well.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: Here are some questions and suggestions for the authors:

1. **Alternative Binarization Techniques**: The paper focuses on BiSR-Conv for binarization. Could the authors provide a comparison with other established techniques like XNOR-Net or DoReFa-Net? This would help understand if the performance gains stem specifically from the BDC unit or are also achievable with other carefully tuned binarization methods.  Furthermore, exploring ternary or higher bit-width quantization could provide a more nuanced understanding of the trade-off between accuracy and efficiency.

2. **[minor] Comparison with Other Compression Methods**:  Beyond BNNs, how does BDC-Occ compare to other model compression techniques like pruning, quantization, and knowledge distillation in the context of 3D occupancy prediction?  Providing quantitative results or discussion comparing these methods would strengthen the argument for BDC-Occ's practical value.

3. **Detailed Analysis of Per-Channel Refinement Branch**:  The ablation study provides some insights, but a more in-depth analysis of the per-channel refinement branch is needed.  How sensitive is its performance to the number of layers in `MulBiconv` (N)? Are there alternative architectures for this branch that could further improve cross-channel feature learning? A visualization of the learned channel weights might also offer insightful qualitative analysis.

4. **[minor] Generalizability to Transformers**: The stated limitation to CNN architectures raises questions about BDC's broader applicability. Have the authors explored applying BDC, or a modified version, to Transformer-based occupancy networks?  Even negative results in this direction would be valuable for understanding the challenges and potential future work.
soundness: 3
strengths: Originality: The paper demonstrates originality in its identification and solution for the performance degradation problem in binarized 3D occupancy networks. While BNNs have been explored in other domains, applying them effectively to 3D occupancy prediction, especially with a focus on maintaining performance with increasing network depth, is novel. The proposed BDC unit, with its 1x1 kernel constraint and the per-channel refinement branch, presents a creative combination of techniques tailored to address the specific challenges of binarization in this context.

Quality: The technical quality of the paper is good. The authors provide theoretical justification for their design choices and conduct thorough experiments to validate the effectiveness of the BDC unit. The results convincingly demonstrate the superiority of BDC-Occ over other state-of-the-art binarized methods and its competitiveness with full-precision models. The ablation studies further strengthen the claims by showcasing the individual contributions of each component of the BDC unit.

Clarity: While the core ideas are presented clearly, the clarity of the paper could benefit from some improvements. The mathematical proofs are clear. Additionally, visually presenting the overall architecture of BDC-Occ aid understanding.

Significance: The significance of the work lies in its potential to enable deployment of accurate 3D occupancy prediction on edge devices. The substantial reduction in computational cost achieved by BDC-Occ, without sacrificing accuracy, is notable. It would be nice to discuss the roadmap to apply on transformer, to inspire further researches.
summary: This paper introduces BDC-Occ, a novel binarized neural network (BNN) for 3D occupancy prediction. Recognizing the computational challenges of deploying existing 3D occupancy networks on edge devices, the authors leverage BNNs for model compression. However, they note that simply binarizing existing models leads to performance degradation, particularly when increasing network depth.

The paper's core contribution is the Binarized Deep Convolution (BDC) unit.  It addresses the identified limitations of binarized convolutions through two key innovations. First, additional convolutional kernels within the BDC are constrained to 1x1 to minimize the impact of binarization errors as network depth increases. Second, a per-channel refinement branch reweights outputs using a first-order approximation, improving the capture of cross-channel feature importance.

Through extensive experiments on the Occ3D-nuScenes dataset, the authors demonstrate that BDC-Occ achieves state-of-the-art performance among BNNs, even rivaling full-precision models in mIoU while significantly reducing parameters and operations.  They further validate the generalizability of the BDC unit by showing its effectiveness in 3D object detection tasks on the nuScenes dataset.
weaknesses: 1. **Limited exploration of binarization strategies**: The paper primarily focuses on binarizing convolutional layers using the BiSR-Conv method.  Exploring alternative binarization techniques, such as XNOR-Net [1] or DoReFa-Net [2], and comparing their performance with BDC-Occ would strengthen the analysis and potentially reveal further insights.  It's also important to investigate the impact of different activation functions specifically designed for BNNs, like those proposed in ReactNet [3].

2. **[minor] Lack of comparison with other compression techniques**: The paper positions BDC-Occ as a solution for deploying occupancy networks on edge devices. However, it lacks a comparison with other model compression methods beyond binarization, such as pruning [4], quantization [5], or knowledge distillation [6].  Demonstrating the advantages of BDC-Occ over these alternatives would significantly bolster its practical significance.

3. **[minor] Limited generalizability**: The paper acknowledges the limitation to CNN architectures. However, this limitation needs further discussion and investigation. This would help understand the challenges and potentially open avenues for future research on binarizing more diverse network architectures.


**References:**

[1] Hubara et al., Binarized Neural Networks, NIPS 2016.

[2] Zhou et al., DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients, arXiv 2016.

[3] Liu et al., Reactnet: Towards precise binary neural network with generalized activation functions, ECCV 2020.

[4] Han et al., Learning both Weights and Connections for Efficient Neural Networks, NIPS 2015.

[5] Jacob et al., Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference, CVPR 2018.

[6] Hinton et al., Distilling the Knowledge in a Neural Network, arXiv 2015.

---

confidence: 3
contribution: 3
initial_rating: 3
presentation: 3
questions: 1. In the paper, binarization is adopted for deployment on edge devices. Is there any experiment related to this?
2. The FPS in Table 3 seems to be a small improvement compared to FlashOcc, which is a floating point model. Is there any experimental results of FPS and run time of BDC-B?
soundness: 3
strengths: 1. This paper is the first study to apply binarization to the task of 3D occupancy prediction. The method of the authors significantly reduces the computational cost while maintaining the performance.
2. The proposed BDC unit significantly improves the performance of the network through theoretical analysis.
summary: BDC-Occ proposes a BDC unit that applies binarization to 3D occupancy prediction tasks. To alleviate binarization errors, they use a 1x1 binarized convolution and design a BDC unit based on it. The authors show that their method reduces the performance gap with floating-point models and significantly reduces hardware resources.
weaknesses: 1. what is the contribution of BDC-V2 in Figure 3? It seems that it only increases the computational cost, with no performance improvement. Furthermore, MultiBiconv seems to be a multiple iteration of the technique from BDC-V1.
2. The paper seems to propose the design of a binarization methodology, not a design for Occupancy Prediction. While this is the first application of binarization to occupancy prediction, other binarization methodologies seem to be easily adaptable. 
3. The design in Section 3.4 seems to be very similar to that of BiSRNet, which may limit the contribution of the paper. Is there anything different about the design of BiSRNet?

---

confidence: 2
contribution: 2
initial_rating: 5
presentation: 3
questions: - The paper would benefit from a clearer description of the hardware platform used and the specific implementation strategy for 1-bit quantized convolution. Key questions remain unanswered, such as whether the quantized operations are run on a specialized computing framework like TensorRT, which is widely used for deploying optimized deep learning inference on edge and server devices. Additionally, details on how 1-bit quantized convolution is implemented—such as the use of custom kernels, acceleration libraries, or optimizations for reduced precision—would provide readers with a clearer understanding of the practical feasibility and performance considerations of this approach.

- If the work relies on standard hardware (e.g., CPUs or GPUs) without the assistance of dedicated quantization frameworks, this would impact performance significantly compared to using more specialized hardware like TPUs or FPGAs, which are often better suited for extreme quantization levels. Specifying these factors is crucial for evaluating the performance claims, as well as the actual benefit of 1-bit quantization in terms of speed and efficiency.
soundness: 3
strengths: Occupancy estimation plays a critical role in real-time applications like autonomous driving, where precise environment mapping is essential for safe and efficient vehicle navigation. However, the computational demands of occupancy estimation, especially when deployed on device-side platforms like Orin GPUs, create significant challenges in terms of efficiency and resource constraints. This research addresses these challenges by exploring strategies to minimize computational overhead, a vital area of study given the increasing emphasis on edge computing in autonomous systems. By carefully engineering these binarization blocks, the proposed approach effectively mitigates quantization loss, thus maintaining model accuracy while reducing resource consumption.
summary: To reduce parameters and computational costs, this paper proposes a binarized occupancy network tailored specifically for CNN-based occupancy networks. The approach introduces two key techniques: first, an additional 1x1 binarized convolution layer is added to increase network depth, thereby enhancing feature extraction while maintaining efficiency. Second, a per-channel refinement branch is incorporated to reduce quantization error, improving the model’s precision despite the constraints of binarization. These enhancements aim to optimize the balance between performance and resource efficiency in the proposed binarized network.
weaknesses: # Major Concern:
While this work serves as an early exploration of binary quantization for the occupancy prediction task, several significant limitations are apparent. The study does present two main findings in the context of binary occupancy networks:
- 1×1 binarized convolution introduces only minimal binarization errors as network depth increases.
- Binarized convolution is notably less effective than full-precision convolution at capturing cross-channel feature importance.

However, the proposed strategies to address these issues—namely, enhancing network depth through additional 1x1 binarized layers and adding a per-channel refinement branch—have already been extensively explored in the quantization literature. This repetition limits the novelty of the work, and the paper does not provide fresh insights into the specific task of occupancy prediction. The binary quantization methods introduced here, while perhaps incrementally improving upon existing binary quantization techniques, do not directly tackle the occupancy task itself. Instead, they represent a basic application of binary quantization to occupancy, yielding largely expected results. As such, I believe this work falls short of the standards expected at a top-tier machine learning conference, and the main track may not be the optimal venue for this submission.

# Minor Comments:
- This study restricts its exploration to CNN-based approaches for occupancy tasks, without incorporating transformer-based occupancy networks. Transformer models have gained traction and demonstrated superior performance in related fields, and their absence here limits the study’s relevance to current research trends.
- The paper outlines a manually crafted binary network architecture aimed at mitigating quantization-induced errors. However, quantizing both weights and activations from 32-bit to 1-bit introduces significant quantization errors. Without innovations specifically targeted at reducing such errors in this context, the proposed method struggles to demonstrate substantial improvements.
- The speed advantage observed (6.22 FPS versus 7.53 FPS) is marginal, casting doubt on the practical benefits and motivation of the approach. If the goal is to address computational cost and memory constraints in existing occupancy networks, a rigorous analysis of actual speedup and memory reduction should be provided. Alternatively, if the study’s focus is to explore novel quantization techniques for this task, one would expect groundbreaking insights tailored to occupancy prediction. Given the current submission, both aspects seem underdeveloped.