BiPointNet: Binary Neural Network for Point Clouds
Authors: Haotong Qin
Zhongang Cai
Mingyuan Zhang
Yifu Ding
Haiyu Zhao
Shuai Yi
Xianglong Liu
Hao Su
Venue: ICLR 2021 Poster (2021)
Keywords: point clouds
efficient deep learning
binary neural networks

Abstract: To alleviate the resource constraint for real-time point cloud applications that run on edge devices, in this paper we present BiPointNet, the first model binarization approach for efficient deep learning on point clouds. We discover that the immense performance drop of binarized models for point clouds mainly stems from two challenges: aggregation-induced feature homogenization that leads to a degradation of information entropy, and scale distortion that hinders optimization and invalidates scale-sensitive structures. With theoretical justifications and in-depth analysis, our BiPointNet introduces Entropy-Maximizing Aggregation (EMA) to modulate the distribution before aggregation for the maximum information entropy, and Layer-wise Scale Recovery (LSR) to efficiently restore feature representation capacity. Extensive experiments show that BiPointNet outperforms existing binarization methods by convincing margins, at the level even comparable with the full precision counterpart. We highlight that our techniques are generic, guaranteeing significant improvements on various fundamental tasks and mainstream backbones. Moreover, BiPointNet gives an impressive 14.7× speedup and 18.9× storage saving on real-world resource-constrained devices.

Published as a conference paper at ICLR 2021
BIPOINTNET: BINARY NEURAL NETWORK FOR POINT CLOUDS
Haotong Qin Zhongang Cai*s, Mingyuan Zhang",, Yifu Ding', Haiyu Zhao
Shuai YT, Xianglong Liu Hao Su*
1 State Key Lab of Software Development Environment, Beihang University Shen Yuan Honors College, Beihang University "SmeseTime Research UUniversity of California, San Diego
{qinhacteng, xlliu] buua.......... edu.cn, zjdyf edu, en
fcai zhongang, zhangmi ngyuan, zhaohaiyu yo, yi shuai J@eesstt ime. com hacsuêeng, sesd
aganganngyuan,
ABSTRACT
To alleviate the resource constraint for real-time point cloud applications that run edge devices, in this paper we present BiPointNet, the first model binarization approach for efficient deep learning on point clouds. We discover that the im- mense performance drop of binarized models for point clouds mainly stems from two challenges: aggregation-induue -nnduudd feature homogenization that leads to a degra- dation of information entropy, and scale distortion that hinders optimization and invalidates scale-sensitive structures. With theoretical justifications and in-depth analysis, our BiPointNet introduces Entropy-Maximizing Aggregation (EMA) modulate the distribution before aggregation for the maximum information en- tropy, and Layer-wise Scale Recovery (LSR) to efficiently restore feature repre- sentation capacity. Extensive experiments show that BiPointNet outperforms ex- isting binarization methods by convincing margins, at the level even comparable with the full precision counterpart. We highlight that our techniques are generic, guarantceing significant improvements on various fundamental tasks and main- stream backbones. Moreover, BiPointNet gives an impressive 14.7x speedup and 18.9 storage saving on real-world resource-constrainee devices.
INTRODUCTION
With the advent of deep neural networks that directly process raw point clouds (PointNet (Qi et al.
2017a) as the pioneering work), great success has been achieved learning on point clouds (Qi et al,
2017b Li et al. 20188; Wang et al. 2019a Wu et al.) 2019 Thomas et al. 2019 Liu al. 2019b
Zhang et al.] 2019b). Point cloud applications, such as autonomous driving and augmented reality, often require real-time interaction and fast response. However, computation for such applications is
edge devices. To address the challenge, novel algorithms, such Grid-GCN (Xu al. 2020b), CandL_A-Ne (Hu et al., 2020), and PointVoxel (Liu et al.) 2019d), have been proposed to accelerate those point cloud processing networks. While significant speedup and memory footprint reduction have been achieved, these works still rely on expensive floating-point operations, leaving room for further optimization of the performance from the model quantization perspective. Model binarization (Rastegari al. 2016 Bulat & Tzimiropoulos 2019 Hubara et al. 2016 Wang et al. 2020 Zhu et al.) [019] Xu et al. 2019) emerged as one of the most promising approaches optimize neural networks for better computational and memory usage efficiency. Binary Neural Networks (BNNs) leverage 1) compact binarized parameters that take small memory space, and 2) highly efficient bitwise operations which are far less costly compared to the floating-poin counterparts.
usually deployed on essource-constraigg
Despite that 2D vision tasks (Krizhevsky et al. 2012 Simonyan Zisserman 2014 Szegedy ct al. 2015 Girshick et al. 2014 Girshick 2015
Russakovsky et al. 2015 Wang et al. 2019 99bb
"equal contributions
1 corresponninng author

---

Published as a conference paper at ICLR 2021
Entropy-Maxviizing Aggregation
Layer-wise Scale Recovery
traneformaioon
usk
rapregation und
beurleed
leomab layer
acaling ficker
tpput feature
trp
benaiized weight
origical
cuspui
input transform
BLMPPs
feature
BiMLPs
EMA-max
input points
transform
BiMLPS
mx3
nxba
ax1024
1024
shared
shared
feature
output scores
Figure 1: Overview of our BiPointNet on PointNet base model, applying Entropy-Maximizing Ag- gregation (EMA) and Layer- wise Scale Recovery (LSR). EMA consists of the transformation unit and the aggregation unit for maximizing the information entropy of feature after binarization. LSR with the learnable layer-wise scaling factor is applied to address the scale distortion of bi-linear layers (which form the BiMLPs), flexibly restore the distorted output to reasonable values
Zhang al. 2021 has been studied extensively by the model binarization community, the methods developed are not readily transferable for 3D point cloud networks due to the fundamental differ- ences between 2D images and 3D point clouds. First, to gain efficiency in processing unordered 3D points, many point cloud learning methods rely heavily pooling layers with large receptive field to aggregate point-wise features. As shown in PointNet (Qi et al., 2017b), global pooling provides strong recognition capability. However, this practice poses challenges for binarization. Our analy- ses show that the degradation of feature diversity, a persistent problem with binarization (Liu et al.) 2019a Qin et al. 2020b Xie et al. (017), is significantly amplified by the global aggregation func- tion (Figure 2) leading to homogenization of global features with limited discriminability. Second, the binarization causes immense scale distortion at the point-wise feature extraction stage, which is detrimental to model performance in two ways: the saturation of forward-propagatee features and backward-propagatee gradients hinders optimization, and the disruption of the scale-sensitive structures (Figure B results in the invalidation of their designated functionality.
In this paper, we provide theoretical formulations of the abooee-entionn insights through in-depth analysis. Such understanding allows us to propose method that turns full- precision point cloud networks into extremely efficient yet strong binarized models (see the overview in Figure D To tackle the homogenization of the binarized features after passing the aggregation function, we study the correlation between the information entropy of binarization features and the performance of point cloud aggregation functions. We thus propose Entropy-Maximiiing Aggrega- tion (EMA) that shifts the feature distribution towards the statistical optimum, effectively improving expression capability of the global features. Moreover, given maximized information entropy, further develop Layer-wise Scale Recovery (LSR) to efficiently restore the output scale that enhances optimization, which allows scale-sensitive structures to function properly. LSR uses only one learn- able parameter per layer, leading to negligible storage increment and computation overhead.
phenomenons and obtain
Our BiPointNet is the first binarization approaches deep learning point clouds, and it outper- forms existing binarization algorithms for 2D vision by convincing margins. It is even almost on conduct most analysis on the PointNet baseline, we show that our methods are generic and can be readily extendable to other pop- ular backbones, such PointNet++ (Qi et al. (2017)), PointCNN (Li al. 2018). DGCNN (Wang et al. 2019a), and PointConv (Wu et al.) 2019), which are the representatives of mainstream cate- gories of point cloud feature extractors. Moreover, extensive experiments on multiple fundamental tasks on the point cloud, such as classification, part segmentation, and semantic segmentation, high- light that our BiPointNet is task-agnostic. Besides, we highlight that our EMA and LSR are efficient and easy to implement in practice: in the actual test popular edge devices, BiPointNet achieves 14.7x speedup and 18.9x storage savings compared to the precisson PointNet. Our code is released at
par (within 1-2%) with the full-precision counterpart. Although
2 RELATED WORK
Network Binarization. Recently, various quantization methods for neural networks have emerged, such uniform quantization (Gong et al. 2019 Zhu 2020), mixed-ppeeiiiion quantization (Wul
2

---

Published as a conference paper at ICLR 2021
et al. 2018: Yu al. 2020), and binarization. Among these methods, binarization enjoys compact binarized parameters and highly efficient bitwise operations for extreme compression and accelera- tion (Rastegari et al.
2016 Qin et al., 2020a). In general, the forward and backward propagation of binarized models the training process can be formulated as:
+1, if 2 0
gb: T (I)
Forward b sign(x) -1, otherwise
Backward 9r
0, otherwise
where denotes the element in floating-point weights and activations, denotes the element in binarized weights Bw and activations Ba. Ir. and Ib donate the gradient 8y where is the cost function for the minibatch. In forward propagation, sign function directly applied to obtain the binary parameters. In backward propagation, the Straight-Through Estimator (STE) (Bengio et al. 2013) is used to obtain the derivative of the sign function, avoiding getting all zero gradients. The existing binarization methods are designed to obtain accurate binarized networks by minimizing the quantization error (Rastegari et al.) 2016 Zhou et al.] 2016)] Lin et al.. 2017).
8C and ac ah respectively,
improving loss function (Ding al. 2019 Hou et al. (017), reducing the gradient error (Liu al.
2018 2020), and designing novel structures and pipelines (Martinez et al.
2020). Unfortunately. we show Sec 3 that these methods, designed for 2D vision tasks, are not readily transferable to 31)
point clouds.
Deep Learning Point Clouds. PointNet (Qi et al. 201 7a) is the first deep learning model that processes raw point clouds directly. The basic building blocks proposed by PointNet such as MLP for point-wise feature extraction and max pooling for global aggregation (Guo al., 2020) have become the popular design choices for various categories of newer backbones: 1) the pointwise ML.--based such as PointNet++ (Qi et al., 2017b); the graph-based such as DGCNN (Xu al., 2020b); 3) the convolution-based such as PointCNN (Li et al. 2018), PointConv (Wu et al. 2019) RS-CNN (Liu et al., 2019c) and KP-Conv (Thomas et al. 2019). Recently, methods are proposed for efficient deep learning point clouds through novel data structuring (Xu et al.) 2020b). faster sampling (Hu et al., 2020). adaptive filters (Xu et al. 2020a), efficient representation (Liu et al.) 2019d) convolution operation (Zhang al., (0019b) However, they still use expensive floating- point parameters and operations, which can be improved by binarization.
3 METHODS
Binarized models operate on efficient binary parameters, but often suffer large performance drop- Moreover, the unique characteristics of point clouds pose even more challenges. We observe there are two main problems: first, aggregation of large number of points leads to a severe loss of feature diversity; second, binarization induces an immense scale distortion, that undermines the functionality of scale- caalesennsiiive structures. In this section, we discuss our observations, and propose our BiPointNet with theoretical justifications.
3.1 BINARIZATION FRAMEWORK
We first give brief introduction to our framework that binarizes floating-poin network. For example, deep learning models on point clouds typically contain multi-layer perceptrons (MLPs) for feature extraction. In contrast, the binarized models contain binary MLPs (BiMLPs), which are composed of binarized linear (bi-linear) layers. Bi-linear layers perform the extremely efficient bitwise operations (XNOR and Bitcount) on the lightweight binary weight/activation. Specifically, the activation of the bi-linear layer is binarized to B,, and is computed with the binarized weight By obtain the output Z:
Z B2 H Bw,
where @ denotes the inner product for vectors with bitwise operations XNOR and Bitcount. When B., and B, denote the random variables By and Ba, we represent their probability mass function as PB. (bw,), and PB. (b.).
Moreover, we divide the BiPointNet into units for detailed discussions. In BiPointNet, the original data feature X x first enters the symmetric function Z which represents composite function built by stacking several permutation equivariant and permutation invariant layers (c.g., nonlinear layer, bi-linear layer, pooling). And then, the output Y B is binarized obtain
3

---

Published as a conference paper at ICLR 2021
i1-i0| ii-ibl:
ii-io|
(a) Polnswise fesures be agarouned.
(3) Fullssodiskon festures eparogats with mas paoling
(c) fouures aggregazed with
(4) Blnznized Sarures 4paroguned
mee pooling
EMA.
Figure 2: Aggregation-indin induced feature homogenization. (a) shows the activation of each test sample in batch of ModelNet40. In (b)-(d), the single feature vectors pooled from all points mapped to colors. The diversity of colors represents the diversity of pooled features. The original aggregation design is incompatible with binarization, leading to the homogenization of output features in (c), whereas our proposed EMA retains high information entropy, shown in (d)
the binary feature B E {-1, where takes n when the feature is modeled independently and takes when the feature aggregated globally. The single unit is thus represented
B sign Y) = sign sign(S((X)).
(3)
Similarly, when B, Y and X denote the random variables sampled from B, Y and X, we represent their probability mass function as Pa (b), (y) and px (x).
3.2 ENTROPY-MAXIMTAA AGGREGATION
Unlike images pixels that are arranged in regular lattices, point clouds are sets of points without any specific order. Hence, features are usually processed in point-wise manner and aggregated explicitly through pooling layers. Our study shows that the aggregation function is performance bottleneck of the binarized model, due to severe homogenization as shown in Figure
We apply information theory (Section I to quantify the effect of the loss of feature diversity, and find that global feature aggregation leads to a catastrophic loss of information entropy. In Sec- tion 3.2.2 we propose the concept of Sitioopy--aximziiggg Aggregation (EMA) that gives the sta- tistically maximum information entropy to effectively tackle the feature homogenization problem.
3.2.1 AGGREGATION
FEATURE HOMOGENIZATION
Ideally, the binarized tensor B should reflect the information in the original tensor Y as much as possible. From the perspective of information, maximizing mutual information can maximize the information flow from the full-precision to the binarized parameters. Hence, our goal is equivalent to maximizing the mutual information Z(Y; B) of the random variables Y and B:
arg max I(Y;B)=H(B)) H(B) H(B | Y) Y.8
(4)
where H(BB) is the information entropy, and H(B Y) is the conditional entropy of given Y. H(B Y) = 0 use the deterministic sign function as the quantizer in binarization (see Section A. for details). Hence, the original objective function Eq. (4) is equivalent to:
arg max HB( = - -- pn(b) (b)),, B
(5)
BEB
where B is the set of possible values of B. We then study the information properties of max pool- ing, which is a common aggregation function used in popular point cloud learning models such as PointNet. Let the max pooling be the last layer & of the multi-layer stacked Q, and the input of ø defined as Xg. The data flow of Eq. (3) can further expressed as B = sign(s((X,) ())), and the information entropy Hn of binarized feature B can be expressed as
Hx(X,) = Σ Px, (xx)) log Σ px.(xa)) - (1- px. (x.((r)))) log (1- (1-(2) px.(ta))) (6) 2.20
1020
2020
where n is the number of elements aggregated by the max pooling, and X is the random variable sampled from X The brief derivation of Eg (6) is shown in Appendix A.2 Theorem showss the information properties of max pooling with the normal distribution input on the binarized network architecture.
4

---

Published as a conference paper at ICLR 2021
Theorem For input X of max pooling & with arbitrary distribution, the information entropy of the binarized output goes to zero as n goes to infinity, i.e., lim HB = 0. And there exists a n-++0 constant e, for any ny and 122, ni > n2 > we have Hn,.. < HB where n the number of elements to be aggregated.
The proof of Theorem is included in Appendix A.2 which explains the severe feature homoge- nization after global feature pooling layers. As the number of points is typically large (e.g. 1024 points by convention in ModelNet40 classification task), it significantly reduces the information en- tropy H. of binarized feature B, i.., the information of hardly retained in B, leading to highly similar output features regardless of the input features to pooling layer as shown in Figure
Furthermore, Theorem | provides a theoretical justification for the poor performance of existing binarization methods, transferred from 2D vision tasks to point cloud applications. In 2D vision, the aggregation functions are often used to gather local features with a small kernel size n (e.g. n = in ResNet (He et al. 2016: Liu et al.] 2018) and VGG-Net (Simonyan & Zisserman, 2014) which use 2 x 2 pooling kernels). Hence, the feature homogenization problem on images is not as significant as that on point clouds.
3.2.2 EMA FOR MAXIMUM INFORMATION ENTROPY
Therefore, we need a class of aggregation functions that maximize the information entropy of B to avoid the eegregationninducee feature homogenization.
We study the correlation between the information entropy HB of binary random variable B and the distribution of the full-precision random variable Y. We notice that the sign function used in binarization has fixed threshold and decision levels, we get Proposition | about information entropy of B and the distribution Y.
Proposition 1 When the distribution of the random variable Y satisfies Eycopr(v) Ey20 Py (v) 0.5, the information entropy HB is maximized.
The proof of Proposition[iis shown in Appendix A..3 Therefore, theoretically, there is a distribution of Y that can maximize the mutual information of Y and B by maximizing the information entropy of the binary tensor B, so as to maximally retain the information of in B.
To maximize the information entropy HB. we propose the EMA for feature aggregation in BiPoint- Net. The EMA is not one, but a class of binarization-friendly aggregation layers. Modifying the aggregation function in the full-precision neural network to EMA keeps the entropy maximized by input transformation. The defimition of EMA is
(7) where yp denotes the aggregation function (e.g. max pooling and average pooling) and r denotes the transformation unit. Note that a standard normal distribution (N 1) is assumed for Xs because batch normalization layers are placed prior to the pooling layers by convention.
Y = EMA(X;) = +x(()))).
can take many forms; we discover that a simple constant offset is already effective. The offset shifts the input so that the output distribution satisfies Eyea Py Py(y) (v) 0.5, maximize the information entropy of binary feature B. The transformation unit our BiPointNet can be defined as r(Xp) X. -
When max pooling is applied as we obtain the distribution offset 5 for the input X that maxi- mizes the information entropy HB by solving the objective function
arg max Ha (8) =
ΣΣ 26 20
1
log
Σ 2020
2x2
2
(10--)2
(1 - Σ 20
log
11
Σ
v2
#620
where n denotes the number of elements in each batch. For each n, we can obtain an optimized 8t for Eq. (8) include the pseudo code in the Appendix A.5]
Moreover, we derive in the Appendix A.6 that when average pooling is used as y. the solution to 0. We thus obtain &vvg = 0. This means the solution
its objective function is expressed
5

---

Published as a conference paper at ICLR 2021
207
Norsal Gredent Zor wilerr
0974
2011
197
Mur
70%
1.2
Momate
1.4
40%
20%
1.20 350 375
202
25
30
75
1.00
325
155
175
202
17%
ProL
54N
(e)
(b)
(c)
Figure 4: (a) Information entropy of the aggregated features. With EMA, our BiPointNet achieves higher information entropy. (b) Regularizer loss comparison. Our PointNet has low loss, indicating that the scale distortion reduced and T-Net is not disrupted. (c) Ratio zero-grann pro-ggradient activations
in back-propagation LSR alleviates the scale distortion, enhancing the optimization process
not related to n. Hence, average pooling can be regarded as flexible alternative because its performance is independent of the input number
In a nutshell, we provide two possible variants yp: first, we show that simple shift sufficient to turn max pooling layer into an EMA (EMA-max); second, average pooling can be directly used (EMA-avg) without modification as a large number of points does not undermine its information entropy, making it adaptive to the dynamically changing number of point input. Note that modi- fying existing aggregation functions is only one way to achieve EMA; the theory also instructs the development of new binarization- friirzll aggregation functions in the future.
3.3 ONE-SCALE-HITS-ALEE LAYER-WISE SCALE RECOVERY
In this section, we show that binarization leads to feature scale distortion and study its cause. We conclude that the distortion directly related to the number of feature channels. More importantly, we discuss the detriments of scale distortion from the perspectives of the functionality of scale- sensitive structures and the optimization.
To address the severe scale distortion in feature due to binarization, we propose the Layer-wise Scale Recovery (LSSR). L.R,, each bi-linear layer added only one learnable scaling factor recover the original scales of all binarized parameters, with negligible additional computational overhead and memory usage.
3.3.1 SCALE DISTORTION
The scale of parameters is defined as the standard deviation of their distribution. As we mentioned in Section balanced bina- rized weights are used in the bi-linear layer aiming to maximize the entropy of the output after binarization, i.c., PB.. (1) = 0.5 and pB. (1) 0.5.
h (b) PointNer
(a) Input point cloud
A (0) BiPointNet
Theorem When let PB_ (1) 0.5 and PB. (1) 0.5 in bi-linear layer to maximize the mutual information, for the
BNN
binarized weight Bw E {- {-1, [-1,+11mxxt and activation B, E
{-1,+1y +1xmm, the probability mass function for the distribution of output Z can represented as pz (2i m) 0.5mM Ci O E {0, 1, m). The oulput has approximately normal distribu- tion ((,,m).
Figure 3: Scale Distortion. Fig- ures (b)-(d) show the trans- formed input. Compared with the input (a), the scales of (b) in full pellisreiiioon PointNet and (c) in our BiPointNet are normal, while the scale of (d) BNN is significantly distorted
The proof of Theorem found Appendix A.4 Theorem shows that given the maximized information entropy, the scale of the output features is directly related to the number of feature channels. Hence, scale distortion is pervasive large number of channels is the design norm of deep learning neural networks for effective feature extraction.
2
We discuss two major impacts of the scale distortion on the performance of binarized point cloud learning models. First, the scale distortion invalidates structures designed for 3DD deep learning that

---

Published as a conference paper at ICLR 2021
Table 1: Ablation study for our BiPointNet of various tasks on ModelNet40 (classification), ShapeNet Parts (part segmentation), and S3DIS (semantic segmentation). EMA and LSR and com- plementary to each other, and they are useful across all three applications
ModelNet40
ShapeNet Parts
S3DIS
Method
Bit-width
Aggr.
mIoU
mIoU
32/32
MAX
54.4
83.5
Full Prec.
32/32
AVG
86.5
84.0
51.5
81.5
BNN
1/1
MAX
7.1
54.0
9.5
45.0
BNN-LSR
1/1
MAX
4.1
58.7
2.0
25.4
1/1
EMA-avg
11.3
53.0
9.9
46.8
BNN-EMA
1/1
EMA-max
16.2
47.3
8.5
47.2
1/1
EMA-avg
82.5
80.3
40.9
74.9
Ours
1/1
EMA-max
80.6
44.3
76.7
are sensitive to the scale of values. For example, the T-Net in PointNet is designed to predict an orthogonal transformation matrix for canonicalization of input and intermediate features (Qi al.
2017a). The predicted matrix is regularized by minimizing the loss term Lyeg = 1- ZZ"
However, this regularization is ineffective for the Z with huge variance as shown Figure 3
Second, the scale distortion leads to a saturation of pewaaddproppagted activations and backward- propagated gradients (Ding al., 2019). In the binary neural networks, some modules (such as sign and Hardtanh) rely on the Straight- Through Estimator (STE) Bengio et al. (2013) for feature binarization or feature balancing. When the scale of their input is amplified, the gradient truncated instead of increased proportionally. Such saturation, shown in Fig[(c) hinders learning and even leads to divergence.
3.3.2 LSR FOR OUTPUT SCALE RECOVERY
To recover the scale and adjustment ability of output, we propose the LSR for bi-linear layers in our BiPointNet. We design a learnable layer-wise scaling factor a in our LSR. a is initialized by the ratio of the standard deviations between the output of bi-linear and full-precisio counterpart:
0g (A & W (0/a(BB, (B, @ Bw),
where a denotes as the standard deviation. And the a is learnable during the training process. The calculation and derivative process of the bi-linear layer with LSR are follows:
Forward
a (Ba 3 Bw) Backward So = gz (Ba о Bw),
(10)
where 9o and gz denotes the gradient Da and respectively. By applying the LSR in BiPointNet. we mitigate the scale distortion of output caused by binarization.
#C
Compared to existing methods, the advantages of LSR is summarized in two folds. First, LSR is efficient. It not only abandons the adjustment of input activations to avoid expensive inference time computation, but also recovers the scale of all weights parameters in a layer collectively instead of expensive restoration in channel-wise manner (Rastegari et al. (016). Second, LSR serves the purpose of scale recovery that we show more effective than other adaptation such as minimizing quantization errors (Qin et al. 2020b Liu et al. 2018).
4 EXPERIMENTS
In this section, we conduct extensive experiments to validate the effectiveness of our proposed Bi- PointNet for efficient learning point clouds. We first ablate our method and demonstrate the con- tributions of EMA and LSR on three most fundamental tasks: classification on ModelNet40 (Wul et al. 2015). part segmentation on ShapeNet (Chang et al. 2015). and semantic segmentation S3DIS (Armeni et al. 2016). Moreover, we compare BiPoint with existing binarization methods where our designs stand out. Besides, BiPointNet is put to the test computational power and achieve extremely high speedup (14.7 xx) and storage saving (18.9x). The details of the datasets and the implementations included in the Appendix E
real-world devices with limited

---

Published as a conference paper at ICLR 2021
Table 2: Comparison of binarization methods on PointNet. EMA is critical; even all meth- ods equipped with EMA, our LSR out- performs others with least number of scaling
Table 3:
Our methods mainstream back- bones. We use XNOR as a strong baseline for comparison. The techniques in our BiPointNet are generic to point cloud learning. Hence, they easily extendable other backbones
factors. OA: Overall Accuracy
Method
Bit-width
Aggr.
Factors
OA
Base Model Method
Bit-width
Aggr.
OA
32/32
MAX
88.2
Full Prec.
32/32
MAX
86.8
Full Prec.
32/32
AVG
86.5
PointNet (Vanilla)
XNOR
1/1
MAX
61.0
1/1
MAX
7.1
Ours
1/1
EMA-max
85.6
BNN
1/1
EMA-avg
11.3
1/1
EMA-max
16.2
Full Prec.
32/32
MAX
88.2
1/1
MAX
10097
7.3
PointNet
XNOR
1/1
MAX
64.9
IR-Net
1/1
EMA-
10097
22.0
Ours
1/1
EMA-max
86.4
1/1
EMA-max
10097
63.5
Full Prec.
32/32
MAX
90.0
1/1
MAX
10097
4.0
Bi-Real
1/1
EMA-avg
10097
77.0
PointNet++
XNOR
1/1
MAX
63.1
1/1
EMA-max
10097
77.5
Ours
1/1
EMA-max
87.8
1/1
MAX
Full Prec.
32/32
AVG
90.0
ABC-Net
1/1
EMA-avg
51
68.9
1/1
EMA-max
51
77.8
PoinCCNN
XNOR
1/1
AVG
83.0
1/1
MAX
18
4.1
Ours
1/1
EMA-avg
83.8
XNOR+
1/1
EMA-avg
73.8
Full Prec.
32/32
MAX
89.2
1/1
EMA-max
18
78.4
DGCNN
XNOR
1/1
MAX
51.5
1/1
MAX
28529
64.9
XNOR
1/1
EMA-avg
28,529
78.2
Ours
1/1
EMA-max
83.4
1/1
EMA-max
28529
81.9
Full Prec.
32/32
90.8
1/1
MAX
4.1
PointConv
XNOR
1/1
83.1
Ours
1/1
EMA-avg
18
82.5
1/1
EMA-max
86.4
Ours
1/1
87.9
4.1
ABLATION STUDY
As shown in Table [ the binarization model baseline suffers catastrophic performance drop in the classification task. EMA and LSR improve performance considerably when used alone, and they further close gap between the binarized model and the full-precision counterpart when used together.
In Figure 4 we further validate the effectiveness EMA and LSR. We show that BiPointNet with EMA has its information entropy maximized during training, whereas the vanilla binarized network with max pooling gives limited and highly fluctuating results. Also, we make use of the regularization loss Lyeg | zzT I| for the feature transformation matrix of T-Net in PointNet as an indicator, the Lreg the BiPointNet with LSR is much smaller than the vanilla binarized network, demonstrating LSR's ability reduce the scale distortion caused by binarization, allowing proper prediction of orthogonal transformation matrices.
Moreover, also include the results of two challenging tasks, part segmentation, and semantic seg- mentation, in Table [ As we follow the original PointNet design for segmentation, which concate- nates pointwise features with max pooled global feature, segmentation suffers from the information loss caused by the aggregation function. EMA and LSR are proven to be effective: BiPointNet is approaching the full precision counterpart with only 4% mloU difference on part segmentation and ~ 10.4% mloU gap semantic segmentation. The full results of segmentation are presented Appendix E.6
4.2
COMPARATIVE EXPERIMENTS
In Table 2 we show that our BiPointNet outperforms other binarization methods such as
BNN (Hubara al. 2016), XNOR (Rastegari al. 2016), Bi-Real eL. 2018), ABC-Net (Lin
8

---

Published as a conference paper at ICLR 2021
PointNet BiPointNet
PaintNet
BiPaintNel
100
90
O ur
140
131.8
Storage Usage (MB)
3.5
3.16
80
XNOR
Time (ms)
120
ABC Net
Bi-Roal IR-Net
100
x2.1x
14.7x
2.5
18.9x
50
80
2
67.3
40
60
30
40
20
BNN
20
10
5.5
0
80
70
80
50
40
30
10
A72
A53
ARM Devices
(ms)
(a)
(b)
(c)
Figure 5: (a) Time cost comparison. Our BiPointNet achieves x speedup ARM A72 CPU device. (b) Storage usage comparison. Our BiPointNet enjoys 18.9 storage saving on all devices. (c) Speed accuracy trade-off plot. We evaluate various binarization methods (with our EMA-max) upon PointNet architecture on ARM A72 CPU device, our BiPointNet is the leading method in both speed and accuracy
ct al. 2017), XNOR+ (Bulat & Tzimiropoulos 2019). and IR-Net (Qin et al. 2020b). Although these methods have been proven effective in 2D vision, they not readily transferable point clouds due to aggregation-induced feature homogenization.
Even if we equip these methods with EMA mitigate information loss, our BiPointNet still performs better. We argue that existing approaches, albeit having many scaling factors, focus on minimizing quantization errors instead of recovering feature scales, which is critical to effective learning on point clouds. Hence, BiPointNet stands out with a negligible increase of parameters that are designed to restore feature scales. The detailed analysis of the performance of XNOR is found in Appendix C Moreover, we highlight that our EMA and LSR are generic, and Table shows improvements across several mainstream categories of point cloud deep learning models, including PointNet (Qi et al., 2017a), PointNet++ (Qi et al. 2017b), PointCNN (Li et al. 2018 DGCNN (Wang et al. 2019a), and PointConv (Wu et al. 2019).
4.3 DEPLOYMENT EFFICIENCY ON REAL-WORLD DEVICES
To further validate the efficiency of BiPointNet when deployed into the real-world edge devices, further implement our BiPointNet on Raspberry Pi 4B with 1.5 GHz 64-bit quad-core ARM CPU Cortex-A72 and Raspberry Pi 3B with 1.2 GHz 64-bit quad-core ARM CPU Cortex-A53.
We compare our BiPointNet with the PointNet in Figure a) and Figure ()). We highlight that BiPointNet achieves 14.7x inference speed increase and 18.9x storage reduction over PointNet, which is recognized as fast and lightweight model itself. Moreover, we implement various bina- rization methods over PointNet architecture and report their real speed performance on ARM A72 CPU device. As Figure 5fc), our BiPointNet surpasses all existing binarization methods in both speed and accuracy. Note that all binarization methods adopt our EMA and report their best aceu- racy, which is the important premise that they can be reasonably applied to binarize the PointNet.
5 CONCLUSION
We propose BiPointNet, the first binarization approach for efficient learning on point clouds. We build a theoretical foundation to study the impact of binarization on point cloud learning models, and proposed EMA and LSR in BiPointNet to improve the performance. BiPointNet outperforms existing binarization methods, and it is easily extendable to a wide range of tasks and backbones, giving impressive 14.7x speedup and 18.9x storage saving resource-constrained devices. Our work demonstrates the great potential of binarization. We hope our work can provide directions for future research.
Acknowledgement This work was supported by National Natural Science Foundation of China (62022009, 61872021), Beijing Nova Program of Science and Technology (2191100001) 11100001119000,,, and State Key Lab of Software Development Environment (SKLS0)--0000ZX000))

---

Published as a conference paper at ICLR 2021
REFERENCES
Iro Armeni, Ozan Sener, Amir R. Zamir, Helen Jiang, Ioannis Brilakis, Martin Fischer, and Silvio Savarese. 3d semantic parsing of large-scale indoor spaces. IEEE CVPR, 2016.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 3308.333,, 2013.
Adrian Bulat and Georgios Tzimiropoulos. Xnor-net++:: Improved binary neural networks. In BMVC, 2019.
Angel X. Chang, T. Funkhouser, L. Guibas, Hanrahan, Qixing Huang, Zimo Li, S. Savarese, M. Savva, Shuran Song, H. Su, J. Xiao, L. Yi, and F. Yu. Shapenet: An information-rich 3d model repository. CoRR, abs/1512.0300 2015.
Ruizhou Ding, Ting- Wu Chin, Zeye Liu, and Diana Marculescu. Regularizing activation distribution for training binarized deep networks. In IEEE CVPR, 2019.
Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. CoRR, htss/1903.02228, 2019.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accu- rate object detection and semantic segmentation. In IEEE CVPR, 2014.
Ross B. Girshick. Fast r-cnn. IEEE ICCV, 2015.
Ruihao Gong, Xianglong Liu, Shenghu Jiang, Tianxiang Li, Peng Hu, Jiazhen Lin, Fengwei Yu, and Junjie Yan. Differentiable soft quantization: Bridging full-precision and low-bit neural networks. In IEEE ICCV, 2019.
Yulan Guo, Hanyun Wang, Qingyong Hu, Hao Liu, Liu, and Mohammed Bennamoun. Deep learning for 3d point clouds: A survey, IEEE TPAMI, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In IEFE CVPR, 2016.
Lu Hou, Quanming Yao, and James T. Kwok. Loss-aware binarization of deep networks. ICLR, 2017.
Qingyong Hu, Bo Yang, Linhai Xie, Stefano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, and Andrew Markham. Randla-net: Efficient semantic segmentation large-scale point clouds. 2020.
Itay Hubara, Matthiuu Courbariaux, Daniel Soudry, Ran El- Danyy,, and Yoshua Bengio. Binarized neural networks. In NeurIPS, 2016.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In NeurIPS, 2012.
Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Di, and B. Chen. Pointenn: Convolution on x-transformed points. In NeurlPS, 2018. 2018.
Xiaofan Lin, Cong Zhao, and Wei Pan. Towards accurate binary convolutional neural network. In NewrlPS, 2017.
Chunlei Liu, Wenrui Ding, Xin Xia, Baochang Zhang, Jiaxin Gu, Jianzhuang Liu, Rongrong Ji, and David S. Doermann. Circulant binary convolutional networks: Enhancing the performance of 1-bit denns with circulant back propagation. In IEEE CVPR, 2019a.
Yongcheng Liu, Bin Fan. Shiming Xiang. and Chunhong Pan. Relation-shape convolutional neural network for point cloud analysis. IEEE CVPR, 2019b.
Yongcheng Liu, Bin Fan, Shiming Xiang. and Chunhong Pan. Relation-shape convolutional neural network for point cloud analysis. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8895-8904, 2019c.
10

---

Published as a conference paper at ICLR 2021
Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Ting Cheng. Bi-real net: Enhancing the performance of 1-bit cnns with improved representational catphaaiiiiaaaaaallllllia capability and advanced training algorithm. In ECCV, 2018.
Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang-Ting Cheng. Reactnet: Towards precise binary neural network with generalized activation functions. In ECCV,
Zhijian Liu, Haotian Tang, Yujun Lin, and Song Han. Point-voxel cnn for efficient 3d deep learning. In NeurlPS, 2019d.
Brais Martinez, Jing Yang, Adrian Bulat, and Georgios Tzimiropoulos. Training binary neural networks with real-to-binary convolutions. In ICLR, 2020.
Charles R. Qi. Hao Su, Kaichun Mo. and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. IEEE CVPR, 2017a.
Charles R. Qi, Li Yi, Hao Su, and Leonidas J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In NeurIPS, 2017b.
Haotong Qin, Ruihao Gong, Xianglong Liu, Xiao Bai, Jingkuan Song, and Nicu Sebe. Binary neural networks: survey, Putlern Recognition, 2020a.
Haotong Qin, Ruihao Gong, Xianglong Liu, Mingzhu Shen, Ziran Wei, Fengwei Yu, and Jingkuan Song. Forward and backward information retention for accurate binary neural networks. In IEEE CVPR, 2020b.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In ECCV, 2016.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale sual recognition challenge. IJCV, 2015.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1409: 1556, 2014.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In IEEE CVPR, 2015.
Hugues Thomas, Charles R. Qi, Jean-EEmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J. Guibas. Kpconv: Flexible and deformable convolution for point clouds. JEEE ICCV, 2019.
Yiru Wang, Weihao Gan, Wei Wu, and Junjie Yan. Dynamic curriculum learning for imbalanced data classification. In IEEE ICCV, 2019a.
Yunhe Wang, Chang Xu, Chao Xu, and Dacheng Tao. Packing convolutional neural networks in the frequency domain. IEEE TPAMI, 2019b.
Ziwei Wang. Ziyi Wu, Jiwen Lu, and Jie Zhou. Bidet: An efficient binarized object detector. In IEEE CVPR, 2020.
B. Wu, Y. Wang, P. Zhang, Yuandong Tian, Vajda, and K. Keutzer, Mixed precision quantization of convnets via differentiable neural architecture search. CoRR,
2018.
Wenxuan Wu, Zhongang Qi, and Fuxin Li. Pointconv: Deep convolutional networks 3d point clouds. IEEE CVPR, 2019.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: deep representation for volumetric shapes. IEEE CVPR, 2015.
Zizhao Wu, Ruyang Shou, Yunhai Wang, and Xinguo Liu. Interactive shape co-segmentation via 2014.
label propagation. Computers & Graphics,

---

Published as a conference paper at ICLR 2021
Bo Xie, Yingyu Liang, and Le Song. Diverse neural network learns true target functions, In Artificial Intelligence and Statistics, 2017.
Chenfeng Xu, Bichen Wu, Zining Wang, Wei Zhan, Peter Vajda, Kurt Keutzer, and Masayoshi Tomizuka. Squeezesegv3: Spatially-adaptive convolution for efficient point-cloud segmentation. In ECCV, 2020a.
Qiangeng Xu, Xudong Sun, Cho-Ying Ying Wu, Panqu Wang, and U. Neumann. Grid-gen for fast and scalable point cloud learning. 2020b.
Yinghao Xu, Xin Dong, Yudian Li, and Hao Su. A main/subsidiary network framework for simpli- fying binary neural networks. In IEEE CVPR, 2019.
Li Yi, Vladimir G Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Sheffer, and Leonidas Guibas. A scalable active framework for region annotation in 3d shape collections. ACM Transactions on Graphics (ToG), 35, 2016.
Haibao Yu, Q. Han, Jianbo Li, Jianping Shi, Guang-Liang Cheng, and Bin Fan. Search what you want: Barrier panelty nas for mixed precision quantization. In ECCV, 2020.
Jianhao Zhang, Yingwei Pan, Ting Yao, He Zhao, and Tao Mei. dabnn: A super fast inference framework for binary neural networks on ARM devices. ACM MM, 2019a.
Xiangguo Zhang, Haotong Qin, Yifu Ding, Ruihao Gong, Qinghua Yan, Renshuai Tao, Yuhang Li, Fengwei Yu, and Xianglong Liu. Diversifying sample generation for data-free quantization. In IEEE CVPR, 2021.
Zhiyuan Zhang. Binh-Son Hua, and Sai-Kit Yeung. Shellnet: Efficient point cloud convolutional neural networks using concentrie shells statistics. IEEE ICCV, 2019b.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv, abs/1606.0616/ 2016.
Feng Zhu, Ruihao Gong, Fengwei Yu, Xianglong Liu, Yanfei Wang, Zhelong Li. Xiuqi Yang, and Junjie Yan. Towards unified int8 training for convolutional neural network. In JEEE CVPR, 2020.
Shilin Zhu, Xin Dong, and Hao Su. Binary ensemble neural network: More bits per network or more networks per bit? In IEEE CVPR, 2019.
12

---

Published as a conference paper at ICLR 2021
APPENDIX FOR BIPOINTNET
A MAIN PROOFS AND DISCUSSION
A.1 PROOF OF ZERO CONDITIONAL ENTROPY
In our BiPointNet, hope that the binarized tensor B reflects the information in the original tensor Y as much as possible. From the perspective of information, our goal is equivalent to maximizing the mutual information I(Y; B) of the random variables Y and B:
arg max I(Y;B) Y,B
(11)
P(Y (v.b)
E DEE.RER
(V. log
(12)
py (y)pbb(b))
Poy.) (v. (y
Σ
(v. log
(13)
DEE
DEYEBEB
Σ
py (v) (og
Σ
P(Y (y. log
(14)
DE).bER
"EEE
Σ Ev)
PBIY= =y (b) logPB\>=
Σ
Σ ((r, logps(b)
(15)
VEY
BEB
bEB
Σ PB Y = v)
Σ PB( log
(16)
yEY
bEB
= -H(B + H(B)
(17)
((BB) - H(B Y),
(18)
where P(Y,B) and PY, ps are the joint and marginal probability mass functions of these discrete variables. H(B) is the information entropy, and H(B Y) is the conditional entropy of B given Y.
According to the Eq. 15 and Eq. (8). the conditional entropy H(Y X) can be expressed as
H(B = Evr(v) Σ PB = beB
log (oggB (0
(19)
DEY
Since we use the deterministic sign function as the quantizer binarization, the value of B fully depends the value of Y, (B)Y= 0 or 1 in Eq. 44, i.e., every value y has fixed mapping to a binary value b. Then we have
H(B (BBYY= Y) Σ (0) .............. +0)
(20)
VEY
Hence, the original objective function is equivalent to maximizing the information entropy H(B):
arg max Hn(B)= -
pn(b) log Pn)).(
(21)
B
bcB
A.2 PROOFS OF THEOREM
Theorem For input X of max pooling & with arbitrary distribution, the information entropy of the binarized output to zero
infinity, i.e. lim HB = 0. And there is constant c, for any n-+x
and nz, if nn > nz G, we have HB,y < HB, where is the number of aggregated elements.
13

---

Published as a conference paper at ICLR 2021
Proof. We obtain the correlation between the probability mass function of input X and output Y max pooling, intuitively, all values are negative to give negative maximum value:
Evr(v) -E px, (ta))
(22)
Since the sign function is applied the quantizer, the Hy( of binarized feature can be expressed as Eq. (6).
(1) When X obeys arbitrary distribution, the probability mass function px, (mis) must satisfies Σ px, (xz) I 1. According to Eq. (6), let
Σ Pxg (xx), we have
lim Hx(Xo) lim -t" log (1 t)" log (1 - 1)" n-000
(23)
1=+00
lim 1*) log nn
lim (*) - a1-00
lim (1 - (1-1)")
log lim (1 - (1-1)))
(24)
0 log 0 - 1 log
(25)
(26)
(2) For any n 2 1, we can obtain the representation of the information entropy H., (X
(X,) = ( px, (1x) log Σ Px Pxx px)())))"
BeCU
(27)
(1 (xxx((())) px, (x)(oso))) log
(1-(En
Σ px. (x))
Let pr. = (E) 12 the HB,((Pn) can be expressed as
(B. = -P log - (1 - Pn) log(1 - pn).
(28)
and the derivative H H.. (pn)
d Hn,)(pp)) = log 1 Pn
(29)
ppp(P) (pn)
Pr
the HB. (Px) maximized when Pn. takes 0.5, and positive correlation with Pr, when Pr. < 0.5
since the (pa)) >> 0 when Pa. < 0.5.
pp(p..
Therefore, when the constant satisfies Pe = (EEscoox cx, ( 2 0.5, given the n1 > n2 > c,
we have Pn, < Png < Pe, and HB, (Xx) < HB,,, (Xx) < HX,.((x)).
A.3 PROOFS OF PROPOSITION
Proposition 1 When the distribution of the random variable Y satisfies Py =
0.5, the information entropy Hi is maximized. PY (y)= =
0.5,
Proof. According to Eq (5) we have
Hn(B) Σ (b(b) log
(30)
BEB
= =-pe( p (-1 (1)(1) log (og
(31)
--Pa( log (-1) -pp( log (1 p=(-1)))
(32)
14

---

Published as a conference paper at ICLR 2021
Then get the derivative of (B( with respect to pB(-1)
HB((
PB(-1)
p
(-1)
(logpa(-1)+
+
[og - -pp(-1)
d pn((1)
pn( pn(-1)
(1 pn( (1)) In
(33)
1
- logpp(-1) + log pB (-1))))
(34)
1n2
In
(- (1-Ps(-1))
=log
(35)
PB (-(
When we let (B) = 0 to maximize the Hy( (B), we have ps(-1) = 0.5. Sine the deterministic
pg(
sign function with the zero threshold is applied as the quantizer, the probability mass function of B represented
Σ (y) dy, if b
PB =
y
(36)
Σ (y) dy, if b = 1,
120
and when the information entropy is maximized, we have
Σ
py dy 0.5.
(37)
V<D
A.4 DISCUSSION AND PROOFS OF THEOREM
The bi-linear layers are widely used in our BiPointNet to model each point independently, and each linear layer outputs an intermediate feature. The calculation of the bi-linear layer is represented as Eq. (2). Since the random variable B is sampled from Bw or B obeying Bernoulli distribution, the probability mass function of B can be represented as
P:
b=
(b(
(38)
P.
if b=
where p is the probability of taking the value +1. The distribution output can be represented by the probability mass function of Bw and Ba.
Proposition 2 In bi-linear layer, for the binarized weight Bw E (-1, +1) [11,+1]mxt mxk and activation
Ba {-1, + +1] mm with probability mass function PB (1) = Pw and PB. (1) Pa. the probability
mass function for the distribution of output Z can be represented as pz (2i- m) C (1- - Pw
Pa 2pwPa) (p + Pa ) m-- 1,.
Proof. To simplify the notation in the following statements, we define A = B and W Bw Then, for each element Z,, in output {-1,+1]mxx we have
m
A Wk.j-
(39)
Observe that Ar is independent to Way and the value of both variables are either -1 or +1. Therefore, the discrete probability distribution of A Way be defined as
PwPa + (1 pw) (1 Pa).
=
p(x)=
Pm (1 pa) + (1 pvo) Pas =
(40)
0,
otherwise.
15

---

Published as a conference paper at ICLR 2021
Simplify the above equation
Pw Pa Pa PproPa
if if x = otherwise.
p(x)
Pw + Po Pa 0,
(41)
Notice that Fi can be parameterized as binomial distribution. Then we have
Pr(zi.j =1 (m 1)) C² 11-p Pw Pa + 2p (pw + Pa 2pwPa)
(42)
Observe that pz obeys the same distribution wiz-- Finally, have
mm) pz) C (1 Pw Po + PP (pw + +o -PpuPo) T MiE 20.,,,,...,, m..
(43)
Proposition 2 shows that the output distribution of the bi-linear layer depends the probability mass functions of binarized weight and activation. Then we present the proof's of Theorem
Theorem 2
When we let pB_ (1) 0.5 and pB, (1) 0.5 in bi-linear layer to maximize the mutual
information, for the binarized weight Bw E {-1,+11mmx mmxk and activation Bg E {-1,+1]axmn the probability mass function for the distribution of output Z can be represented pz (2i m) 0.5 Cm. 20,1,2,..., m). The distribution of output is approximate normal distribution
(0.
Proof. First, we prove that the distribution of Z can approximated a normal distribution. For bi-linear layers in our BiPointNet, all weights and activations are binarized, which can be represented as Bw and Ba, respectively, And the value of an element ((I.f) Z can expressed as
(Der(1,)) x ba(k.)))
((())
and the value of the element (2,8) x ba(k.)) can be expressed as
11, if (ro(e V ba(k.)) (-1, if bwotk)) V bo(k.)) -1
(wi( ba(k..)
(44)
The (a(a,)) X (b.)).)) only can take from two values and its value can be considered as the result of one Bernoulli trial. Thus for the random variable Z sampled from the output tensor Z, the probability mass function, pz can be expressed as
Pz m) C 1 p-)
(45)
where Pe denotes the probability that the element (i.A) ba(k.j) takes 1. Note that the Eq. 455
completely equivalent to the representation in Proposition 2 According to the Moivre- Laplace
theorem, the normal distribution (u,, o²) can be used as approximation of the binomial distri- bution under certain conditions, and the pz (2i - m) can be approximated as
(2 m)) Ci, pt pe) n
1
(k- (k-mpe) (1
(46)
v /2mmp (1 pe)
and then, get the mean = and variance vm of the approximated distribution / Now we give proof of this below.
with the help of equivalent representation of pz in Proposition 2
According to Proposition 2 when Pio Pa 0.5, rewrite the equation
16

---

Published as a conference paper at ICLR 2021
(2 - m) = 0.5 20.,,2...,,m..
(47)
Then we move to calculate the mean and standard variation of this distribution. The mean of this distribution is defined as
pPz)) E m 20,1, m).
(48)
By the virtue binomial coefficient, we have
(2i C, + (2(m - mm)..."mmm = 0.5 0.5" C + (m 21) 2i)
(49)
0.5 ((2i m m) + (m 2i)
(50)
O.
(51)
Besides, when is even number, have (2i - m)0.5 C 0, These equations prove the symmetry of function (2i mm)05 C Finally, have
u(p2) = Σ (2i - m)0.5" E 10,1,2,...
(52)
Σ ((2) m)).5" + (2(m i) 10m)).5CCmmm)) 10.1.2.... m
(53)
(54)
The standard variation of pz is defined
(pz))
2i ml 0.55
(55)
(4i2 4im + m²)0.55 O.5Th...
(56)
0055m (+> C 4m Σ iC + m² Σ C
(57)
To calculate the standard variation of pz, we use Binomial Theorem and have several identical equations:
Σ cr (1 1)th = m
(58)
Σ Dick m(1 1)m-1 m22-1
(59)
Σ PC =mmm + 1)(1 + 1)m-2 = m(m + 1)m2m-2
(60)
These identical equations help simplify Eq. 57
(pz))
0.5m 4 Σ iC m - 4m Σ iC + +m² Σ Ci
(61)
0.5th (4m (m 1)2m 2m + mmm
(62)
= 00.5"'((m) + m)2m 2m + mm
(63)
(64)
Vm.
(65)
Now proved that, the distribution of output is approximate normal distribution A/ m).
17

---

Published as a conference paper at ICLR 2021
-MAX A.5 DISCUSSION OF THE OPTIMAL 8 FOR EMA-MAX
When the X (0, 1), the objective function of EMA-max to obtain optimal 8* is represented Eq. (8) It is difficult to directly solve the objective function. To circumvent this issue,
Monte Carlo simulation to approximate the value of the optimal max as shown Algorithm 1
Algorithm Monte Carlo Simulation for EMA-max
Input: The number of points to be aggregated; the number of simulations m (c.g. 10000)
Output: Estimated optimal iiax for EMA-max
1: Creating an empty list (represents elements sampled form distribution of aggregated feature)
2: for i = 0 to do
3: Creating an empty list T, (representing one channel of input feature)
4: for j == 0 do
5: Sampling an element eij from the distribution, (00, 1)
6: Adding the sampled element Cij to the list T;
7: end for
8: Adding an element represents the aggregated feature MAX(T:) to F
9: end for
Estimating the optimal max 8max Median F)) (follow Proposition
A.6 DISCUSSION OF THE OPTIMAL 5 FOR EMA-AVG
When the X ~~( 1). the Y (6 !) and the objective funetion of EMA-avg for obtaining optimal Sivg can be represented
arg max -
1
(sa-0)²
log
Σ
(16-0)²
n2 V=
20 n1
n-
22
Tip <0
(66)
Σ
1
(1o-))²
log
(15-0)
n2
2tT
n1 1
F.20
Ee 20
The solution of Eq. (66) expressed as 8 we thus obtain davg = 0. This means the solution not related to
B IMPLEMENTATION OF BIPOINTNET ON ARM DEVICES
B. OVERVIEW
We further implement our BiPointNet on Raspberry Pi 4B with 1.5 GHz 64-bit quad-core ARM Cortex-A72 and Raspberry Pi 3B with 1.2 GHz 64-bit quad-core ARM Cortex-A53, and test the real speed that one can obtain in practice. Although the PointNet is a recognized high-efficiency model, the inference speed of BiPointNet is much faster. Compared PointNet, BiPointNet enjoys up to 14.7 x speedup and 18.9x storage saving.
We utilize the SIMD instruction SSHL ARM NEON make inference framework daBNN (Zhang et al., 2019a) compatible with our BiPointNet and further optimize the implementa- tion for more efficient inference.
B.2 IMPLEMENTATION DETAILS
Figure 6 shows the detailed structures of six PointNet implementations. In Full-Precision version (a), BN is merged into the later fully connected layer for speedup. which is widely chosen for deployment in real-world applications. In Binarization version (b)(c)d)(e), we have keep BN unmerged due to the binarization of later layers. Instead, we merge the scaling factor of LSR into BN layers. The HardTanh function is removed because it does not affect the binarized value of
18

---

Published as a conference paper at ICLR 2021
input for the later layers. We test the quantization for the first layer and last layer in the variants (b)(c)d)(e). In the last variant(f), we drop the BN layers during training. The scaling factor is ignored during deployment because it does not change the sign of the output.
Fall Precision FC
Binarization
Max Pooling
Input Points
n5.3
Bx3
B33
BA3
Linear 3x64
Linear 64x128
Linear 128x1024
Max Pooling
Linear 1024x512
Linear 512x256
Linear 256x40
Output
Class
Class
Class
Class
Class
Class
(#)
(b)
(c)
(d)
(6)
(
Figure 6: Structures of different PointNet implementations. Three fully connected layers are used in all six variants: Full Precision FC, Binarization FC with BN, Binarization FC w/o BN. Full Precision FC contains a full precision fully connected layer and a ReLU layer. Original BN is merged into the later layer. Binarization with BN also contains two layers: quantized fully connected layer and a batch normalization layer. Binarization FC w/o BN is formed by a single quantized fully connected layer
B.3 ABLATION ANALYSIS OF TIME COST AND QUANTIZATION SENSITIVITY
Time & Speedup Ratio
Setup
Bit-width
L.
BN
OA
Storage & Saving Rutio
A72
A53
32/32
32/32
32/32
Merged
3.16MB 1.0x
131ms 1.0x
67ms / 1.0x
Merged
5.5ms
(c)
32/32
1/1
Not Merged
84.60
0.12MB / 2
9.0mss 14.7x
5.3ms/ 12.6x
32/32
Not Merged
5.31
0.16MR 19.7x
11.5ms/ 11.4x
6.5ms 10.3
(c)
1/1
1/1
1/1
Not Mergod
4.86
0.12MB
26. x
11.4ms 11, 11.5x
6.4ms 10 10.4x
16.1x
4.8ms 13.9x
Table Comparison different configurations in deployment on ARM devices. The soorage-saiing ratio and speedup ratio are calculated according to the full precision model as the first row illustrates. the models use PointNet as the base model and EMA-max as aggregation function. The accuracy performance is reported on the point cloud classification task with the ModelNer40 dataset. FL: First Layer; LL: Last Layer
Table shows the detailed configuration including overall accuracy, storage usage, and time cost of the above-mentioned six implementations. The result shows that binarization of the middle fully connected layers can extremely speed up the original model. We achieve 18.9x storage saving, 14.7x speedup on A72, and 12.1x speed on A53, The quantization of the last layer further helps save storage consumption and improves the speed with a slight performance drop. However, the quantization of the first layer causes a drastic drop in accuracy without discernible computational cost reduction. The variant (f) without BN achieves comparable performance with variant (b). It suggests that our LSR method could be an ideal alternative to the original normalization layers to achieve fully quantized model except for the first layer.
19

---

Published as a conference paper at ICLR 2021
C COMPARISON BETWEEN LAYER- LAYER-WISE SCALE RECOVERY AND OTHER METHODS
In this section, we will analyze the difference between the LSR method with other model binarization methods. Theorem 2 shows the significance of recovering scale point cloud learning. However, IRNet and BiReal only consider the scale of weight but ignore the scale of input features. Therefore, these two methods cannot recover the scale of output due to scale distortion on the input feature. A major difference between these two methods is that LSR opts for layer-wise scaling factor while XNOR opts for point-wise one. Point-wise scale recovery needs dynamical computation during inference while proposed LSR only has layer-wise global scaling factor, which is independent of the input. As a result, our method can achieve higher speed in practice.
10a
10
Information Entropy
10-2
10-3
10
BNN
10-5
XNOR
BiPointNet
0
25
50
75
100
125
150
175
200
epoch
Figure 7: The information entropy of BNN, XNOR and our BiPointNet
ssoos that XNOR can alleviate the aggregation-indueed feature homogenization. The point- wise scaling factor helps the model to achieve comparable adjustment capacity as full-precision linear layers. Therefore, although XNOR suffers from feature homogenization at the beginning of the training process, it can alleviate this problem with the progress of training and achieve acceptable performance, as shown in Figure
D COMPARISON WITH OTHER EFFICIENT LEARNING METHODS
We compare computation speedup and storage savings with several recently proposed methods to accelerate deep learning models on point clouds. Note that the comparison for reference only; tests conducted different hardware, and for different tasks. Hence, direct comparison cannot give any meaningful conclusion. In Table we show that BiPointNet achieves the most impressive acceleration.
E EXPERIMENTS
E. DATASETS
ModelNet40: ModelNet40 (Wu et al. 2015) for part segmentation. The ModelNet40 dataset is the most frequently used dataset for shape classification. ModelNet is a popular benchmark for point cloud classification. It contains 12,311 CAD models from 40 representative classes of objects.
20

---

Published as a conference paper at ICLR 2021
Table Comparison between BiPointNet and other approaches efficient learning on point clouds. Grid-GCN (Xu et al. 20206) leverages novel data structuring strategy; RAND-LA Net (Hu et al.) 2020) designs a faster sampling method; PointVoxel (Liu et al.) (001d)) proposes an efficient repre- sentation. These works, albeit achieving high performance, are not as effective our binarization method in terms model acceleration. The asterisk indicates the vanilla version
Metric/
Method
Hardware
Dataset
Base Model
Speedup
Performance
BiPointNee
ARM Cortex-A72
ModelNet4
PointNet*
OA/85.6
12.1 x
BiPointNet
ARM Cortex-A53
ModelNet40
PointNet*
OA/85.6
14.7x x
Grid-GCN
RTX 2080 GPU
S3DIS
PointNet
mloU/53.2
1.62x
RandLA-NNet
RTX 2080Ti GPU
S3DIS
PointNer*
mloU/70.0
1.04x
PointVoxel
GTX 1080Ti GPU
ShapeNet
PointNet
mloU/46.9
2.46
ShapeNet Parts: ShapeNet Parts (Chang et al. 2015) for part segmentation. ShapeNet contains
16,881 shapes from 16 categories, 2,048 points are sampled from each training shape. Each shape split into two to five parts depending on the category, making up to 50 parts in total.
S3DIS:
S3DIS for semantic segmentation (Armeni et al.) 2016). S3DIS includes 3D scan point clouds for 6 indoor areas including 272 rooms in total, each point belongs to of 13 semantic categories. We follow the official code eQi al. 2017a) for training and testing.
E.2 IMPLEMENTATION DETAILS OF BIPOINTNET
We follow the popular PyTorch implementation of PointNet and the recent geometric deep learning codebase (Fey & Lenssen 2019) for the implementation of PointNet baselines. Our BiPointNet built by binarizing the full precision PointNet. All linear layers in PointNet except the first and last one are binarized to bi-linear layer, and we select Hardtanh as our activation function instead of ReLU when we binarize the activation before the bi-linear layer. For the part segmentation task, we follow the convention (Wu et al., 2014 Yi ct al. 2016) to train a model for each the 16 classes. We also provide our PointNet baseline under this setting.
Following previous works, we train 200 epochs, 250 epochs, 128 epochs on point cloud classifica- tion, part segmentation, semantic segmentation respectively. To stably train the binarized models, we use learning rate of 0.001 with Adam and Cosine Annealing learning rate decay for all binarized models on all three tasks.
E.3 MORE BACKBONES
We also propose four other models: BiPointCNN, BiPointNett+, BiDGCCN, and BiPointConv, which are binarized versions of PointCNN (Li et al. 2018). PointNet++ (Qi et al., 2017b) DGCNN (Wang et al. 2019a), and PointConv (Wu et al. 2019), respectively. This is attributed to the fact that all these variants have characteristics in common, such as linear layers for wise feature extraction and global pooling layers for feature aggregation (except PointConv, which does not have explicit aggregators). In PointNet++, DGCNN, and PointConv, we keep the first layer and the last layer full-pree full-precision and binarize all the other layers. PointCNN, we keep every first layer of XConv full precision and keep the last layer of the classifier full precision.
E.4 BINARIZATION METHODS
For comparison, we implement various representative binarization methods for 2D vision, including BNN (Hubara et al. (016). XNOR-Net (Rastegari et al. 2016). Bi-Real Net (Liu et al., 2018). XNOR+ (Bulat & Tzimiropoulos 2019). ABC-Net (Lin et al. 2017). and IR-Net (Qin et al. 2020b), to be applied on 3D point clouds. Note that the Case version of XNOR+ used in our experiments for a fair comparison, which applies layerwise learnable scaling factors to mini- implemented according their open-source code
mize the quantization error. These methods or the description their papers, and we take reference of their 3x3 convolution design when im- plementing the corresponding bi-linear layers. We follow their training process and hyperparameter
21

---

Published as a conference paper at ICLR 2021
settings, but note that the specific shortcut structure in Bi-Real and IR-Net is ignored since it only applies to the ResNet architecture.
E.5 TRAINING DETAILS
Our BiPointNet is trained from scratch (random initialization) without leveraging any pre-trained model. Amongst the experiments, we apply Adam our optimizer and use the cosine annealing learning rate scheduler to stably optimize the networks. To evaluate our BiPointNNe on various network architectures, we mostly follow the hyper-parametee settings of the original papers (Qi
al. 2017a Li al. 2018 Qi al. 2017b Wang al. 2019a).
E.6 DETAILED RESULTS OF SEGMENTATION
We present the detailed results of part segmentation on ShapeNet Part in Table 6 and semantic segmentation on S3DIS in T The detailed results further prove the conclusion of Section 4.1 EMA and LSR improve performance considerably in most of the categories (instead of huge performance in only few categories). This validates the effectiveness and robustness of our method.
22

---

Published as a conference paper at ICLR 2021
table
81.5
81.8
66.8
63.3
50.30
61.4
79.7
79.5
skate
board
5271
70.6
72.2
43.7
43.7
43.7
43.7
59.1
56.9
rocket
152
54.5
56.2
31.3
25.7
26.8
36.2
47.2
47.7
pistol
66
80.9
81.6
57.5
36.3
25.2
60.2
75.9
76.3
mug
283
93.5
93.8
68.3
48.8
49.0
51.0
91.8
91.6
Table 6: Detailed results of our BiPointNet for part segmentation on ShapeNet Parts.
motor
184
64.4
61.7
23.1
16.9
16.9
25.4
42.7
44.8
laptop
202
95.6
95.5
90.1
81.0
48.6
93.8
94.8
95.1
lamp
451
81.9
80.8
48.6
53.0
50.8
50.0
76.0
74.9
knife
1547
85.6
84.3
62.2
50.0
42.6
75.5
82.8
83.0
guitar
392
91.0
48.8
44.0
33.5
47.8
84.1
87.4
ear
phone
787
70.2
73.1
57.1
56.2
38.2
46.0
65.6
68.5
chair
69
90.8
90.0
55.8
58.2
61.3
75.3
87.7
88.6
car
3758
76.8
76.3
26.5
24.1
24.1
39.2
66.1
67.4
cap
898
92.5
90.8
65.5
57.5
44.6
80.2
85.5
86.1
bag
55
79.4
78.5
48.1
46.5
46.2
46.2
71.9
69.7
aero
76
83.6
83.4
35.1
39.8
37.9
41.5
79.3
79.5
mean
2690
84.3
84.0
54.0
53.0
47.3
58.7
80.3
80.6
ema-ave
cma ema-avy avg
ema- ema-max
max
aggr.
avg
max
max
shapes
BNN
BNN
BNN
LSR
Ours
Ours
FP
23

---

Published as a conference paper at ICLR 2021
ceiling floor wall beam column window door table chair sofa bookease board clutter
loU
45.2
41.8
2.1
0.6
33,9
IoU
43.4
38.7
18.5
26.2
0
0
0
loU
44.4
40.4
30.5
36,9
IoU
51.3 14.5
9.4
0
0
5.8
9.6
IoU
47.8
7.7
9.3
0
40.6
IoU
55.8
53.8
0.6
43.0 43.5
45.7
0
0
IoU
58.0
53.8
0
0
0
0
48.0
IoU
52.0
53.5
39.6
43.0
0
0
0
0
IoU
34.6
33.7
19.0
23.8
Table Detailed results of our BiPointNet for semantic segmentation on S3DIS.
IoU
50.3
46.1
84.6 60.5 32.0
62.6 34.5
IoU
93.8 72.8
71.7
28.1
28.1
32.1
43.5 15.9
0
IoU
89.1
40.6
35.1
0
86.1
IoU
91.1
90.1
45.5
51.5
50.8
25.4
84,6
85.1
(mloU/acc.) (mloU/acc.) (mloU/ace.) (mloU/acc.) (mloU/acc.) (mloU/cc..)
70.0/89.2
68 05..2/88.3
9.8/416
8.6/11.6
7.2/5.3
1.9/24.5
53.3/79.8
57.9/82.3
area6
45.3/83.3
42.0/81.2
9.5/45.8
11.4/54.8
7.6/9..2
2.0/25.8
34.7/76.1
36.9/77.6
area5
45.0/82.4
43.8/81.2
9.3/42.5
9.8/46.0
7.8/39.7
1.8/22.8
34.2/73.2
36.6/73.9
area4
62,4/88.0
61.2/87.2
8.3/419
7.1/36.5
7.1/66.8
2.0/25.7
48.0/79.9
53.4/82.4
area3
38 08.0/76.8
35 35.4/72.4
9.8/50.5
11.2/51.2
10.1/54.4
2.1/27.0
31.0/70.3
area.2
29.
61,7/86.2
59,9/84.6
9.6/44.0
7.6/36.6
7.7/4.0
2.0/66.0
47.1/75.8
50.9/78.3
areal
overall overall overall
acc.
83.5
81.5
45.0
46.8
47.2
25.4
74.9
76.7
mloU
54.4
51.5
9.5
9.9
8.5
2.0
40.9
44.3
BNN ema-max
ema-ave
ema. max
aggr
avg
max
method
BNN
BNN
LSR
Ouss
Onrs
FP
FP
24


Reviews:
[{"review_id": "wm1jqoskSUm", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method for binarization of neural networks of 3d point clouds. Two modules of entropy maximum aggregation and layer-wise scale recovery are proposed to conquer the problems of discrimination loss induced by feature homogenization and scale imbalance, which are caused by model binarization. The authors provide theoretical analysis about the proposed method. Experiments on various backbones and tasks demonstrate the effectiveness of the proposed method. A practical implantation of BiPointNet on ARM also demonstrates significant speedups over PointNet and large memory savings.\n\nStrength:\n1. Binarization of CNN models designed for 2D images has been studied in the past years, this paper extends this problem into 3D point cloud models. The authors show that the existing methods for binarization of 2D CNN models can not work well on this new problem. \n2. For this new problem, the authors analysis its performance degradation based on PointNet and proposed effective solutions.\n3. Experiments on several tasks show that the proposed method can obtain highly compact models with acceptable accuracy degradation. Experiments on other backbones also show that the proposed method is general, although its analysis is based on PointNet.\n\nFor the weakness, I only have some minor comments.\n1. The discussion on related work could be enlarged. For example, the following papers are well known point cloud networks proposed recently\n(a) PointConv: Deep Convolutional Networks on 3D Point Clouds. CVPR 2019\n(b) Relation-Shape Convolutional Neural Network for Point Cloud Analysis. CVPR 2019\n(c) ShellNet: Efficient Point Cloud Convolutional Neural Networks using Concentric Shells Statistics. ICCV 2019\n\nMixed precision quantization is also an active direction after binarization of neural networks, it could also be mentioned as a possible improvement in the future.\n(d) Mixed Precision Quantization of Convnets via Differentiable Neural Architecture Search. ICLR 2019\n(e) Search What You Want: Barrier Panelty NAS for Mixed Precision Quantization. ECCV 2020.\n\n2. Some references miss publication type, i.e.,  conference or journal and where they are published.\n3. Figure 1 can be improved. It is unclear how LSR works in the whole framework.", "extra_scores": {}, "extra_text": {}}, {"review_id": "YcjiPdc21k9", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method to apply binary networks on point clouds. To my knowledge this is the first time that this is attempted so unquestionably the topic of the paper is interesting. From what the authors show a vanilla BNN (XNOR-Net) applied to point clouds does not give very good results and for this reason the authors identify solutions that boil down to applying a shift and a scaling. This is really my main problem with the paper: the proposed methods are too simple and the accompanying theory does not look to be so convincing in order to theoretically support the contributions which are just a simple shift and scaling. Actually the authors show that one of the variants of their method  can be reduced to average pooling which does not require some sophisticated explanation to convince the reader why it works.\n\nMoreover the proposed learnable layer-wise scaling factor is not new and was previously usedat a) layer-level ( Towards Accurate Binary Convolutional Neural Network, Lin et al, NeurIPS’17) and b) channel-level (XNOR-Net++: Improved Binary Neural Networks, Bulat&Tzimiropoulos, BMVC’19). In fact, the problem itself is known since at least 2016, where in the (XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks, Rastegari etal, ECCV’16) identifies this problem and proposes an analytically-computed scaling factor.\n\nOther issues:\n“even global pooling provides strong recognition performance. However, this practice poses challenges for binarization” – there is no justification provided of why pooling may pose issues for binarization. Avg and max-pooling is used with success in contemporary binary networks. \n\nExisting BNNs “are not readily transferable to point clouds.” – In the paper it is mentioned that this is shown and evaluated in the method section. However, many of the listed methods are not in fact evaluated. This is especially important for methods that also learn to recover the scaling factors.\n\nGiven that for image classification, at least for ResNet the last layer before the linear classifier is a global pooling operation, how does the proposed EMA changes the results when applied to image classification, on a ResNet18 on Imagenet? Are there any improvements measurable in that case too?\n\n“Despite that model binarization has been studied extensively in 2D vision tasks (Krizhevsky et al.,2012; Simonyan & Zisserman, 2014; Szegedy et al., 2015; Girshick et al., 2014; Girshick, 2015;Russakovsky et al., 2015; Wang et al., 2019b)\" – the cited works don’t support the author statement, since none them perform binarization. \n\n$\\textbf{Final Rating}$\n\nBased on the authors' responses during the rebuttal period, I don't believe that the paper makes a sufficient contribution for ICLR. Hence I will stick to my original score.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "SzkAQRv4OiE", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method for learning binary neural networks on point cloud inputs. They provide an entropy analysis of the binarized distributions as well as an offset transform to maximize entropy. Then they analyze the scale of binary activations and propose a learnable scaling to reduce the effects of scale distortion. They show that their method is competitive with other binary neural network methods and even full precision methods. Finally they show performant speed and storage results on a Raspberry Pi.\n\nStrengths:\n- Strong empirical results backed by theoretical analysis\n- Experiments are comprehensive and show competitive results on accuracy and speed\n\nConcerns:\n- If speed vs accuracy is the main trade-off, I would like to see a more thorough evaluation of all the models and baselines on a speed vs accuracy tradeoff plot\n\nGiven the strong empirical and theoretical results, I would recommend an accept. I would still like to see the authors strengthen their paper with a more detailed speed/accuracy trade off.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "P1whqbQiFM", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a binarization approach for efficient deep learning on point clouds, called BiPointNet. The authors claim that the immense performance drop of binarized models is caused by the aggregation-induced feature homogenization and scale distortion. The authors propose Entropy-Maximizing Aggregation(EMA) and Layer-wise Scale Recovery(LSR) to reduce the side-effects of binarization. Experiment results demonstrate that the proposed BIPointNet is able to achieve state-of-the-art results and gives an impressive speedup and storage saving. \n\nBesides the major contribution of the paper, the writing of the paper is concise and the illustrations are clear. However, the paper mainly focuses on PointNet-kind of structure, such as PointNet++, and DGCNN, etc. It would be better if the authors could give more discussion on the application of the EMA and LSR on more advanced methods, such as KPConv and PointConv, etc.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose techniques to deal with binarization of 3D point clouds and propose EMA and layer wise scale recovery that improve results across the board for PointNet style models.
An accept.
Decision: Accept (Poster)