BiMatting: Efficient Video Matting via Binarization
Authors: Haotong Qin
Lei Ke
Xudong Ma
Martin Danelljan
Yu-Wing Tai
Chi-Keung Tang
Xianglong Liu
Fisher Yu
Venue: NeurIPS 2023 poster (2023)
Keywords: Video Matting
Model Binarization
Deep Learning
TL;DR: We propose BiMatting, an accurate and efficient video matting model using binarization.

Abstract: Real-time video matting on edge devices faces significant computational resource constraints, limiting the widespread use of video matting in applications such as online conferences and short-form video production. Binarization is a powerful compression approach that greatly reduces computation and memory consumption by using 1-bit parameters and bitwise operations. However, binarization of the video matting model is not a straightforward process, and our empirical analysis has revealed two primary bottlenecks: severe representation degradation of the encoder and massive redundant computations of the decoder. To address these issues, we propose BiMatting, an accurate and efficient video matting model using binarization. Specifically, we construct shrinkable and dense topologies of the binarized encoder block to enhance the extracted representation. We sparsify the binarized units to reduce the low-information decoding computation. Through extensive experiments, we demonstrate that BiMatting outperforms other binarized video matting models, including state-of-the-art (SOTA) binarization methods, by a significant margin. Our approach even performs comparably to the full-precision counterpart in visual quality. Furthermore, BiMatting achieves remarkable savings of 12.4$\times$ and 21.6$\times$ in computation and storage, respectively, showcasing its potential and advantages in real-world resource-constrained scenarios. Our code and models are released at https://github.com/htqin/BiMatting .

BiMatting: Efficient Video Matting via Binarization
Haotong Qin Lei Ke* Xudong Ma', Martin Danelljan?, Yu-Wing Tai",
Chi-Keung Tang'. Xianglong Liu Fisher Yu?
'Beihang University 2ETH Zürich *HKUST *Dartmouth College
Abstract
Real-time video matting edge devices faces significant computational resource constraints, limiting the widespread use of video matting in applications such as online conferences and short-form video production. Binarization is powerful compression approach that greatly reduces computation and memory consumption by using 1-bit parameters and bitwise operations. However, binarization of the video matting model is not straightforward process, and our empirical analysis has revealed two primary bottlenecks: severe representation degradation of the encoder and massive redundant computations of the decoder. To address these issues, we propose BiMatting, an accurate and efficient video matting model using binarization. Specifically, we construct shrinkable and dense topologies of the binarized encoder block enhance the extracted representation. We sparsify the binarized units to reduce the low-information decoding computation. Through extensive experiments, we demonstrate that BiMatting outperforms other binarized video matting models, including sfae-of-the-aa (SOTA) binarization methods, by a significant margin. Our approach even performs comparably to the full-precision counterpart in visual quality. Furthermore, BiMatting achieves remarkable savings of 12.4x and 21.6x in computation and storage, respectively, showcasing its potential and advantages in real-world resource-constrained scenarios. Our code and models are released at https//gggtubb.co//tggi////i///t
1 Introduction
200
RVAA-NNI
The success of deep neural networks has led remarkable
advancements in computer vision tasks, including video mat- ting [1, 2. 3, 4, 5, 6, 1, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]. How- ever, many practical applications based on deep networks re- quire real-time processing with minimal latency, which chal- lenging due to the high computational and storage demands. To address this issue, researchers have developed lightweight video matting algorithms such as Robust Video Matting (RVM) [1] and BGMv2 [10]. While these methods achieve significant speedups and memory reductions, they still rely on expensive floating-poin operations, leaving room for further compression.
Moan Absolutte Difference
(-40)
BCM-
10. A--MMB
(38-8))
RUM
223MB
(1
10P
10'
10"
10
FLOP&
Figure 1: BiMatting enjoys impres- sive computation/sttrage savings while surpassing SOTA 1-bit and some 32-bit models in accuracy.
One of the most promising approaches to improving the effi- ciency of neural networks is network binarization [17, 18, 19, 20, 21, 22, 23, 24, 25, 26]. Binary neural networks (BNNs) have emerged as a highly effective technique for optimizing neural networks by reducing parameter bit width to 1 1-bit. BNNs leverage compact binary parameters that occupy less memory space and use efficient bitwise operations, which are much less computation- ally expensive than floating-pont -painn operations. By employing the binarization approach, researchers can significantly reduce the computational and storage demands of video matting applications.
"qqual contribution
corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Deveuret
Doakkr
SBB 641
SBB
CLICK
SAB
Input
Output
Shriakable Rimarteed Blookh (SRR)( (SRR)
Sparse-Essisted Rinorication (SAR)
Figure 2: BiMatting overview. We apply Shrinkable Binarized Block (SBB) for the accurate encoder and Aparse-Assisted Binarization (SAB) for the efficient decoder. Arrows - indicate the flow of features, # Ø1 and avg indicate concatenation. splitting, and averaging of features, respectively.
Despite that the generic binarization methods have been studied extensively by the binarization community, the direct binarization for existing video matting models still leaves severe accuracy and efficiency bottlenecks. First, to construct an efficient architecture, many video matting methods contain encoders with lightweight full cell-precison backbones to obtain intermediate features. For example, MobileNet architectures are widely applied as lightweight extractors in several video matting models to provide strong features [1, 11J. However, this practice introduces an accuracy bottleneck in the binarized video matting model. Our analysis shows that the computing units and topology of existing lightweight encoder architectures are unfriendly to binarization, leading to severe degradation in the quality of intermediate features extracted by its binarized counterpart, Second, we identify that the architecture of the existing video matting decoder [i] causes immense redundant computation at the multi-scale decode stage. Intensive computation is still needed in large portions of certain foreground or background regions (spatial regions outside unknown parts of the trimap) of the righ-resolution feature maps. This weakens the savings brought by binarization, which hinders efficiency improvements in the context of extremely compressed bit- bidth..th.
In this paper, we provide empirical studies of the above-mentione bottlenecks. This leads to propose BiMatting, Binarized model for accurate and efficient video Matting (overview Fig. 2). To tackle the accuracy bottleneck, we first investigate the limitations of existing binarized encoders in representation extraction. Then, we propose Shrinkable Binarized Block (SBB), which follows iitariizationfrrenndd computation-dense paradigm to construct flexible block structure. SBB effectively extracts high-quality features with improved topology and operators. In addition, given reliable binarized encoder, we further develop Sparse-Assisted Binarization (SAB) to effectively reduce the computational consumption of the binarized decoder. The overall burden is thus greatly reduced. SAB removes repeated computation by spatial sparseness [27] while preserving the accuracy of binarized units, resulting in a notable computation reduction without compromising performance.
Our BiMatting is the first binarization solution for video matting tasks, which surpasses 1-bit matting models using existing binarization algorithms by significant margin [21, 17, 23, 18]. Notably, BiMatting even outperforms some full- full-precision models [28, ]] in terms of accuracy while being highly efficient. Our extensive experiments on fundamental tasks across VideoMatte240K (VM) [10], Distinctions-646 (D646) [29], and Adobe Image Matting (AIM) [30] datasets demonstrate that the advantages BiMatting are bask-independent. efficient, allowing BiMatting achieve 12.4x FLOPs and 21.6x storage savings compared to the full- precisoon counterpart, leading a promising way for the video matting on edge scenarios (Fig. 1).
addition, our SBB and SAB components are highly
2 A Baseline for Binarized Video Matting
In this section, we first build a baseline to study the binarized video matting model. The baseline is based straightforward binarization and existing lightweight video matting architecture.
2.1 Binarization Framework
Binarization applies the sign function to compress weights and activations to 1-bit for computing units of the binarized network [17, 19, 20, 22, 31], J, and the propagation process can be expressed as
sign(x)) = +1, x
ac
8C
if I E (-1,1) otherwise
ssign(m) 0.
(1)
otherwise
dx

---

Bi-All
BI-GEP
Bi-DGE
Bi-Decodr
Bi-Decoder
Bi-Aspp
Hi-Backhone
MAD
Bi-Aupp
#Param (M)
Original
Bi-Backbonc
(G)
S
(a) Accuracy perspective
(b) Efficiency perspective
Figure 3; Analysis of bottlenecks for the binarized video matting baseline respectively in accuracy and efficiency the VM dataset. We report (a) the MAD metrics of binarizing each component, and (b) the FLOPs and storage consumption of each component in the binarized baseline.
where L as the loss function. To build strong binarized baseline by existing techniques, we apply floating-poin scaling factors for weights W [19, 23] and learnable thresholds for activations [23]:
B = sign(M W);
Ba = sign( A +), -= a & Ba:
(2)
where Bw and Ba denote the binarized weight and activation, respectively. The channel-wise a is obtained by mean( W)), and the layerwise threshold is initialized as O. Note that the Straight- Through Estimator (STE) [32] is uniformly applied to all experiments and binarization methods, approximating the gradient of the sign function as clip function (as the backward in Eq. (1)), since most other estimators consume unaffordable GPU memory [31, 22]. Binarization is applied to computation layers except for those who access original inputs produce final outputs.
2.2 Video Matting Architecture
Matting model aims to break down frame I into foreground F and background B, using coefficient to represent the linear combination of the two [1, 7J.
I = aF + (1 B.
Compared to image matting [2, 3, 4, 5, 6, 33, 34, 35, 36), matting methods specifically developed for video sequence should be more effective in utilizing spatial-temporal information in videos [l, 8, 9, 10, 11, 37, 38]. Among them, the recent Robust Video Matting (RVM) [1] method achieves the SOTA accuracy of the video matting task and stands out significantly in terms of efficiency. The architecture of RVM mainly comprises a lightweight MobileNetV3 based encoder (including backbone and Atrous Spatial Pyramid Pooling (ASPP) [39] module), recurrent decoder, and deep guided filter (DGF) [40] module. The concise architecture brings efficiency, even realizing real-tin time matting at high resolution. While RVM is one of the lightest video matting models available, there is still significant potential for compression in terms of reducing bit width, given the high cost of both the floating-point parameters and computations involved.
3
The Rise of BiMatting
3.1 Bottlenecks of Binarized Video Matting Baseline
Our goal is to achieve practical and cesfuice-efficiett video matting via binarization. However, the encoder and decoder of the binarized baseline pose accuracy and efficiency bottlenecks, respectively.
From accuracy perspective, Fig. 3 (a) compares the accuracy drops the VideoMatting (VM) dataset by binarizing every single part in the RVM model. Specifically, we find that binarizing the existing lightweight MobileNetV3 backbone in the encoder causes the most significant drop in accuracy among all parts, with a drop almost equivalent to directly binarizing the entire network (Bi-All 28 48.49 vs. Bi-Backbone 23.56 for MAD metric). In contrast, binarizing the ASPP, decoder, and DGF parts produce less harm to accuracy (less than a 3.05 MAD increase). Thus, improving the backbone of the encoder to make it more amenable to binarization is of the highest priority in addressing the accuracy drop in the binarized video matting model.
From the efficiency perspective, show in Fig. 3 (b) the computation and storage usage for each part of the binarized video matting model to demonstrate the individual consumption efficiency, Our analysis from an efficiency perspective reveals that the decoder consumes significant amount

---

of computational resources, accounting for 71.8% (0.38G) FLOPs with only 12.1% parameters after binarization. Conversely, the binarized encoder has a higher parameter count (81.8%). but its FLOP consumption is only 16.6%. Negligible consumption is observed for the other parts. Based on these observations, it evident that the computational redundancy of the decoder significantly impacts the acceleration performance of the overall model. Thus, there is ample room for computational optimization after direct binarization to further enhance the model's efficiency.
Given the observations from the aforementioned experiments, we find that the current baseline presents two major challenges: 1) the encoder's lightweight backbone unsuitable for binarization and fails to generate practical features, and 2) the decoder continues to exhibit considerable computa- tional redundancy even after binarization. Thus, we propose a binarization-friendee encoder and a computationally-efiiiei decoder to construct an accurate and efficient binarized matting model.
3.2
Shrinkable Binarized Block
3.2.1
Binarization-evoked Encoder Degradation
The binarized video matting baseline in Sec. 2 employs backbone and directly binarizes its convolutional and linear layers. However, the extensive use of groupwise and pointwise convolutions presents significant challenges binarization [41, 23], resulting in severe accuracy degradation. Fig. 3 shows that directly using binarized MobileNetV3 leads to over 4x MAD metric degradation. On the other side, existing binarized networks are also far from the efficiency level of binarized MobileNetV3,, making it hard to be transferred straightforwardly to construct the binarized matting model. For example, some binarized MobileNet-be based models suffer at least 3-8x FLOPs than binarized MobileNerV3 [23, 41, 26). Therefore, new binarized backbone is necessary for feature extraction to achieve high-quality features for the decoder while keeping the model ultra-iigtteiiht..
2mployys a lightweight MobileNetV3 as the encoder's
We first analyze the degradation of the binarized encoder in existing baselines. We note the binarized convolution as BiConv,, (-) or GBiConv,, (-), where n is the kernel size of the convolution and G mark denotes groupwise. Their superscripts "p, eq, and da indicate the number of their output channels is greater than, equal to, or less than that of their input channels, respectively. Then the binarized MobileNetV3 block in the baseline's 's encoder can expressed
MBV3 Block (1): o BiConv (C (BCCoo"""" + x, s.t. c e
(4)
MBV3 Block (2): o BiConv dn (GBiConv #4 x (BiConv (z)) + +cc = c//
where x and o denote the input and output, respectively, kernel size n E (3. 5}. and [-] denotes the Iverson bracket [42], evaluating to if the condition inside the parentheses is true, and 0 otherwise. The batch normalization and the activation layers following convolutions are omitted.
Eq. (4) suggests at least two issues that impede accurate binarization. Firstly, all convolutions are groupwise or pointwise that have fewer parameters than regular convolutions, thus are sensitive to binarization and hard to recover from crashes caused by mutual influence. Secondly, utilizing only per-block short connection is insufficient since convolution-specine shortcuts are critical to achieving performance recovery [22, 23]. Several existing binarization methods aim to address the aforementioned issues, e.g., applying regular convolutions instead of grouped ones [23, 26, 41], or creating shortcuts for channel-constae effective, they are significantly more computational-cot -xpponniiieeeeevieeeeeee than directly binarized MobileNetV3 and still suffer representation loss. Therefore, it is necessary to develop stronger feature extraction encoders for binarized video matting models to address these limitations.
convolutions [23, 41]. Though these techniques may be
3.2.2 Shrinkable Binarized Block for Accurate Encoder
To mitigate the degradation of the encoder caused by binarization while keeping it lightweight, we present an efficient Shrinkable Binarized Block (SBB) for building a binarized video matting encoder, which is conducive to flexible and lightweight architecture via channel-shrinkan design while retaining the representations and gradients.
Based on the analysis in Sec. 1, we determine that the crucial paradigm of an accurate binarized encoder is the computation-dense form of binarized block. In terms of topology, we ensure that every binarized convolution has corresponding/dense connection to preserve the representation

---

accurately. In terms of the operator, introducing computation-dens regular convolutions can prevent groupwise and pointwise ones from becoming exclusive. Following this paradigm, SBB recovers the representation in the binarized block such that enables the backbone to effective feature extraction in flexible dimensional space.
Specifically, we first introduce the sub-blocks of SBB. Based on the above analysis, we find that the key element constructing the flexible binarized architecture is to allow various sub-blocks in the feature extractor to freely adjust (increase, maintain, or shrink) the channel number of the outputs, while the shorteut should accompany with each binarized convolution to maintain representation. Therefore, the channel-shrunt SBB sub-block can be first constructed as follows:
( (a') + mean (1=0"", a0((09
Sub-SBB (1)
(dn(() (a:)
o BiConv
(<) x²' BiConv 24 (x) + x, s.t. c = 2c"
where x (m,.) denotes taking the m to n channels of x'. In the sub-block, we introduce shortcuts for channel-shrunk binarized convolution with channel splitting and averaging operations, allowing the channel to shrink while constructing a shortcut with negligible overhead. This design enables the binarized encoder to leverage computing units with higher output channels (such 32 or 64) for extracting low-channel representations (16 or 32, correspondingly). resulting in dependable features for the decoder. Inspired by [23, 41J, the channel-maintain and increased sub-blocks are follows:
Sub-SBB (2)
0 (x):
BiConv ( ( + x', x' BiCoov (()) + œ,
S.I. c co
Sub-SBB (3) 0 (x)(
((BiConv) 0g (x') + x') @ (BiConv 1 (xx) + xx))
(6)
x BiConv (x: x, s.t. c² c2/2
where @ denotes the concatenate operation. Benefiting from the above sub-block variants, SBB can implement a flexible feature extraction architecture.
Next, we create SBB by assembling these sub-blocks. Each SBB is made up of three sub-blocks: the head, middle, and tail. The head and tail sub-blocks produce identical output channels, while the middle sub-block extracts feature in higher dimensions with double output channels:
SBB o 6dm 8TT ('T(2=)) + x', x' x'= 0z) = co) + luP (x))
(7)
where x' the output of the head sub-block that varies 089 or luP depending on the channel constraint in Iverson brackets. The binarized sub-blocks in the middle and at the end of the sequence respectively increase and decrease the feature dimension for thorough extraction, while also incorporating shortcut across sub-blocks that allows the representations to remain intact. Using the block-level crossing shortcut also mitigates the influence of the splitting and averaging procedure on the representations in the trail channel-contracted sub-block.
3.3 Sparse-Assisted Binarization
3.3.1 Computational Decoder Redundancy
In Sec. 3.1, we discover that while binarizing the decoder prevents accuracy performance from deteriorating, it results in significant computational costs (greater than 71.8%) despite having relatively small parameter size (less than 12.1%). This inefficiency creates a bottleneck for the binarized video matting model. This is attributed to the decoder where high-resolution features are computationally demanding. To illustrate, consider the decoder's s output block, which uses two xx3 convolutions to perform computations that are unfriendly to binarization are not utilized here, the accuracy reduction decoder binarization is not as severe it is the encoder. But the computation of this single block in the decoder (the last one in 5 decoder blocks) is even equivalent to 103% of the entire encoder in a binarized baseline.
the original scale's features. Since grouped convolutions
However, it is worth noting that not all spatial computations are equally crucial. In the case of video matting, the majority of frames have large, uninterrupted foreground and background areas, which can usually be identified at a lower resolution. Nevertheless, the current decoder architecture repeats these computations at various scales, leading to significant computational overhead, particularly on high-resolution feature maps. This issue renders constructing efficient decoder challenging,
5

---

with the most aggressive 1-bit binarization approach. As a result, fundamental redesign of the computation unit is required to create an efficient decoder that minimizes redundant computation while maintaining accurate matting results. Such revamp should consider the context of binarization and focus on streamlining the computation process while preserving the quality of the output.
3.3.2 Sparse-Assisted Binarization for Efficient Decoder
To address the computational redundancy problem in the decoder of the binarized video matting model, we propose the Sparse-Assiste Binarization (SAB) method in BiMatting.
Fortunately, benefiting from the analysis in Sec. 3.3.1, the solution to the computational redundaney of the decoder intuitive, that is, to reduce the repeated intensive computation of continuous regions in the decoder, especially at the high-resolution features. First, inspired by the sparse segmentation and spatial region classification of the trimap, we hypothesize that most computation inside the matting decoder should be used in the unknown alpha matting regions, instead of an equal distribution on the certain foreground/backgronn regions, we compute the incoherent regions Mine following [27] using the low-resolution output mask M E {0, 1) to x to from the first decoder block, where N denotes the original input scale.
regions of the whole image. To approximate such unknown
Incoherent regions are primarily found along object instance boundaries or high-frequency areas (blue grids of the sparse feature in Fig. 2), and the points inside the incoherent regions are considered to be uncertain in alpha estimation. The remaining regions outside of the incoherent regions can be skipped/reduuced in computation. Since the features of this process are the lowest resolution among the 4 sets of features input by the decoder, we obtain the mask Mine at a low cost.
Then, we employ the detected mask Mine to transform the convolutions in high-resolution blocks into sparse-assisted binarized convolutions for efficient computation. Our designed masked sparse convolution is different from [27,45 on performing pointwise attention for the whole incoherent regions, resulting in a more binarization-friendly [44] and computation-effiiien design. Specifically, we transform the 3x3 convolution in the up-sample block of the decoder into:
SAB : O = SA-BiConvy bilinear" (Min)) + BiConvi (x),
where SA-BiConvs( M) represents the sparse- assiseed binarized 3x3 convolution under the guidance of sparse mask Mine in which the weight and sparse activation are binarized, and bilincar* denotes a bilinear interpolation up-sampling operation, and the low-resolution sparse mask Mine up-sampled by A times, k E (2, 4, 8, 16). For this binarized convolution SA-BiConvs, the implementation of sparse computation follows [45, 27] to skip Mis y) = O masked regions during inference. However, this does not change that the essence of this convolution is still non-grouped 3 3 convolution, which retains binarization frrnon--frrinnneessss in practice. Moreover, as in Eq. (8). besides the assisted binarized 3 3 convolution, we also apply one 1 I binarized convolution layer to process extraction in the whole feature and fuse the output obtained to that of SA-BiConv to guide the finer predictions of continuous regions.
Our SAB first obtains a sparse binary mask prediction based on low-resolution features, which is then used assist sparse binarization in the higher-resolution features of the decoder. As validated in the experiment section (Tab. 1), SAB greatly reduces the computation of the binarized video matting model by reducing computational FLOPs by 30% while maintaining the accuracy of video matting.
3.4 Training Pipeline of BiMatting
We present the training pipeline for our BiMatting model, which involves additional training steps and iterations to ensure complete convergence of the binarized video matting model, as compared to the training of the full-precision RVM counterpart [1]. The training pipeline of our BiMatting is comprised of two phases, namely the pre-training phase and the matting training phase.
Pre-training phase: In the pre-training phase, we trained the binarized SBB backbone on the ImageNet classification dataset for 200 epochs to obtain well-pre-trained backbone. This pre- training phase made it easier to converge during the matting training phase. Moreover, since the direct binarization of the full-precision pre-trained model, such as MobileNetV3, can lead to almost crashing, we apply the full pre-training phase for all compared binarized video matting models.

---

Table I: Ablation result of BiMatting on VM [10] dataset. We ablate SBB and SAB in BiMatting at the encoder and decoder, respectively. Binarized RVM with MobileNetV3 is used the baseline.
Model
Method
#Bit
#FLOPs))
#arrammm
MAD
MSE
Grad
Conn
dtSSD
RVMmbu3
32
4.57
14.5
6.08
1.47
0.88
0.41
1.36
RVMmbrs
1
0.55
0.64
28.49
18.16
6.80
3.74
3.64
RVMsBB
SBB
1
0,57
0.67
14.81
7.63
3.16
1.70
2.70
RVMSAB
SAB
1
0.35
0.67
189.13
184.33
15.01
27.39
3.65
BiMatting (Ours)
SBB+SAB
1
0.37
0.67
12.82
6.65
2.97
1.44
2.69
Matting training phase: This phase is binarization- aware and built following [1], divided into stages. Stage 1 involves training on the low-rcsolution VM dataset for 20 epochs without DGF, with T = 15 frames for quick updates. The SBB backbone's 's learning rate is set as 1e and the rest 2e- Additionally, the input resolution h, sampled independently from 256-512px for improving robustness. Stage 2, the network is trained with T = 50, with halved learning rate and 2 more epochs enable learning of long-term dependencies. In Stage 3, the DGF module is attached, and epoch is trained on both low-resolution long and high-resolution short sequences from the VM dataset. The low-resolution pass is T 40, and w are the same with stage I without DGF, while the high-resolution pass employs DGF with downsample factor 0.25, T h,, di (1024, 2048). The learning rate of DGF 24 and that of the rest 1e-5. In Stage 4, the network is trained for 5 epochs on D646 and AIM, increasing the decoder's learning rate to 5e" 5e-b.
and
4 Experiments
We extensively evaluate the accuracy and efficiency of our proposed BiMatting. We first ablate our method and illustrate the contributions of SBB and SAB on the VM [10] dataset. Then we compare BiMatting with binarized video matting models that utilize existing binarization techniques on VM [10], D646 [29]. and AIM [30], where our designs excel and even outperform some full-precision video matting models. Concerning efficiency, BiMatting impressively reduces computational FLOPs and model size by 11.2x and 21.6x, respectively. On metrics, we evaluate a using mean absolute difference (MAD), mean squared error (MSE), spatial gradient (Grad), and connectivity (Conn) for quality, and dtSSD for temporal coherence. We also measure pixels where a > 0 by MSE for [1]. All stages of our experiments batch size splits across 4 Nvidia A100 GPUs.
4.1 A Ablation Study
Tab. 1 demonstrates that the binarized video matting baseline experiences a significant drop in performance across all accuracy metrics for VM data recall. Despite its 22.7 x parameter compression, the model only realizes an 8.3x computational savings in the efficiency metric. Upon substituting the encoder with our SBB alone, the binary model's accuracy is considerably restored, affirming that the encoder is the principal performance bottleneck in the baseline. However, the decoder's implementation of the efficient SAB does not fully resolve the performance bottleneck, where the accuracy of the binarized model is perilously close to collapsing. By integrating both of our contributions, both accuracy and efficiency performance is substantially enhanced. Notably, in BiMatting, combining these two improvements can produce remarkable enhancement in accuracy performance, underscoring the importance that the decoder should concentrate on less but crucial representations to heighten model performance by providing high-quality features.
4.2 Comparison Results
To create the comparison benchmark, we combine test samples from the VM, D646, and AIM datasets with 20 video backgrounds and 20 image backgrounds, following the settings in [1, 47]. Each test clip contains 100 frames where motion augmentation is applied to image samples. We compare our BiMatting model with other video matting models that have been binarized using existing methods. These binarization methods include classical BNN [41] and DoReFa [18], as well as state-of-the
(SOTA) ReActNet [23] and ReCU [21], where the latter two are considered as best practices for generic binarization [44]. In addition, we have also included results from some full-precision video matting methods for comparisons, such as the RVM [1] with MobileNetV3 [48] backbone (oracle),
DeepLabV3 [28], and background-basse BGMv2 [10] with MobileNetV2 [49] backbone. To ensure
7

---

Table 2: Low-resolution comparison on VM, D646, and AIM datasets. Bold indicates the best performance among binarized video matting models and indicates the results is crashed.
Alpha
PG
Dataset
Method
#Bit
#FLOPs(G) #ParamıM))
MAD
MSE
Grad
Conn
dtSSD
MSE
VM
DeepLabbV3
32
136.06
223.66
14.47
9.67
8.55
1.69
5.18
282
BGMv2
32
8.46
19.4
25.19
19.63
2.28
3.26
2.74
RVM (oracle)
32
4.57
14.5
6.08
1.47
0.88
0.41
1.36
RVM-BNN
0.50
0,57
189.13
184.33
15.01
27.39
3.65
RVM-DoReFa
0.52
0.57
51.64
34.50
8.85
7.14
4.09
RVM-ReCU
0.52
0.64
189.13
184.33
15.01
27.39
3.65
RVM-ReAct
0.55
0.64
28.49
18.16 16
6.80
3.74
3.64
BiMatting (Ours)
0.37
0.67
12.82
6.65
2.97
1.42
2.69
D646
D3
32
241.89
223.66
24.50
20.1
20.30
6.41
4.51
312x312
BGMv2
32
16.48
19.4
43.62
38.84
5,41
11.32
3.08
2.60
RVM (oracle)
32
8.12
14.5
7.28
3.01
2.81
1.83
1.01
2.93
RVM-BNNI
0.57
281.20
276.85
25.26
73.59
1.08
6.95
RVM-DoReFa
0.92
0.57
133.63
116.69
17.09
35.08
2.58
6.97
RVM-ReCU
0.92
0.64
281.20
276.85
25.26
73.59
1.08
6.95
RVM-ReAct
0.97
0.64
56.41
43.10
14.05
14.85
2.56
6.85
BiMatting (Ours)
0.66
0.67
32.74
24.48
9.34
8.62
2.21
5.86
AIM
D3
32
241.89
223.66
29.64
23.78
20.17
7.71
4.32
312x512
BGMv2
32
16.48
19.4
44.61
39.08
5.54
11.60
2.69
3.31
RVM (oracle)
32
8.12
14.5
14.84
8.93
4.35
3.83
1.01
5.01
RVM-BNN
0.88
0.57
327.02
321.15
23.80
85.55
0.75
7.84
RVM-DoReFa
0.57
129.29
107.79
17.31
34.18
2.62
7.85
RVM-ReCU
0.92
0.64
327.02
321.15
23.80
85.55
0.75
7.84
RVM-ReAct
0.97
0.64
$9.90
44.08
14.32
15.90
2.37
8.00
BiMatting (Ours)
0.66
0.67
35.17
26.53
9.42
9.24
1.82
7.00
Table 3: High-resee resolutooo comparison on VM, D646, and AIM datasets. released model directly [46].
indicates using the officially
Dataset
Method
#Bit
#FLOPs
#Paramaи
SAD
MSE
Grad
dtSSD
VM
RVM
32
4.15
14.5
6.57
1.93
10.55
1.90
1920x1080
BGMv2
9,86
19.4
49.83
44.71
74.71
4.09
RVM-ReAct ReAct
0.53
0.64
31.60
20.29
34.28
4.08
BiMatting (Ours)
0.38
0.67
18.16
11.15
21.90
3.25
D646
RVM
32
8.37
14.5
8.67
4.28
30.06
1.64
x048
BGMv2
15.19
19.4
57.40
52.00
149.20
2.56
RVM-ReAct
1.07
0.64
57.38
42.14
71.24
3.03
BiMatting (Ours)
0.77
0.67
52.85
44.08
61.60
3.12
AIM
RYM
8.37
34.97
2018X22048
BGMv2
32
15.19
19.4
45.76
38.75
124.06
2.02
RVM-ReAct
1.07
0.64
57.38
42.14
71.24
3.03
BiMatting (Ours)
0.77
0.67
48.27
38.37
61.72
2.80
a fair comparison, we apply the exact same training pipeline all binarized video matting networks as we did to BiMatting, as explained in Sec. 3.4. As for the full-precision video matting model, we follow the results reported in previous studies [1], unless otherwise specified.
Tab. presents comparison of methods that use low-resolution input. The findings indicate that applying BNN and ReCU directly to full- fill-precision RVM leads completely collapsed results. This is surprising given that the latter is among the SOTA binary methods, highlighting that binarizing existing video matting architectures not straightforward task. In contrast, our BiMatting model. performs significantly better than all existing binarization models across all datasets, which predicts alpha with higher accuracy and consistency, resulting more coherent and accurate performance. Further details will be presented in Sec. 4.3 with comprehensive visualizations. Furthermore, BiMat- ting even outperforms some 32-bit full- aell-preiision models when using only 1-bit limit bit width. For instance, BiMatting surpasses BGMv2 on VM, D646, and AIM datasets, well DeepLab on

---

(a) Frame matting comparison. Figure 4: We compare our BiMatting with the existing SOTA binarized RVM-ReActNet and 32-bit full-precision (32-bit) RVM models to demonstrate its excellent accuracy.
(b) Temporal coherence comparison.
VM datasets. These results demonstrate the enormous potential of binarization for efficient video matting. Tab. 3 presents a comparison between our BiMatring approach and other methods using high- resolution datasets. Our method consistently outperforms existing binarization models and BGMv2 across multiple metrics, demonstrating the robustness of BiMatting under varying resolutions.
Moreover, BiMatting has demonstrated its high potential for video matting in efficiency with impres- sive accuracy. As Tab. and Tab. 3 show, comparison to the full-precision counterpart (RVM), BiMatting achieves computational FLOPs savings of 12.4 times and parameter savings of 21.6 times, making it the most promising solution for edge deployment. Moreover, BiMatting outper- forms existing binarization methods, which offer significant acceleration advantages with only tiny parameter overhead, such as a 0.1M increase compared to RVM-BNN.
4.3 Visualization
Fig. 4 and Fig. show qual- itative comparisons of nat- ural videos. Fig. 4 com- pares our BiMatting model with
Input
BiMatting (ours) Figure Alpha detail comparison.
RVM-ReActNet
RVM (32-bit)
(the
current SOTA binarized video matting model) and the full full-precision RVM (can be seen as the 32-bit counterpart of BiMatting). general, BiMatting's performance close to the full-precision counterpart with greatly reduced resource consumption. Specifically, in Fig. 4(a), we conduct experi- ments on videos from diverse video frames. Our results reveal that BiMatting more robust against semantic errors. Furthermore, BiMatting outperforms the other two models in matting edge regions. Fig. 4(b) compares temporal coherence, where our BiMatting consistently produces superior results, while RVM-ReActNet produces different areas of error. In Fig. 5, comparison is made among the alpha predictions various methods. Observe that our method outperforms the SOTA binarized video matting model by more accurately predicting intricate details, such individual strands of hair.
Limitation. Though significantly improved, our BiMatting not yet on par with its full-precision counterpart, the 32-bit RVM, particularly when it comes to local details such as the hair ends which tend to get blurred, while also favoring simpler backgrounds that lead to accurate matting results.
Conclusion
Our proposed model, BiMatting, is an efficient and accurate solution that utilizes binarization to achieve real-time video matting on edge devices constrained by computational resources. In this paper, we address the primary bottlenecks of binarization by constructing shrinkable and dense topologies for the binarized encoder block to enhance representation and sparsifying the binarized units to reduce redundant decoder computation. Our exhaustive experiments demonstrate that the proposed BiMatting outperforms existing binarized video matting models by significant margin while producing a comparable performance to the full-precision counterpart visual quality. BiMatting achieves significant savings in computation and storage, making attractive solution for world resource-constrainee scenarios such as online conferences and short-form video production.
Acknowledgement This work was supported by the National Natural Science Foundation of China (No. 62022009), the State Key Laboratory of Software Development Environment (SKLSDEE. 2022ZX-23).

---

References
[1] Shanchuan Lin, Linjie Yang, Imran Saleemi, and Soumyadip Sengupta. Robust high-resolution video matting with temporal guidance. In WACV, 2022.
[2] Yagiz Aksoy, Tune Ozan Aydin, and Mare Pollefeys. Designing effective inter-pixel information flow for natural image matting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 29-37, 2017.
[3] Xue Bai and Guillermo Sapiro. A geodesic framework for fast interactive image and video segmentation and matting. In 2007 IEEE 1 Ith International Conference on Computer Vision, pages 1-8. IEEE, 2007.
[4] Qifeng Chen, Dingzeyu Li, and Chi-Keung Tang. Knn matting. IEEE transactions on pattern analysis and machine intelligence, 35(9):2155-2188 2013.
[5] Yung- Yu Chuang, Brian Curless, David H Salesin, and Richard Szeliski. A bayesian approach to digital matting. In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, volume 2, pages II-II. IEEE, 2001.
[6] Xiaoxue Feng, Xiaohui Liang, and Zili Zhang. A cluster sampling method for image matting via sparse coding. In Computer Vision-ECCC 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 204- 219. Springer, 2016.
[7] Jiachen Li, Vidit Goel, Marianna Ohanyan, Shant Navasardyan, an, Yunchao Wei, and Humphrey Shi. Vmformer: End-to-end video matting with transformer. arXiv preprint arXiv:2208, 12801, 2022.
[8] Yanan Sun, Guanzhi Wang, Qiao Gu, Chi-Keung Tang, and Yu- Wing Tai. Deep video matting via spatio-temporal alignment and aggregation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 6975-6984 2021.
[9] Yunke Zhang, Chi Wang, Miaomiao Cui, Peiran Ren, Xuansong Xie, Xian-Sheng Hua, Hujun Bao, Qixing Huang, and Weiwei Xu. Attention prentiio-guided temporally coherent video object matting. In Proceedings of the 29th ACM International Conference on Multimedia, pages 5128-5137, 2021.
[10] Shanchuan Lin, Andrey Ryabtsev, Soumyadip Sengupta, Brian L Curless, Steven M Seitz, and Ira
Real-time high-resolution background matting. In Proceedings of the IEFE/CVF Conference Computer Vision and Pattern Recognition, pages 8762-8771, 2021.
[11] Soumyadip Sengupta. Vivek Jayaram, Brian Curless, Steven M Seitz, and Ira Kemelmacher Shlizerman. Background matting: The world is your green screen. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 2291-2300 2020.
[12] Zixiang Zhao, Jiangshe Zhang, Xiang Gu Chengli Tan, Shuang Xu, Yulun Zhang, Radu Timofte, and Luc Van Gool. Spherical space feature decomposition for guided depth map super-resolution. In /CCV, 2023.
[13] Zixiang Zhao, Jiangshe Zhang, Shuang Xu, Zudi Lin, and Hanspeter Pfister. Discrete cosine transform network for guided depth map
In CVPR, 2022.
[14] Zixiang Zhao, Jiangshe Zhang, Haowen Bai, Yicheng Wang, Yukun Cui, Lilun Deng, Kai Sun, Chunxia Zhang, Junmin Liu, and Shuang Xu. Deep convolutional sparse coding networks for interpretable image fusion. In CVPRW, 2023.
[15] Zixiang Zhao, Shuang Xu, Jiangshe Zhang, Chengyang Liang, Chunxia Zhang, and Junmin Liu. Efficient and model-based infrared and visible image fusion via algorithm unrolling. IEEE TCSVT, 2022.
[16] Zixiang Zhao, Haowen Bai, Yuanzhi Zhu, Jiangshe Zhang, Shuang Xu, Yulun Zhang, Kai Zhang, Deyu Meng, Radu Timofte, and Lue Van Gool. Ddfm: Denoising diffusion model for mudti-modality image fusion. In ICCV, 2023.
10

---

[17] Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- Vamiv, and Yoshua Bengio. Binarized neural networks. In NeurIPS. 2016.
[18] Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. CoRR, Abs//600.06166, 2016.
[19] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks, In ECCV, 2016.
[20] Adrian Bulat and Georgios Tzimiropoulos. Xnor-net++: Improved binary neural networks.
CoRR, abs/1909, 13863, 2019.
[21] Zihan Xu, Minghao Lin, Jianzhuang Liu, Jie Chen, Ling Shao, Yue Gao, Yonghong Tian, and Rongrong Ji. Recu: Reviving the dead weights in binary neural networks. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1198-52088 2021.
[22] Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng. Bi-real net: Enhancing the performance of 1 1-bit cnns with improved representational capability and advanced training algorithm. In ECCV, 2018.
[23] Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang- Ting Cheng. Reactnet: Towards precise binary neural network with generalized activation functions. In ECCV, 2020.
[24] Yuzhang Shang, Dan Xu, Bin Duan, Ziliang Zong, Liqiang Nie, and Yan Yan. Lipschitz continuity retained binary neural network. In ECCV, 2022.
[25] Yuzhang Shang, Dan Xu, Ziliang Zong, Liqiang Nie, and Yan Yan. Network binarization via contrastive learning. In ECCV, 2022.
[26] Joseph Bethge, Christian Bartz, Haojin Yang, Ying Chen, and Christoph Meinel. Meliusnet: Can binary neural networks achieve mobilenet-level accuracy? arXiv preprint arXiv:2001.05990 2020.
[27] Lei Ke, Martin Danelljan, Xia Li, Yu-Wing Tai, Chi-Keung Tang, and Fisher Yu. Mask transfiner for high-quality instance segmentation. In CVPR, 2022.
[28] Liang-Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXXi:1106.00530 2017.
[29] Yu Qiao, Yuhao Liu, Xin Yang, Dongsheng Zhou, Mingliang Xu, Qiang Zhang, and Xiaopeng Wei. Attention-guica guided hierarchical structure aggregation for image matting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13676-13685. 2020.
[30] Ning Xu, Brian Price, Scott Cohen, and Thomas Huang. Deep image matting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2970-2979, 2017.
[31] Yixing Xu, Kai Han, Chang Xu, Yehui Tang. Chunjing Xu, and Yunhe Wang. Learning frequency domain approximation for binary neural networks. In NeurIPS, 2021.
[32] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 1308.3432, 2013.
[33] Leo Grady, Thomas Schiwietz, Shmuel Aharon, and Rüdiger Westermann. Random walks for interactive dipha-matting In Proceedings of VIIP, volume 2005, pages 423-429. Citeseer, 2005.
[34] Quan Chen, Tiezheng Ge, Yanyu Xu, Zhiqiang Zhang, Xinxin Yang, and Kun Gai. Semantic human matting. Proceedings of the 26th ACM international conference on Multimedia, pages 618-626, 2018.
[35] Jinlin Liu, Yuan Yao, Wendi Hou, Miaomiao Cui, Xuansong Xie, Changshui Zhang, and Xian- sheng Hua. Boosting semantic human matting with coarse annotations. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2020.
11

---

[36] Qihang Yu, Jianming Zhang, He Zhang, Yilin Wang. Zhe Lin, Ning Xu, Yutong Bai, and Alan Yuille. Mask guided matting via progressive refinement network. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1 1154-1163. 2021.
[37] Zhanghan Ke, Kaican Li, Yurou Zhou, Qiuhua Wu, Xiangyu Mao, Qiong Yan, and Ryn- son WH Lau. Is a green screen really necessary for real-time portrait matting? arXiv preprint xrXiv:2011.11961 2020.
[38] Zeqi Gu, Wenqi Xian, Noah Snavely, and Abe Davis. Factormatte: Redefining video matting for re-composition tasks, arXiv preprini arXiv:2211.02145. 2022.
[39] Liang-Chich Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4):834 848, 2017.
[40] Huikai Wu, Shuai Zheng, Junge Zhang, and Kaiqi Huang. Fast end-to-end trainable guided filter. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1838-1847, 2018.
[41] Hai Phan, Zechun Liu, Dang Huynh, Marios Savvides, Kwang- Cheng, and Zhiqiang Shen. Binarizing mobilenet via evolution-based searching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13420-1344 13499, 2020.
[42] Kenneth E Iverson. A programming language. In Proceedings of the May 1-3, 1962, spring joint computer conference, pages 345-351, 1962.
[43] Lei Ke, Henghui Ding. Martin Danelljan, Yu- Ku-Wing Tai, Chi-Keung Tang, and Fisher Yu. Video mask transfiner for high-quality video instance segmentation. In ECCV, 2022.
[44] Haotong Qin, Mingyuan Zhang, Yifu Ding, Aoyu Li, Ziwei Liu, Fisher Yu, and Xianglong Liu. Bibench: Benchmarking and analyzing network binarization. In ICML, 2023.
[45] Keyu Tian, Yi Jiang, Qishuai Diao, Chen Lin, Liwei Wang, and Zehuan Yuan. Designing bert for convolutional networks: Sparse and hierarchical masked modeling. arXiv preprint arXiv:2301.03580, 2023.
[46] Shanchuan Lin, Andrey Ryabtsev, Soumyadip Sengupta, Brian L Curless, Steven M Seitz, and Ira saanellmmache-SShzeeemee Backgroundmattingy2. 05 2023.
[47] Shanchuan Lin, Linjie Yang, Imran Saleemi, and Soumyadip Sengupta. Robustvideomatting. 05 2023.
[48] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chiel Chen, Bo Chen, Mingxing Tan, Weijun Wang. Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1314-1324, 2019.
[49] Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018.
[50] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, ligh-eerformance deep learning library. Advances in neural information processing systems, 32, 2019.
12

---

Supplementary Material: BiMatting: Efficient Video Matting via Binarization
6 Overview
We offer further information in this supplementary material. Section elaborates on our network architecture, providing detailed insights. In Section 8, we present additional results for detailed comparison and also showcase visual examples of our composited matting data samples. We highlight that video results are included in our supplementary material. We encourage readers to refer to our provided video for extensive matting results comparisons.
7 Details of Network Architecture
For the encoder (Table 4) constructed by our proposed SBB, operates on individual frames to extract feature maps at various spatial scales ranging from to 1 Within each SBB, the number of feature channels is either doubled or stays the same in the first sub-block, then doubled again in the second sub-block, and finally halved in the third sub-block. This design enables each SBB to carry out feature extraction channel dimension space that exceeds the input dimension. Meanwhile, the computation-dense structure guarantees the utilization of binarized convolutions to acquire high- quality features. While traditional full- cull-precision MobileNetV3 backbones operate at 32 scale, we made modifications to the last block. Specifically, utilize convolutions with a dilation rate of and a stride of 1, following the design principles from [1]. Furthermore, the final feature map 16 scale) is passed to the LR-ASPP module, which compresses it into 128 channels.
Mislule
SBR_1 Sub
Modile
Sub
SBB Sub SBB
Sub
Sub SBB
SBB
In/Qui Chemnel
(32, 64)
30)
(32, 64)
(64, 128)
(178. 64)
(64.64)
(64. 128)
(178,
Extracted Feature
SBB_5
Sab-SB8
Sub-SBB
Sul-SBB ())
Sab-SB8
Sub-SBB (3)
Sub-SBB Sub-8B8 (3)
Sub-SB8
(64,
(128,250) 256)
(128, 128)
(128, 256)
(256, 128)
(128.256)
(1024,
Featuee
Table 4: The details in the encoder of BiMatting, where "Feature Scale" indicates the scale of features extracted by this sub-block that utilized by the decoder. Sub-SBB (1), (2), and (3) follow the notations of Eq. (5) and (6) in paper to represent different types of sub-blocks.
For the decoder (Table 5), mentioned in our paper, the SAB is employed in every decoder block except the first one to accelerate computations. The binary mask used by the SABs is obtained from the first non-sparse binarized block, which has the smallest feature scale and acquires the mask at minimal cost. This design significantly improves the efficiency of the decoder.
Module
BottleNeck
SAB_ (Upsampling) SAB.2 2 (Upsampling) SAB 3 (Upsampling)
SAB (Ourput)
Feature Scale
T
I
Inpur Mask
M
Mnc
Mic
Mic
Produced Mask
Mise
Table 5: The details in the decoder of our BiMatting, where "Extracted Feature" indicates that the features extracted by this blocklock are utilized by the decoder. Sub-SBB (1), (2), and (3) follow the notations of Eq. (5) and (6) in paper represent different types of sub-blocks, Mine is the sparse mask to guide the decoder computation mainly in "difficult" regions.
13

---

Input (Original)
BNN (1-bit)
RVM-DoRdFa (1-bit)
RVM-RoAciNa (l-bit)
BiMatting (1-bit)
RVM (32-bi)
Figure 6: More visual results. Compared to 1-bit video matting models using existing binarization methods, our BiMatting significantly surpasses them and achieves near full-precision performance. Note that the results of RVM-BNN indicate the model fully crashes.
For other parts, the Deep Guided Filter (DGF) incorporates limited number of binarized 1 x 1 convolutions internally. For more detailed specifications, please refer to [40, 1]. The complete network is constructed and trained using Py PyTccc [50].
8
Additional Visualizations
8.1 Visual Results
We show more visual results in Fig. 6, where more clearly the advantages of our BiMatting over other binarization methods, both in edge details and local region matting. At the same time, we also provide video (BiMatting.nn mpp file in the supplementary material) to show the advantages of our BiMatting in more detail.
8.2 Composited Datasets
We follow [1] as guide to constructing composite training and test samples. We show some examples of composited training samples from the matting datasets in Fig. The clips contain natural movements when compositing with videos well artificial movements generated by the motion augmentation. Motion augmentation was exclusively applied to the foreground and background of the image in the testing samples (Fig. 8). The motion augmentation solely involved affine transforms. Moreover, the strength of the augmentation was deliberately toned down in comparison to the training augmentation, ensuring that testing samples possess high-degree realism.
14

---

Composited Frames
Std Dev
Figure 7: Composite training samples. The last column is the pixels' temporal standard deviation.
Figure 8: Example testing samples.
15


Reviews:
[{"review_id": "jTdXlMJI5i", "reviewer": "Reviewer_PTWg", "summary": "The authors analyzed the operations inside the deep video matting networks and proposed an efficient binarization method to greatly reduce the computation cost. Specifically, they re-designed the encoder and also sparsified the decoding process. The proposed methods are shown to outperform the existing baselines and achieves reasonable visual quality. ", "questions": "- In section 3.2.2, ep(5), the authors proposed to use mean to compute the values for 'short connection'. How did the authors come up with that solution? Why not using another convolution layers to reduce the channel number or taking the addition? Has the authors done some comparison?\n- Any plans to extend the work to other video applications?", "limitations": "- The SAB designs may fail when the video is dynamic or having more foreground object movement. \n- Complicated training procedures and vague stopping conditions for each stage. It makes the results in the paper hard to be reproduced. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Section 3 shows binarization of the encoder parts bring the most harmful degradation of the accuracy, and the current decoder consumes the most computational resources. The analysis and preliminary experiments demonstrate the reason of the proposed methods. The logic and story-telling of the paper make a lot of sense. \n- The review likes the reasoning of the encoder (lacking of short connection within the blocks) and decoder (many redundancy in computation), and the connections between the analysis and the proposed designs. \n- The design of SAB makes sense for any video applications. ", "weaknesses": "- The training pipeline is too complicated. \n- Table 1 is confused. So for BiMatting (Ours), is it the same as RVM (SBB+SAB)? It's better to state the difference of RVM and BiMatting, and show it clearly in the table. Why RVM + SAB has much worse results than the one without any of SBB and SAB? Is it possible the bad performance is due to the bad training or need a better parameter set different from the proposed one?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1**: The training pipeline is too complicated.\n\n**A1**: Please note that our training pipeline follows that of the baseline RVM for a fair comparison, using the code from their public GitHub repository. We do not add additional training stages or other complications. We also adopt the same stopping conditions as RVM. In addition, we provide the detailed training pipeline of BiMatting in our General Response and will release our training code in the final version. Please see the general response above for more details.\n\n> **Q2a**: Table 1 is confused. So for BiMatting (Ours), is it the same as RVM (SBB+SAB)? It's better to state the difference of RVM and BiMatting, and show it clearly in the table.\n\n**A2a**: Yes, BiMatting (ours) would be the same as \"RVM (SBB+SAB)\" in Table 1 of the original manuscript. To make the table more clear, we revise its notation, as shown in Table A2a. In the revised table, the checkmark and crossmark on the SBB column represent applying proposed SBB architecture and binarized MobileNetV3, respectively, and the checkmark and crossmark on the SAB column represent the applying SAB technique and direct binarization in the decoder, respectively. We will add this clarification in the next paper revision.\n\nTable A2a (*Table 1 of the attached PDF*): Ablation result of BiMatting on VM dataset.\n| SBB          | SAB          | \\#Bit | \\#FLOPs(G) | \\#Param(MB) | MAD     | MSE    | Grad   | Conn   | dtSSD  |\n|--------------|--------------|-------|-------------------|--------------------|---------|--------|--------|--------|--------|\n| -            | -            | 32    | 4.57              | 14.5               | 6.08    | 1.47   | 0.88   | 0.41   | 1.36   |\n| ✕ | ✕ | 1     | 0.55              | 0.64             | 28.49   | 18.16  | 6.80   | 3.74   | 3.64   |\n| ✓   | ✕ | 1     | 0.57              | 0.67               | 14.81   | 7.63   | 3.16   | 1.70   | 2.70   |\n| ✕ | ✓   | 1     | 0.35            | 0.67               | 189.13  | 184.33 | 15.01  | 27.39  | 3.65   |\n| ✓   | ✓   | 1     | 0.37            | 0.67            | 12.82 | 6.65 | 2.97 | 1.44 | 2.69 |\n\n> **Q2b**: Why RVM + SAB has much worse results than the one without any of SBB and SAB?  Is it possible the bad performance is due to bad training or needs a better parameter set ...?\n\n**A2b**: Our SAB reduces the computation of the decoder by masking the repeated intensive computation of continuous regions, which means that the remaining (not masked) regions in the feature need to provide enough effective information, and these features are provided by the encoder. Fig. 3 shows that when using the original encoder (directly binarized MobileNetV3), even if all other parts are restored to the full-precision counterparts, the performance still drops significantly. Therefore, the crashing can be expected when we use the original binarized encoder together with our efficient and lightweight (binarized) SAB decoder. We will clarify this point by revising Sec. 4.1.\n\nIn terms of settings, for fairness, we used exactly the same training settings and pipelines in all ablation experiments, which were completely consistent with the official code of the RVM baseline without any adjustments. This helps us reveal the real performance of our proposed techniques under equal settings.\n\n> **Q3**: In section 3.2.2, eq (5), the authors proposed to use mean to compute the values for 'short connection'. How did the authors come up with that solution? Why not using another convolution layers to reduce the channel number or taking the addition? Has the authors done some comparison?\n\n**A3**: As concluded from the analysis in Sec. 3.2.1, it is important to create a shortcut connection for each binarized convolution. We use the current form of connection for the following accuracy and efficiency considerations: (1) effectively recovering the representation and (2) constructing by parameter-free operations.\n\nFor accuracy consideration, when we introduce an additional binarized convolution in connection to change the channel number, the introduced convolutions in connection also suffer representation degradation caused by binarization, which makes it hard to solve the degradation of the original binarized unit. The results in *Table 2 in the attached PDF* (BiMatting-BiConv) also show that this method does not bring significant improvement over the no-connection method (BiMatting-NoConn). We also compared the direct addition method (BiMatting-SumConn) mentioned by the reviewers. Although it avoids the information loss caused by binarization, directly summing up the features leads to the loss of fine-grained representation, making it still inferior to BiMatting in accuracy.\n\nFor the efficiency consideration, even though it is binarized, the convolution introduced in the connection still brings additional computation and storage burden (see *Table 2 in the attached PDF*).\n\n> **Q4**: Any plans to extend the work to other video applications?\n\n**A4**: In this paper, we focus on the video matting task that is challenging for binarization, because the video matting model requires predicting continuous alpha instead of binary segmentation mask, and it also has a wider range of real-time applications on mobile devices. In future work, we plan to apply our proposed methods to speed up other dense tasks such as video segmentation and depth estimation.\n\n> **Q5**: Limitation: The SAB designs may fail when the video is dynamic or having more foreground object movement.\n\n**A5**: As we mentioned in our limitation paragraph, BiMatting does not perform as well as full-precision models in some highly dynamic scenes, due to the representation loss caused by the extreme bit-width compression. However, considering BiMatting is the first binarization model in the video matting field and achieves 12.4x acceleration and 21.6x compression gains, we believe the binarized video matting model has significant potential for improving accuracy in future work."}, {"review_id": "50s3gmSwxN", "reviewer": "Reviewer_wDXB", "summary": "The paper proposes a new video matting method called BiMatting. It is based on Binary neural networks (BNNs), a more compact network to reduce the computational and storage requirements of video matting. Specifically, the authors addressed the accuracy bottleneck of BNNs by re-designing its encoder and decoder architecture. Performance was evaluated on several video benchmarks and compared with SOTA methods. ", "questions": "1. The authors claimed that binarizing the encoder causes a significant drop in performance. Why is that the case? Do the authors mean binarizing the activation of the encoder?  \n2. Is there a reason why the paper does not have a related work section? I would be curious to see recent works and other applications of BNNs to image segmentation and other relevant fields. It would help me evaluate the novelty and impact of current methods.", "limitations": "The author addressed the limitation that the method is not yet on par with 32-bit RVM. Quality-wise, it tends to get blurred and prefers simpler backgrounds. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This is the first time BNNs was used for video matting application. This problem was well-motivated and several challenges were addressed by careful architecture design and new training procedures. The authors provided a comprehensive comparison with other SOTA methods and demonstrated that this method offers a good tradeoff between performance and storage. It is computationally efficient that BiMatting reduces computational FLOPs by 11 times and storage by 21 times. ", "weaknesses": "1. Although this method outperforms existing binarized video matting models, it is not yet on par with its full-precision counterpart in visual quality.\n2. Another potential weakness is the complexity of the training pipeline of Bi-Matting, which contains a pre-training phrase and a matting training phrase with four stages. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1**: Although this method outperforms existing binarized video matting models, it is not yet on par with its full-precision counterpart in visual quality.\n\n**A1**: As we present in our limitations paragraph and figures, BiMatting is not as accurate as full-precision models in certain highly dynamic scenes, due to the representation loss caused by the extreme bit-width compression. Nonetheless, our BiMatting significantly reduces the accuracy gap between the binarized video matting models and full-precision ones. Considering that BiMatting is the first binarization model in the video matting domain and has attained impressive gains in acceleration (12.4x) and compression efficiency (21.6x), we believe that the binarized video matting model holds substantial potential for further improved accuracy.\n\n> **Q2**: Another potential weakness is the complexity of the training pipeline of Bi-Matting, which contains a pre-training phrase and a matting training phrase with four stages.\n\n**A2**: Please note that our training pipeline completely follows that of the baseline RVM for a fair comparison, using the code from their public GitHub repository. We do not add additional training stages or other complications. We also adopt the same stopping conditions as RVM. We also provide the detailed training pipeline of BiMatting in our General Response and will release our training code in the final version.\n\n> **Q3**: The authors claimed that binarizing the encoder causes a significant drop in performance. Why is that the case? Do the authors mean binarizing the activation of the encoder?\n\n**A3**: We clarify here that the main reason for the significant performance drop when binarizing the encoder. It is caused by loss of representation capability induced by the coarse discretization (1-bit) of **both the model weights and activations**, especially when binarizing already lightweight architectures, like MobileNetV3. In the binarization process, 32-bit weights and activations are compressed to 1-bit. The representation capability and accuracy are therefore greatly reduced (from 2^32 to 2 possible states per weight or activation), which is the direct cause of the performance degradation of the binarized backbone. See Sec 2.1 and 3.1 for further discussion.\n\n> **Q4**: Is there a reason why the paper does not have a related work section? I would be curious to see recent works and other applications of BNNs to image segmentation and other relevant fields. It would help me evaluate the novelty and impact of current methods.\n\n**A4**: We discuss related work in Sec 2, including the related work on binarization (Sec 2.1) and video matting (Sec 2.2). We will follow the reviewer’s suggestion and add the section title of related work to make the manuscript more clear. \n\nWe will also add a discussion on more related work on BNNs for image segmentation as suggested by the reviewer: Group-Net [1] demonstrates successful application to the semantic segmentation task on PASCAL VOC. Frickenstein et al. introduce Binary DAD-Net [2], the first BNN-based semantic segmentation network for drivable area detection in the autonomous driving field. Zhou et al. present CBNN [3], which incorporates multiple subnets with learnable global lateral paths and evaluates its performance on a segmentation dataset. However, the corresponding full-precision counterparts of these binarized networks are almost classical ResNet-18 architectures, and are thus not applicable to ultra light-weight architectures, such as MobileNetV3, which are susceptible to accuracy collapse when binarized. While image segmentation methods ultimately predict a discrete class, alpha matting requires the dense accurate prediction of a continuous alpha value. Segmentation methods are therefore not easily transferable to the video matting task.\n\nPlease note that including those related works will not affect the main claims and findings of this paper.\n\n[1] Zhuang B, et al. Structured binary neural networks for accurate image classification and semantic segmentation. CVPR 2019.\n\n[2] Frickenstein A, et al. Binary dad-net: Binarized driveable area detection network for autonomous driving. ICRA, 2020.\n\n[3] Zhou X, et al. Cellular Binary Neural Network for Accurate Image Classification and Semantic Segmentation. IEEE TMM, 2022."}, {"review_id": "k2Zlomlwye", "reviewer": "Reviewer_busB", "summary": "This paper propose an efficient solution that utilizes binarization to achieve real-time video matting on for devices constrained by computational resources. The proposed BiMatting constructs shrinkable and dense topologies of the binarized encoder block to enhance the extracted representation, while sparsifying the binarized units to reduce the low-information decoding computation. Extensive experiments shows that it outperforms SOTA binarized video matting methods by a large margin.", "questions": "See weaknesses.", "limitations": "NA\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This work claimed itself as the first binarization solution for video matting tasks, which may provide a new effective solution to the real-time matting community.\nThe work is based on reasonable analysis and observation that shown in the Section 3.1. The proposed method also effectively address the pointed challenge. Moreover, although the performance is lower than the full-precision models, the proposed method achieves satisfactory results compared with the binarized video matting models.\n", "weaknesses": "\n1. The authors need to check the equations and make sure that all the notations are explained. For example, the ⊗ in Eq. (2) is not mentioned in the passage. The |W| is also remained unexplained.\n\n2. Some key ideas are not fully verified by experiments. For examples, as the author claims that dense connection is very important to recover the performance, this assertion is somewhat unwarranted without the comparison of the proposed SBB and SBB w/o dense skip connections for each BiConv. \n\n3. Since the Matting training phase has 4 stages, and each stage has its own independent training settings, the reviewer doubted that whether the Matting training phase can work well without extensive ablation study of different training settings. It would be better to provide experiments to demonstrate the effectiveness and robustness of the proposed method under different training settings. Otherwise, the robustness of the proposed method is hindered.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1**: The authors need to check the equations and make sure that all the notations are explained. For example, the ⊗ in Eq. (2) is not mentioned in the passage. The |W| is also remained unexplained.\n\n**A1**: $|{\\mathbf{W}}|$ means to take the absolute value of the weight ${\\mathbf{W}}$, $\\otimes$ denotes the inner product of two binary vectors with bitwise XNOR and Bitcount operations. We will clarify our notation in the paper. \n\n> **Q2**: Some key ideas are not fully verified by experiments. For example, as the author claims that dense connection is very important to recover the performance, this assertion is somewhat unwarranted without the comparison of the proposed SBB and SBB w/o dense skip connections for each BiConv.\n\n**A2**: Thanks for pointing it out, we followed your suggestion to include more ablation experiments to show that dense connections of SBB are important to restore performance. \n\nWe compared our BiMatting with the binarized model removing connections in SBB (BiMatting-NoConn) in *Table 2 of the attached PDF*. We find that after removing the connections, the performance of the binarized video matting model dropped significantly, which means that the connections in SBB are impossible. We also ablate the connection in SBB with the 1x1 binarized convolutions (BiMatting-BiConv) and summing up (BiMatting-SumConn), and then conduct the ablation experiments. The results in *Table 2 of the attached PDF* show it does not bring significant improvements while the binarized convolutions incur additional computation. The results further demonstrate the strengths of our proposed SBB, and we will update these results and discussion in our final version.\n\n> **Q3**: Since the Matting training phase has 4 stages, and each stage has its own independent training settings, the reviewer doubted that whether the Matting training phase can work well without extensive ablation study of different training settings. It would be better to provide experiments to demonstrate the effectiveness and robustness of the proposed method under different training settings. Otherwise, the robustness of the proposed method is hindered.\n\n**A3**: Please note that our training pipeline completely follows that of the baseline RVM, using the code from their public GitHub repository. We do not add additional training stages or other complications. We also adopt the same stopping conditions as RVM for a fair comparison. We also provide the detailed training pipeline of BiMatting in our General Response and will release our training code in the final version.\n\nWe also provide the accuracy of checkpoints at the end of every training stage in Table A3. The results show that the accuracy of BiMatting is steadily increasing at each stage, and it already shows obvious advantages in the first few stages over existing binarized video matting models (RVM-BNN, RVM-DoReFa, RVM-ReCU, and RVM-ReAct). This phenomenon means that the results of our BiMatting are robust.\n\nTable A3 (*Table 3 of the attached PDF*): Low-resolution comparison on VM, D646, and AIM datasets for each stage.\n|         |           |       |       |       |       | Alpha |       |       | FG    |\n|---------|-----------|-------|-------|-------|-------|-------|-------|-------|-------|\n| Dataset | Method    | Stage | \\#Bit | MAD   | MSE   | Grad  | Conn  | dtSSD | MSE   |\n| VM      | BiMatting (Ours) | 1     | 1     | 15.06 | 8.75  | 2.83  | 1.76  | 2.70  | -     |\n| 512x288 | BiMatting (Ours) | 2     | 1     | 13.50 | 7.02  | 3.32  | 1.52  | 2.69  | -     |\n|         | BiMatting (Ours) | 3     | 1     | 12.75 | 7.03  | 2.78  | 1.41  | 2.64  | -     |\n|         | BiMatting (Ours) | 4     | 1     | 12.82 | 6.65  | 2.97  | 1.42  | 2.69  | -     |\n| D646    | BiMatting (Ours) | 1     | 1     | 61.52 | 52.10 | 11.70 | 16.21 | 2.59  | 22.30 |\n| 512x512 | BiMatting (Ours) | 2     | 1     | 82.81 | 73.84 | 12.36 | 21.80 | 2.40  | 24.75 |\n|         | BiMatting (Ours) | 3     | 1     | 66.98 | 59.05 | 12.06 | 17.61 | 2.52  | 23.59 |\n|         | BiMatting (Ours) | 4     | 1     | 32.74 | 24.48 | 9.34  | 8.62  | 2.21  | 5.86  |\n| AIM     | BiMatting (Ours) | 1     | 1     | 54.26 | 44.24 | 13.31 | 14.30 | 2.41  | 23.40 |\n| 512x512 | BiMatting (Ours) | 2     | 1     | 61.19 | 51.44 | 13.99 | 16.11 | 2.21  | 24.12 |\n|         | BiMatting (Ours) | 3     | 1     | 63.19 | 53.88 | 13.69 | 16.59 | 2.30  | 20.23 |\n|         | BiMatting (Ours) | 4     | 1     | 35.17 | 26.53 | 9.42  | 9.24  | 1.82  | 7.00  |\n"}, {"review_id": "sjZiL4XgnR", "reviewer": "Reviewer_N9ej", "summary": "The authors propose the first binarized video matting network, namely BiMatting. They first analyze the bottlenecks of the direct binarization of video matting models and propose an accurate and efficient binarization method. Compared with other full-precision neural networks and other binarization methods, the authors confirm the effectiveness and great potential of BiMatting. It is worth noting this paper constructs a binary backbone that achieves similar acceleration compared to binarized mobilenetv3, a super-lightweight architecture, for the first time and realizes the practical application on video matting. The proposal and application of this binary network mean that the practicality of binarization has been significantly improved in general.", "questions": "Please see weaknesses\n\n--After rebuttal--\nThanks for the detailed response, which has well addressed my concerns. I also read other reviewers' comments and the authors' responses. I am satisfied with the rebuttal and increase the rating score.", "limitations": "The authors discuss the limitations of their works in section 4.3.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The authors propose the first binary neural network for video matting tasks for reducing computation consumption significantly while retaining practical accuracy. Since video matting tasks usually run on resource-constrained devices, this research is practically significant.\n\nIn terms of method, the authors successfully design an ultra-lightweight video matting network by binarization and make it have a significantly improved accuracy rate. In general, the authors' method tightly combines video matting tasks and architectures.\n\n1. The binary backbone with SBB is interesting, and I think it is the major contribution of this paper. Compared with the existing binarization schemes, the authors create a backbone network with reasonable accuracy and comparable efficiency with the binarization mobilenetv3 through the careful design of the architecture. This is very important for the practical application of binary neural networks.\n\nMoreover, the authors’ motivation (or paradigm) for designing the binary backbone network has a general contribution to the binarization community, that is, \"the crucial paradigm of an accurate binarized encoder is the computation-dense form of binarized block\", this motivation can explain the success of binary networks such as Bi-Real and IR-Net, and may lead to more lightweight binary networks with higher accuracy.\n\n2. Using representations of different scales, SAB successfully uses sparse masks to further reduce the computational load of the binary decoder without significant performance degradation. This demonstrates that the representational ability of the binarized computational unit allows the model to exploit less informative representations.\n\nThe experimental and video results are solid. The accuracy of BiMatting not only far exceeds the binarized version of the existing model but also exceeds the full-precision matting model such as BGMv2 while achieving significant storage savings. The visualization results also show that BiMatting has improved significantly in detail.", "weaknesses": "1. For the backbone with SBB, the authors should provide more detailed information. \nFirst of all, I suggest the authors give a more detailed ablation about the efficiency, including the FLOPs and the number of parameters of direct full-precision/binarized mobilenetv3 and SBB backbone. \nIn addition, since the authors add the pre-training process on the ImageNet dataset as stage 0, they should also provide the pre-training ablation results on this dataset, which has clarified the performance improvement of the binarized backbone alone.\n\n2. For SAB, a strange phenomenon is that the results in Table 1 show that using SAB in the direct binarized matting model will cause a crash. Although SAB does not directly target the accuracy bottleneck, it even causes the model to be worse than direct binarization. Can the authors explain this phenomenon?\n\n3. In Table 2, ReCU and BNN present the same collapse, however, as reported in their original papers, the former usually performs significantly better than the latter on ImageNet, even better than methods such as DoReFa-Net. I suggest the authors discuss this phenomenon and report the ImageNet pre-training accuracy of these binary beckbones.\n\n4. I suggest the authors discuss the possibility of BiMatting's actual deployment, including how to deploy it on edge devices and possible hardware inference performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1**: For the backbone with SBB, the authors should provide more detailed information. First of all, I suggest the authors give a more detailed ablation about the efficiency, including the FLOPs and the number of parameters of direct full-precision/binarized mobilenetv3 and SBB backbone. In addition, since the authors add the pre-training process on the ImageNet dataset as stage 0, they should also provide the pre-training ablation results on this dataset, which has clarified the performance improvement of the binarized backbone alone.\n\n**A1**: We thank you for your attention and provide detailed information on BiMatting following your suggestions. We first compare the FLOPs and number of parameters of full-precision/binarized MobileNetV3 and SBB backbones, as well as their pre-training accuracy on ImageNet in Table A1. The results show that there will be severe performance degradation with the direct binarization of MobileNetV3, which directly causes the performance degradation of binarized video matting models. Our SBB backbone in BiMatting achieves significantly improved pre-training performance (56.1% vs. MBV3-BNN 24.62%) with 8.9x speedup and 21.4x compression ratio.\n\nTable A1 (*Table 4 of the attached PDF*): Pretrained results comparison on ImageNet.\n| Backbone              | \\#Bit | \\#FLOPs(G) | \\#Param(M) | \\#Accurary@1 |\n|--------------------|-------|-------------------|--------------------|--------------|\n| MBV3               | 32    | 1.07              | 11.34               | 63.00        |\n| MBV3-BNN           | 1     | 0.07              | 0.46               | 24.62        |\n| MBV3-DoReFa           | 1     | 0.07              | 0.46               | 22.38        |\n| MBV3-ReCU         | 1     | 0.08              | 0.53               | 32.76        |\n| SBB                | 1     | 0.12              | 0.53               | 56.09        |\n\n> **Q2**: For SAB, a strange phenomenon is that the results in Table 1 show that using SAB in the direct binarized matting model will cause a crash. Although SAB does not directly target the accuracy bottleneck, it even causes the model to be worse than direct binarization. Can the authors explain this phenomenon?\n\n**A2**: We hereby explain this phenomenon. As mentioned in our paper Sec 4.1 L279, the reason for the poor results of RVM + SAB is that SAB is designed to break the bottleneck of efficiency rather than accuracy, which leads to its cooperation with the original encoder or even results in poorer accuracy. Our SAB reduces the calculation of the decoder by masking the repeated intensive computation of continuous regions, which means that the remaining (not masked) regions in the feature need to provide enough effective information, and these features are provided by the encoder. Fig. 3 shows that when using the original encoder (directly binarized MobileNetV3), even if all other parts are restored to the full-precision counterparts, the performance still drops significantly. Therefore, the crashing can be expected when we use the original binarized encoder together with our efficient and lightweight (binarized) SAB decoder.\n\n> **Q3**: In Table 2, ReCU and BNN present the same collapse, however, as reported in their original papers, the former usually performs significantly better than the latter on ImageNet, even better than methods such as DoReFa-Net. I suggest the authors discuss this phenomenon and report the ImageNet pre-training accuracy of these binary backbones.\n\n**A3**: We further show the results of ImageNet pre-training in Table A1 according to the reviewer's suggestion. From these results, we can see that the backbone network of RVM-ReCU is significantly better than RVM-BNN. These results imply that the ReCU binarization method will fail in the video matting task, although it belongs to SOTA methods in the evaluation on ImageNet. Some existing work [1] also shows that the transfer of SOTA binarization methods on different tasks is not straightforward. For example, ReCU also leads to model collapse on 3D ShapeNet and GLUE benchmark tasks. It further verifies our motivation, *i.e.*, accurate and efficient binarization models should be tailored for the video matting task.\n\n[1] Qin H, et al. BiBench: Benchmarking and Analyzing Network Binarization. ICML 2023.\n\n> **Q4**: I suggest the authors discuss the possibility of BiMatting's actual deployment, including how to deploy it on edge devices and possible hardware inference performance.\n\n**A4**: The deployment of 1-bit BiMatting is supported by open-source libraries such as Larq [2] and daBNN [3] for ARM devices. Referring to the daBNN library and the performance of binarized operators implemented by it on the Raspberry Pi 4B, we believe that the binarized BiMatting can achieve similar speedup and compression on the hardware claimed in our paper. \n\n[2] Geiger L, Team P. Larq: An open-source library for training binarized neural networks. JOSS, 2020.\n\n[3] Zhang J, et al. dabnn: A super fast inference framework for binary neural networks on arm devices. ACM MM, 2019."}]

Author rebuttal:
We deeply appreciate all reviewers for the positive reviews and constructive feedback. All reviewers agree that our BiMatting is highly efficient and contributes to both video matting and binarization fields significantly. Your expertise and insightful comments greatly help us to further improve our paper. 

**Training details**:

Here, we first clarify the training details of our BiMatting, which was asked by three reviewers (Reviewer PTWg, wDXB, and busB).

We want to emphasize that our training pipeline strictly follows the baseline Robust Video Matting (RVM) for a fair comparison, using the code provided in their public GitHub repository. In particular, the training configurations and commands for BiMatting are identical to those outlined in the train.py#L8-L75 of the repository, without any modifications. Furthermore, we have made the complete network definition of BiMatting available in the BiMatting_code/model folder in the supplementary materials, making it easy for others to train and reproduce our results.

Additionally, since the RVM baseline uses MobileNetV3 pre-trained on the ImageNet dataset by default (from PyTorch official), our SBB backbone of the BiMatting network is equivalently pre-trained on ImageNet. Thus, we do not add any additional training stages or other complications. In the final version, we will release our complete training code and saved checkpoints, further facilitating the training and reproduction of BiMatting.

In the following responses to each reviewer, we provide detailed answers to all the questions raised.

Meta-review: The paper proposes an efficient video matting approach. The reviewers highlighted the analysis done in the paper, substantial improvements in latency, and storage, while maintaining strong performance. Throughout the discussion the reviewers kept being positive about the paper and its contribution. The authors did a good job in addressing the questions and concerns during the discussion stage. Hence the paper got positive ratings from all the reviewers and acceptance is recommended. Congrats!
Decision: Accept (poster)