Title: BiMatting: Efficient Video Matting via Binarization
Venue: NeurIPS 2023
Accepted: yes
Keywords: Video Matting

Model Binarization

Deep Learning
Abstract: Real-time video matting on edge devices faces significant computational resource constraints, limiting the widespread use of video matting in applications such as online conferences and short-form video production. Binarization is a powerful compression approach that greatly reduces computation and memory consumption by using 1-bit parameters and bitwise operations. However, binarization of the video matting model is not a straightforward process, and our empirical analysis has revealed two primary bottlenecks: severe representation degradation of the encoder and massive redundant computations of the decoder. To address these issues, we propose BiMatting, an accurate and efficient video matting model using binarization. Specifically, we construct shrinkable and dense topologies of the binarized encoder block to enhance the extracted representation. We sparsify the binarized units to reduce the low-information decoding computation. Through extensive experiments, we demonstrate that BiMatting outperforms other binarized video matting models, including state-of-the-art (SOTA) binarization methods, by a significant margin. Our approach even performs comparably to the full-precision counterpart in visual quality. Furthermore, BiMatting achieves remarkable savings of 12.4$\times$ and 21.6$\times$ in computation and storage, respectively, showcasing its potential and advantages in real-world resource-constrained scenarios. Our code and models are released at https://github.com/htqin/BiMatting .

BiMatting: Efficient Video Matting via Binarization

Abstract

Real-time video matting on edge devices faces significant computational resource constraints, limiting the widespread use of video matting in applications such as online conferences and short-form video production. Binarization is a powerful compression approach that greatly reduces computation and memory consumption by using 1-bit parameters and bitwise operations. However, binarization of the video matting model is not a straightforward process, and our empirical analysis has revealed two primary bottlenecks: severe representation degradation of the encoder and massive redundant computations of the decoder. To address these issues, we propose BiMatting , an accurate and efficient video matting model using binarization. Specifically, we construct shrinkable and dense topologies of the binarized encoder block to enhance the extracted representation. We sparsify the binarized units to reduce the low-information decoding computation. Through extensive experiments, we demonstrate that BiMatting outperforms other binarized video matting models, including state-of-the-art (SOTA) binarization methods, by a significant margin. Our approach even performs comparably to the full-precision counterpart in visual quality. Furthermore, BiMatting achieves remarkable savings of 12.4 × and 21.6 × in computation and storage, respectively, showcasing its potential and advantages in real-world resource-constrained scenarios. Our code and models are released at https://github.com/htqin/BiMatting.

1 Introduction

The success of deep neural networks has led to remarkable advancements in computer vision tasks, including video matting [1, 2, 3, 4, 5, 6, 1, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]. However, many practical applications based on deep networks require real-time processing with minimal latency, which is challenging due to the high computational and storage demands. To address this issue, researchers have developed lightweight video matting algorithms such as Robust Video Matting (RVM) [1] and BGMv2 [10]. While these methods achieve significant speedups and memory reductions, they still rely on expensive floating-point operations, leaving room for further compression.

One of the most promising approaches to improving the efficiency of neural networks is network binarization [17, 18, 19, 20, 21, 22, 23, 24, 25, 26]. Binary neural networks (BNNs) have emerged as a highly effective technique for optimizing

0

15

30

45

200

10 -1

DeepLabV3

(32-bit)

�

BGMv2

(32-bit)

19.4MB

�

0.64MB

RVM-ReActNet

(1-bit)

�

0.57MB

RVM-BNN

(1-bit)

�

0.57MB

RVM-DoReFa

(1-bit)

�

BiMatting

(1-bit)

0.67MB

14.5MB

RVM

(32-bit)

�

10 1

10 0

FLOPs [G]

10 3

10 2

neural networks by reducing parameter bit width to 1-bit. BNNs leverage compact binary parameters that occupy less memory space and use efficient bitwise operations, which are much less computationally expensive than floating-point operations. By employing the binarization approach, researchers can significantly reduce the computational and storage demands of video matting applications.

Model

DeepLabV3

BGMv2

RVM

bs1-bnn bs1-dorefa

bs1-reactnet ours

FLOPs(G)

136

8.46

4.57

0.50

0.52

0.55

0.37

Ma

14.

25.

6.0

189

51.

28.

12.

Encoder

SBB

(16, 32)

Encoder

SBB

×

2

(32, 64)

Encoder

SBB

×

2

(64, 1024)

Conv (3, 16)

ASPP

(1024, 128)

Decoder

neck

(1/16)

Decoder

SAB

(1/8)

Decoder

SAB

(1/4)

Decoder

SAB

(1/2)

Decoder

SAB

(1)

Downsample

Input

DGF

Output

Shrinkable Binarized Block

(SBB)

sparse mask guidance

Sparse-Assisted Binarization

(SAB)

SA-BiConv

3

×

3

BiConv

1

×

1

feature

mask

sparse-feature

⊕

𝐁𝐁𝐁𝐁𝐁𝐁𝐁𝐁𝐁𝐁𝐯𝐯

𝟏𝟏

(

𝑘𝑘

,

𝑘𝑘

)

𝐁𝐁𝐁𝐁𝐁𝐁𝐁𝐁𝐁𝐁𝐯𝐯

𝟑𝟑

(

𝑘𝑘

,

𝑘𝑘

)

𝐁𝐁𝐁𝐁𝐁𝐁𝐁𝐁𝐁𝐁𝐯𝐯

𝟏𝟏

(

2

𝑘𝑘

,

𝑘𝑘

)

𝐁𝐁𝐁𝐁𝐁𝐁𝐁𝐁𝐁𝐁𝐯𝐯

𝟑𝟑

(

2

𝑘𝑘

,

2

𝑘𝑘

)

a

vg

⊕

⊘

Shrinkable Binarized Block (SBB) Shrinkable Binarized Block (SBB) ⊕ ⊘ Despite that the generic binarization methods have been studied extensively by the binarization community, the direct binarization for existing video matting models still leaves severe accuracy and efficiency bottlenecks. First, to construct an efficient architecture, many video matting methods contain encoders with lightweight full-precision backbones to obtain intermediate features. For example, MobileNet architectures are widely applied as lightweight extractors in several video matting models to provide strong features [1, 11]. However, this practice introduces an accuracy bottleneck in the binarized video matting model. Our analysis shows that the computing units and topology of existing lightweight encoder architectures are unfriendly to binarization, leading to severe degradation in the quality of intermediate features extracted by its binarized counterpart. Second, we identify that the architecture of the existing video matting decoder [1] causes immense redundant computation at the multi-scale decode stage. Intensive computation is still needed in large portions of certain foreground or background regions (spatial regions outside unknown parts of the trimap) of the high-resolution feature maps. This weakens the savings brought by binarization, which hinders efficiency improvements in the context of extremely compressed bit-width.

In this paper, we provide empirical studies of the above-mentioned bottlenecks. This leads us to propose BiMatting , a Bi narized model for accurate and efficient video Matting (overview in Fig. 2). To tackle the accuracy bottleneck, we first investigate the limitations of existing binarized encoders in representation extraction. Then, we propose Shrinkable Binarized Block (SBB), which follows a binarization-friendly computation-dense paradigm to construct a flexible block structure. SBB effectively extracts high-quality features with improved topology and operators. In addition, given a reliable binarized encoder, we further develop Sparse-Assisted Binarization (SAB) to effectively reduce the computational consumption of the binarized decoder. The overall burden is thus greatly reduced. SAB removes repeated computation by spatial sparseness [27] while preserving the accuracy of binarized units, resulting in a notable computation reduction without compromising performance.

Our BiMatting is the first binarization solution for video matting tasks, which surpasses 1-bit matting models using existing binarization algorithms by a significant margin [21, 17, 23, 18]. Notably, BiMatting even outperforms some full-precision models [28, 11] in terms of accuracy while being highly efficient. Our extensive experiments on fundamental tasks across VideoMatte240K (VM) [10], Distinctions-646 (D646) [29], and Adobe Image Matting (AIM) [30] datasets demonstrate that the advantages of BiMatting are task-independent. In addition, our SBB and SAB components are highly efficient, allowing BiMatting to achieve 12.4 × FLOPs and 21.6 × storage savings compared to the full-precision counterpart, leading a promising way for the video matting on edge scenarios (Fig. 1).

2 A Baseline for Binarized Video Matting

In this section, we first build a baseline to study the binarized video matting model. The baseline is based on straightforward binarization and an existing lightweight video matting architecture.

2.1 Binarization Framework

Binarization applies the sign function to compress weights and activations to 1-bit for computing units of the binarized network [17, 19, 20, 22, 31], and the propagation process can be expressed as

0

1

2

3

4

5

6

Bi-Decoder

Bi-DGF

Bi-All

#Param (M)

#FLOPs (G)

(a) Accuracy perspective

0

5

10

15

20

25

30

Original

Bi-Decoder

Bi-DGF

Bi-All

MAD

(b) Efficiency perspective

where L as the loss function. To build a strong binarized baseline by existing techniques, we apply floating-point scaling factors for weights W [19, 23] and learnable thresholds for activations A [23]:

where B w and B a denote the binarized weight and activation, respectively. The channel-wise α is obtained by mean( | W | ) , and the layerwise threshold τ is initialized as 0. Note that the StraightThrough Estimator (STE) [32] is uniformly applied to all experiments and binarization methods, approximating the gradient of the sign function as clip function (as the backward in Eq. (1)), since most other estimators consume unaffordable GPU memory [31, 22]. Binarization is applied to computation layers except for those who access original inputs or produce final outputs.

2.2 Video Matting Architecture

Matting model aims to break down a frame I into a foreground F and a background B , using an α coefficient to represent the linear combination of the two [1, 7],

Compared to image matting [2, 3, 4, 5, 6, 33, 34, 35, 36], matting methods specifically developed for video sequence should be more effective in utilizing spatial-temporal information in videos [1, 7, 8, 9, 10, 11, 37, 38]. Among them, the recent Robust Video Matting (RVM) [1] method achieves the SOTA accuracy of the video matting task and stands out significantly in terms of efficiency. The architecture of RVM mainly comprises a lightweight MobileNetV3-based encoder (including backbone and Atrous Spatial Pyramid Pooling (ASPP) [39] module), a recurrent decoder, and a deep guided filter (DGF) [40] module. The concise architecture brings efficiency, even realizing real-time matting at high resolution. While RVM is one of the lightest video matting models available, there is still significant potential for compression in terms of reducing bit width, given the high cost of both the floating-point parameters and computations involved.

3 The Rise of BiMatting

3.1 Bottlenecks of Binarized Video Matting Baseline

Our goal is to achieve practical and resource-efficient video matting via binarization. However, the encoder and decoder of the binarized baseline pose accuracy and efficiency bottlenecks, respectively.

From an accuracy perspective , Fig. 3 (a) compares the accuracy drops on the VideoMatting (VM) dataset by binarizing every single part in the RVM model. Specifically, we find that binarizing the existing lightweight MobileNetV3 backbone in the encoder causes the most significant drop in accuracy among all parts, with a drop almost equivalent to directly binarizing the entire network (Bi-All 28.49 vs. Bi-Backbone 23.56 for MAD metric). In contrast, binarizing the ASPP, decoder, and DGF parts produce less harm to accuracy (less than a 3.05 MAD increase). Thus, improving the backbone of the encoder to make it more amenable to binarization is of the highest priority in addressing the accuracy drop in the binarized video matting model.

From the efficiency perspective , we show in Fig. 3 (b) the computation and storage usage for each part of the binarized video matting model to demonstrate the individual consumption in efficiency. Our analysis from an efficiency perspective reveals that the decoder consumes a significant amount

of computational resources, accounting for 71.8% (0.38G) FLOPs with only 12.1% parameters after binarization. Conversely, the binarized encoder has a higher parameter count (81.8%), but its FLOP consumption is only 16.6%. Negligible consumption is observed for the other parts. Based on these observations, it is evident that the computational redundancy of the decoder significantly impacts the acceleration performance of the overall model. Thus, there is ample room for computational optimization after direct binarization to further enhance the model's efficiency.

Given the observations from the aforementioned experiments, we find that the current baseline presents two major challenges: 1) the encoder's lightweight backbone is unsuitable for binarization and fails to generate practical features, and 2) the decoder continues to exhibit considerable computational redundancy even after binarization. Thus, we propose a binarization-friendly encoder and a computationally-efficient decoder to construct an accurate and efficient binarized matting model.

3.2 Shrinkable Binarized Block

3.2.1 Binarization-evoked Encoder Degradation

The binarized video matting baseline in Sec. 2 employs a lightweight MobileNetV3 as the encoder's backbone and directly binarizes its convolutional and linear layers. However, the extensive use of groupwise and pointwise convolutions presents significant challenges to binarization [41, 23], resulting in severe accuracy degradation. Fig. 3 shows that directly using binarized MobileNetV3 leads to over 4 × MADmetric degradation. On the other side, existing binarized networks are also far from the efficiency level of binarized MobileNetV3, making it hard to be transferred straightforwardly to construct the binarized matting model. For example, some binarized MobileNet-based models suffer at least 3-8 × FLOPs than binarized MobileNetV3 [23, 41, 26]. Therefore, a new binarized backbone is necessary for feature extraction to achieve high-quality features for the decoder while keeping the model ultra-lightweight.

We first analyze the degradation of the binarized encoder in existing baselines. We note the binarized convolution as BiConv n ( · ) or GBiConv n ( · ) , where n is the kernel size of the convolution and G mark denotes groupwise. Their superscripts up , eq , and dn indicate the number of their output channels is greater than, equal to, or less than that of their input channels, respectively. Then the binarized MobileNetV3 block in the baseline's encoder can be expressed as

where x and o denote the input and output, respectively, kernel size n ∈ { 3 , 5 } , and [ · ] denotes the Iverson bracket [42], evaluating to 1 if the condition inside the parentheses is true, and 0 otherwise. The batch normalization and the activation layers following convolutions are omitted.

Eq. (4) suggests at least two issues that impede accurate binarization. Firstly, all convolutions are groupwise or pointwise that have fewer parameters than regular convolutions, thus are sensitive to binarization and hard to recover from crashes caused by mutual influence. Secondly, utilizing only per-block short connection is insufficient since convolution-specific shortcuts are critical to achieving performance recovery [22, 23]. Several existing binarization methods aim to address the aforementioned issues, e.g. , applying regular convolutions instead of grouped ones [23, 26, 41], or creating shortcuts for channel-constant convolutions [23, 41]. Though these techniques may be effective, they are significantly more computational-expensive than directly binarized MobileNetV3 and still suffer representation loss. Therefore, it is necessary to develop stronger feature extraction encoders for binarized video matting models to address these limitations.

3.2.2 Shrinkable Binarized Block for Accurate Encoder

To mitigate the degradation of the encoder caused by binarization while keeping it lightweight, we present an efficient Shrinkable Binarized Block (SBB) for building a binarized video matting encoder, which is conducive to a flexible and lightweight architecture via channel-shrinkable design while retaining the representations and gradients.

Based on the analysis in Sec. 3.2.1, we determine that the crucial paradigm of an accurate binarized encoder is the computation-dense form of binarized block. In terms of topology, we ensure that every binarized convolution has a corresponding/dense connection to preserve the representation

accurately. In terms of the operator, introducing computation-dense regular convolutions can prevent groupwise and pointwise ones from becoming exclusive. Following this paradigm, SBB recovers the representation in the binarized block such that enables the backbone to effective feature extraction in flexible dimensional space.

Specifically, we first introduce the sub-blocks of SBB. Based on the above analysis, we find that the key element to constructing the flexible binarized architecture is to allow various sub-blocks in the feature extractor to freely adjust (increase, maintain, or shrink) the channel number of the outputs, while the shortcut should accompany with each binarized convolution to maintain representation. Therefore, the channel-shrunk SBB sub-block can be first constructed as follows:

where x ′ ( m,n ) denotes taking the m to n channels of x ′ . In the sub-block, we introduce shortcuts for channel-shrunk binarized convolution with channel splitting and averaging operations, allowing the channel to shrink while constructing a shortcut with negligible overhead. This design enables the binarized encoder to leverage computing units with higher output channels (such as 32 or 64) for extracting low-channel representations (16 or 32, correspondingly), resulting in dependable features for the decoder. Inspired by [23, 41], the channel-maintain and increased sub-blocks are as follows:

where ⊕ denotes the concatenate operation. Benefiting from the above sub-block variants, SBB can implement a flexible feature extraction architecture.

Next, we create SBB by assembling these sub-blocks. Each SBB is made up of three sub-blocks: the head, middle, and tail. The head and tail sub-blocks produce identical output channels, while the middle sub-block extracts feature in higher dimensions with double output channels:

where x ′ is the output of the head sub-block that varies θ eq or θ up depending on the channel constraint in Iverson brackets . The binarized sub-blocks in the middle and at the end of the sequence respectively increase and decrease the feature dimension for thorough extraction, while also incorporating shortcut across sub-blocks that allows the representations to remain intact. Using the block-level crossing shortcut also mitigates the influence of the splitting and averaging procedure on the representations in the trail channel-contracted sub-block.

3.3 Sparse-Assisted Binarization

3.3.1 Computational Decoder Redundancy

In Sec. 3.1, we discover that while binarizing the decoder prevents accuracy performance from deteriorating, it results in significant computational costs (greater than 71.8%) despite having a relatively small parameter size (less than 12.1%). This inefficiency creates a bottleneck for the binarized video matting model. This is attributed to the decoder where high-resolution features are computationally demanding. To illustrate, consider the decoder's output block, which uses two 3 × 3 convolutions to perform computations on the original scale's features. Since grouped convolutions that are unfriendly to binarization are not utilized here, the accuracy reduction of decoder binarization is not as severe as it is on the encoder. But the computation of this single block in the decoder (the last one in 5 decoder blocks) is even equivalent to 103% of the entire encoder in a binarized baseline.

However, it is worth noting that not all spatial computations are equally crucial. In the case of video matting, the majority of frames have large, uninterrupted foreground and background areas, which can usually be identified at a lower resolution. Nevertheless, the current decoder architecture repeats these computations at various scales, leading to significant computational overhead, particularly on high-resolution feature maps. This issue renders constructing an efficient decoder challenging, even

with the most aggressive 1-bit binarization approach. As a result, a fundamental redesign of the computation unit is required to create an efficient decoder that minimizes redundant computation while maintaining accurate matting results. Such a revamp should consider the context of binarization and focus on streamlining the computation process while preserving the quality of the output.

3.3.2 Sparse-Assisted Binarization for Efficient Decoder

To address the computational redundancy problem in the decoder of the binarized video matting model, we propose the Sparse-Assisted Binarization (SAB) method in BiMatting.

Fortunately, benefiting from the analysis in Sec. 3.3.1, the solution to the computational redundancy of the decoder is intuitive, that is, to reduce the repeated intensive computation of continuous regions in the decoder, especially at the high-resolution features. First, inspired by the sparse segmentation and spatial region classification of the trimap, we hypothesize that most computation inside the matting decoder should be used in the unknown alpha matting regions, instead of an equal distribution on the certain foreground/background regions of the whole image. To approximate such unknown regions, we compute the incoherent regions M inc following [27] using the low-resolution output mask M ∈ { 0 , 1 } N 16 × N 16 from the first decoder block, where N denotes the original input scale.

Incoherent regions are primarily found along object instance boundaries or high-frequency areas (blue grids of the sparse feature in Fig. 2), and the points inside the incoherent regions are considered to be uncertain in alpha estimation. The remaining regions outside of the incoherent regions can be skipped/reduced in computation. Since the features of this process are the lowest resolution among the 4 sets of features input by the decoder, we obtain the mask M inc at a low cost.

Then, we employ the detected mask M inc to transform the convolutions in high-resolution blocks into sparse-assisted binarized convolutions for efficient computation. Our designed masked sparse convolution is different from [27, 43] on performing pointwise attention for the whole incoherent regions, resulting in a more binarization-friendly [44] and computation-efficient design. Specifically, we transform the 3 × 3 convolution in the up-sample block of the decoder into:

where SA-BiConv 3 ( · ; M ) represents the sparse-assisted binarized 3 × 3 convolution under the guidance of sparse mask M inc, in which the weight and sparse activation are binarized, and bilinear k denotes a bilinear interpolation up-sampling operation, and the low-resolution sparse mask M inc is up-sampled by k times, k ∈ { 2 , 4 , 8 , 16 } . For this binarized convolution SA-BiConv 3 , the implementation of sparse computation follows [45, 27] to skip M inc ( x, y ) = 0 masked regions during inference. However, this does not change that the essence of this convolution is still a non-grouped 3 × 3 convolution, which retains binarization-friendliness in practice. Moreover, as in Eq. (8), besides the sparse-assisted binarized 3 × 3 convolution, we also apply one 1 × 1 binarized convolution layer to process extraction in the whole feature and fuse the output obtained to that of SA-BiConv to guide the finer predictions of continuous regions.

Our SAB first obtains a sparse binary mask prediction based on low-resolution features, which is then used to assist sparse binarization in the higher-resolution features of the decoder. As validated in the experiment section (Tab. 1), SAB greatly reduces the computation of the binarized video matting model by reducing computational FLOPs by 30% while maintaining the accuracy of video matting.

3.4 Training Pipeline of BiMatting

We present the training pipeline for our BiMatting model, which involves additional training steps and iterations to ensure complete convergence of the binarized video matting model, as compared to the training of the full-precision RVM counterpart [1]. The training pipeline of our BiMatting is comprised of two phases, namely the pre-training phase and the matting training phase.

Pre-training phase: In the pre-training phase, we trained the binarized SBB backbone on the ImageNet classification dataset for 200 epochs to obtain a well-pre-trained backbone. This pretraining phase made it easier to converge during the matting training phase. Moreover, since the direct binarization of the full-precision pre-trained model, such as MobileNetV3, can lead to almost crashing, we apply the full pre-training phase for all compared binarized video matting models.

Matting training phase: This phase is binarization-aware and built following [1], divided into 4 stages. Stage 1 involves training on the low-resolution VM dataset for 20 epochs without DGF, with T = 15 frames for quick updates. The SBB backbone's learning rate is set as 1 e -4 and the rest as 2 e -4 . Additionally, the input resolution h, w is sampled independently from 256-512px for improving robustness. In Stage 2 , the network is trained with T = 50 , with halved learning rate and 2 more epochs to enable learning of long-term dependencies. In Stage 3 , the DGF module is attached, and 1 epoch is trained on both low-resolution long and high-resolution short sequences from the VM dataset. The low-resolution pass is T = 40 , h and w are the same with stage 1 without DGF, while the high-resolution pass employs DGF with downsample factor s = 0 . 25 , ˆ T = 6 , and ˆ h, ˆ w ∼ (1024 , 2048) . The learning rate of DGF is 2 e -4 , and that of the rest is 1 e -5 . In Stage 4 , the network is trained for 5 epochs on D646 and AIM, increasing the decoder's learning rate to 5 e -5 .

4 Experiments

We extensively evaluate the accuracy and efficiency of our proposed BiMatting. We first ablate our method and illustrate the contributions of SBB and SAB on the VM [10] dataset. Then we compare BiMatting with binarized video matting models that utilize existing binarization techniques on VM[10], D646 [29], and AIM [30], where our designs excel and even outperform some full-precision video matting models. Concerning efficiency, BiMatting impressively reduces computational FLOPs and model size by 11 . 2 × and 21 . 6 × , respectively. On metrics, we evaluate α using mean absolute difference (MAD), mean squared error (MSE), spatial gradient (Grad), and connectivity (Conn) for quality, and dtSSD for temporal coherence. We also measure pixels where α > 0 by MSE for F [1]. All stages of our experiments use batch size 4 splits across 4 Nvidia A100 GPUs.

4.1 Ablation Study

Tab. 1 demonstrates that the binarized video matting baseline experiences a significant drop in performance across all accuracy metrics for VM data recall. Despite its 22.7 × parameter compression, the model only realizes an 8.3 × computational savings in the efficiency metric. Upon substituting the encoder with our SBB alone, the binary model's accuracy is considerably restored, affirming that the encoder is the principal performance bottleneck in the baseline. However, the decoder's implementation of the efficient SAB does not fully resolve the performance bottleneck, where the accuracy of the binarized model is perilously close to collapsing. By integrating both of our contributions, both accuracy and efficiency performance is substantially enhanced. Notably, in BiMatting, combining these two improvements can produce a remarkable enhancement in accuracy performance, underscoring the importance that the decoder should concentrate on less but crucial representations to heighten model performance by providing high-quality features.

4.2 Comparison Results

To create the comparison benchmark, we combine test samples from the VM, D646, and AIM datasets with 20 video backgrounds and 20 image backgrounds, following the settings in [1, 47]. Each test clip contains 100 frames where motion augmentation is applied to image samples. We compare our BiMatting model with other video matting models that have been binarized using existing methods. These binarization methods include classical BNN [41] and DoReFa [18], as well as state-of-the-art (SOTA) ReActNet [23] and ReCU [21], where the latter two are considered as best practices for generic binarization [44]. In addition, we have also included results from some full-precision video matting methods for comparisons, such as the RVM [1] with MobileNetV3 [48] backbone (oracle), DeepLabV3 [28], and background-based BGMv2 [10] with MobileNetV2 [49] backbone. To ensure

a fair comparison, we apply the exact same training pipeline to all binarized video matting networks as we did to BiMatting, as explained in Sec. 3.4. As for the full-precision video matting model, we follow the results reported in previous studies [1], unless otherwise specified.

Tab. 2 presents a comparison of methods that use low-resolution input. The findings indicate that applying BNN and ReCU directly to full-precision RVM leads to completely collapsed results. This is surprising given that the latter is among the SOTA binary methods, highlighting that binarizing existing video matting architectures is not a straightforward task. In contrast, our BiMatting model performs significantly better than all existing binarization models across all datasets, which predicts alpha with higher accuracy and consistency, resulting in more coherent and accurate performance. Further details will be presented in Sec. 4.3 with comprehensive visualizations. Furthermore, BiMatting even outperforms some 32-bit full-precision models when using only 1-bit limit bit width. For instance, BiMatting surpasses BGMv2 on VM, D646, and AIM datasets, as well as DeepLabV3 on

Input

RVM-ReActNet

BiMatting (ours)

RVM (32-bit)

Input

RVM-ReActNet

BiMatting (ours)

RVM (32-bit)

2

VMdatasets. These results demonstrate the enormous potential of binarization for efficient video matting. Tab. 3 presents a comparison between our BiMatting approach and other methods using highresolution datasets. Our method consistently outperforms existing binarization models and BGMv2 across multiple metrics, demonstrating the robustness of BiMatting under varying resolutions.

Moreover, BiMatting has demonstrated its high potential for video matting in efficiency with impressive accuracy. As Tab. 2 and Tab. 3 show, in comparison to the full-precision counterpart (RVM), BiMatting achieves a computational FLOPs savings of 12.4 times and parameter savings of 21.6 times, making it the most promising solution for edge deployment. Moreover, BiMatting outperforms existing binarization methods, which offer significant acceleration advantages with only a tiny parameter overhead, such as a 0.1M increase compared to RVM-BNN.

4.3 Visualization

Fig. 4 and Fig. 5 show qualitative comparisons of natural videos. Fig. 4 compares our BiMatting model with RVM-ReActNet (the

current SOTA binarized video matting model) and the full-precision RVM (can be seen as the 32-bit counterpart of BiMatting). In general, BiMatting's performance is close to the full-precision counterpart with greatly reduced resource consumption. Specifically, in Fig. 4(a), we conduct experiments on videos from diverse video frames. Our results reveal that BiMatting is more robust against semantic errors. Furthermore, BiMatting outperforms the other two models in matting edge regions. Fig. 4(b) compares temporal coherence, where our BiMatting consistently produces superior results, while RVM-ReActNet produces different areas of error. In Fig. 5, a comparison is made among the alpha predictions of various methods. Observe that our method outperforms the SOTA binarized video matting model by more accurately predicting intricate details, such as individual strands of hair.

Limitation . Though significantly improved, our BiMatting is not yet on par with its full-precision counterpart, the 32-bit RVM, particularly when it comes to local details such as the hair ends which tend to get blurred, while also favoring simpler backgrounds that lead to accurate matting results.

5 Conclusion

Our proposed model, BiMatting, is an efficient and accurate solution that utilizes binarization to achieve real-time video matting on edge devices constrained by computational resources. In this paper, we address the primary bottlenecks of binarization by constructing shrinkable and dense topologies for the binarized encoder block to enhance representation and sparsifying the binarized units to reduce redundant decoder computation. Our exhaustive experiments demonstrate that the proposed BiMatting outperforms existing binarized video matting models by a significant margin while producing a comparable performance to the full-precision counterpart in visual quality. BiMatting achieves significant savings in computation and storage, making it an attractive solution for real-world resource-constrained scenarios such as online conferences and short-form video production.

Acknowledgement This work was supported by the National Natural Science Foundation of China (No. 62022009), the State Key Laboratory of Software Development Environment (SKLSDE2022ZX-23).

References

Shanchuan Lin, Linjie Yang, Imran Saleemi, and Soumyadip Sengupta. Robust high-resolution video matting with temporal guidance. In WACV , 2022.

Yagiz Aksoy, Tunc Ozan Aydin, and Marc Pollefeys. Designing effective inter-pixel information flow for natural image matting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 29-37, 2017.

Xue Bai and Guillermo Sapiro. A geodesic framework for fast interactive image and video segmentation and matting. In 2007 IEEE 11th International Conference on Computer Vision , pages 1-8. IEEE, 2007.

Qifeng Chen, Dingzeyu Li, and Chi-Keung Tang. Knn matting. IEEE transactions on pattern analysis and machine intelligence , 35(9):2175-2188, 2013.

Yung-Yu Chuang, Brian Curless, David H Salesin, and Richard Szeliski. A bayesian approach to digital matting. In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001 , volume 2, pages II-II. IEEE, 2001.

Xiaoxue Feng, Xiaohui Liang, and Zili Zhang. A cluster sampling method for image matting via sparse coding. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14 , pages 204-219. Springer, 2016.

Jiachen Li, Vidit Goel, Marianna Ohanyan, Shant Navasardyan, Yunchao Wei, and Humphrey Shi. Vmformer: End-to-end video matting with transformer. arXiv preprint arXiv:2208.12801 , 2022.

Yanan Sun, Guanzhi Wang, Qiao Gu, Chi-Keung Tang, and Yu-Wing Tai. Deep video matting via spatio-temporal alignment and aggregation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6975-6984, 2021.

Yunke Zhang, Chi Wang, Miaomiao Cui, Peiran Ren, Xuansong Xie, Xian-Sheng Hua, Hujun Bao, Qixing Huang, and Weiwei Xu. Attention-guided temporally coherent video object matting. In Proceedings of the 29th ACM International Conference on Multimedia , pages 5128-5137, 2021.

Shanchuan Lin, Andrey Ryabtsev, Soumyadip Sengupta, Brian L Curless, Steven M Seitz, and Ira Kemelmacher-Shlizerman. Real-time high-resolution background matting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8762-8771, 2021.

Soumyadip Sengupta, Vivek Jayaram, Brian Curless, Steven M Seitz, and Ira KemelmacherShlizerman. Background matting: The world is your green screen. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2291-2300, 2020.

Zixiang Zhao, Jiangshe Zhang, Xiang Gu, Chengli Tan, Shuang Xu, Yulun Zhang, Radu Timofte, and Luc Van Gool. Spherical space feature decomposition for guided depth map super-resolution. In ICCV , 2023.

Zixiang Zhao, Jiangshe Zhang, Shuang Xu, Zudi Lin, and Hanspeter Pfister. Discrete cosine transform network for guided depth map super-resolution. In CVPR , 2022.

Zixiang Zhao, Jiangshe Zhang, Haowen Bai, Yicheng Wang, Yukun Cui, Lilun Deng, Kai Sun, Chunxia Zhang, Junmin Liu, and Shuang Xu. Deep convolutional sparse coding networks for interpretable image fusion. In CVPRW , 2023.

Zixiang Zhao, Shuang Xu, Jiangshe Zhang, Chengyang Liang, Chunxia Zhang, and Junmin Liu. Efficient and model-based infrared and visible image fusion via algorithm unrolling. IEEE TCSVT , 2022.

Zixiang Zhao, Haowen Bai, Yuanzhi Zhu, Jiangshe Zhang, Shuang Xu, Yulun Zhang, Kai Zhang, Deyu Meng, Radu Timofte, and Luc Van Gool. Ddfm: Denoising diffusion model for multi-modality image fusion. In ICCV , 2023.

Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural networks. In NeurIPS . 2016.

Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. CoRR , abs/1606.06160, 2016.

Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In ECCV , 2016.

Adrian Bulat and Georgios Tzimiropoulos. Xnor-net++: Improved binary neural networks. CoRR , abs/1909.13863, 2019.

Zihan Xu, Mingbao Lin, Jianzhuang Liu, Jie Chen, Ling Shao, Yue Gao, Yonghong Tian, and Rongrong Ji. Recu: Reviving the dead weights in binary neural networks. In Proceedings of the IEEE/CVF international conference on computer vision , pages 5198-5208, 2021.

Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng. Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm. In ECCV , 2018.

Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang-Ting Cheng. Reactnet: Towards precise binary neural network with generalized activation functions. In ECCV , 2020.

Yuzhang Shang, Dan Xu, Bin Duan, Ziliang Zong, Liqiang Nie, and Yan Yan. Lipschitz continuity retained binary neural network. In ECCV , 2022.

Yuzhang Shang, Dan Xu, Ziliang Zong, Liqiang Nie, and Yan Yan. Network binarization via contrastive learning. In ECCV , 2022.

Joseph Bethge, Christian Bartz, Haojin Yang, Ying Chen, and Christoph Meinel. Meliusnet: Can binary neural networks achieve mobilenet-level accuracy? arXiv preprint arXiv:2001.05936 , 2020.

Lei Ke, Martin Danelljan, Xia Li, Yu-Wing Tai, Chi-Keung Tang, and Fisher Yu. Mask transfiner for high-quality instance segmentation. In CVPR , 2022.

Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587 , 2017.

Yu Qiao, Yuhao Liu, Xin Yang, Dongsheng Zhou, Mingliang Xu, Qiang Zhang, and Xiaopeng Wei. Attention-guided hierarchical structure aggregation for image matting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13676-13685, 2020.

Ning Xu, Brian Price, Scott Cohen, and Thomas Huang. Deep image matting. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2970-2979, 2017.

Yixing Xu, Kai Han, Chang Xu, Yehui Tang, Chunjing Xu, and Yunhe Wang. Learning frequency domain approximation for binary neural networks. In NeurIPS , 2021.

Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432 , 2013.

Leo Grady, Thomas Schiwietz, Shmuel Aharon, and Rüdiger Westermann. Random walks for interactive alpha-matting. In Proceedings of VIIP , volume 2005, pages 423-429. Citeseer, 2005.

Quan Chen, Tiezheng Ge, Yanyu Xu, Zhiqiang Zhang, Xinxin Yang, and Kun Gai. Semantic human matting. In Proceedings of the 26th ACM international conference on Multimedia , pages 618-626, 2018.

Jinlin Liu, Yuan Yao, Wendi Hou, Miaomiao Cui, Xuansong Xie, Changshui Zhang, and Xiansheng Hua. Boosting semantic human matting with coarse annotations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8563-8572, 2020.

Qihang Yu, Jianming Zhang, He Zhang, Yilin Wang, Zhe Lin, Ning Xu, Yutong Bai, and Alan Yuille. Mask guided matting via progressive refinement network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1154-1163, 2021.

Zhanghan Ke, Kaican Li, Yurou Zhou, Qiuhua Wu, Xiangyu Mao, Qiong Yan, and Rynson WH Lau. Is a green screen really necessary for real-time portrait matting? arXiv preprint arXiv:2011.11961 , 2020.

Zeqi Gu, Wenqi Xian, Noah Snavely, and Abe Davis. Factormatte: Redefining video matting for re-composition tasks. arXiv preprint arXiv:2211.02145 , 2022.

Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence , 40(4):834-848, 2017.

Huikai Wu, Shuai Zheng, Junge Zhang, and Kaiqi Huang. Fast end-to-end trainable guided filter. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 1838-1847, 2018.

Hai Phan, Zechun Liu, Dang Huynh, Marios Savvides, Kwang-Ting Cheng, and Zhiqiang Shen. Binarizing mobilenet via evolution-based searching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13420-13429, 2020.

Kenneth E Iverson. A programming language. In Proceedings of the May 1-3, 1962, spring joint computer conference , pages 345-351, 1962.

Lei Ke, Henghui Ding, Martin Danelljan, Yu-Wing Tai, Chi-Keung Tang, and Fisher Yu. Video mask transfiner for high-quality video instance segmentation. In ECCV , 2022.

Haotong Qin, Mingyuan Zhang, Yifu Ding, Aoyu Li, Ziwei Liu, Fisher Yu, and Xianglong Liu. Bibench: Benchmarking and analyzing network binarization. In ICML , 2023.

Keyu Tian, Yi Jiang, Qishuai Diao, Chen Lin, Liwei Wang, and Zehuan Yuan. Designing bert for convolutional networks: Sparse and hierarchical masked modeling. arXiv preprint arXiv:2301.03580 , 2023.

Shanchuan Lin, Andrey Ryabtsev, Soumyadip Sengupta, Brian L Curless, Steven M Seitz, and Ira Kemelmacher-Shlizerman. Backgroundmattingv2, 05 2023.

Shanchuan Lin, Linjie Yang, Imran Saleemi, and Soumyadip Sengupta. Robustvideomatting, 05 2023.

Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF international conference on computer vision , pages 1314-1324, 2019.

Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR , 2018.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Supplementary Material: BiMatting: Efficient Video Matting via Binarization

6 Overview

We offer further information in this supplementary material. Section 7 elaborates on our network architecture, providing detailed insights. In Section 8, we present additional results for detailed comparison and also showcase visual examples of our composited matting data samples. We highlight that video results are included in our supplementary material. We encourage readers to refer to our provided video for extensive matting results comparisons.

7 Details of Network Architecture

For the encoder (Table 4) constructed by our proposed SBB, it operates on individual frames to extract feature maps at various spatial scales ranging from 1 2 to 1 16 . Within each SBB, the number of feature channels is either doubled or stays the same in the first sub-block, then doubled again in the second sub-block, and finally halved in the third sub-block. This design enables each SBB to carry out feature extraction in a channel dimension space that exceeds the input dimension. Meanwhile, the computation-dense structure guarantees the utilization of binarized convolutions to acquire highquality features. While traditional full-precision MobileNetV3 backbones operate at 1 32 scale, we made modifications to the last block. Specifically, we utilize convolutions with a dilation rate of 2 and a stride of 1, following the design principles from [1]. Furthermore, the final feature map ( 1 16 scale) is passed to the LR-ASPP module, which compresses it into 128 channels.

For the decoder (Table 5), as mentioned in our paper, the SAB is employed in every decoder block except the first one to accelerate computations. The binary mask used by the SABs is obtained from the first non-sparse binarized block, which has the smallest feature scale and acquires the mask at a minimal cost. This design significantly improves the efficiency of the decoder.

RVM-ReActNet (1-bit)

BiMatting (1-bit)

RVM (32-bit)

RVM-DoReFa (1-bit)

Input (Original)

RVM-BNN (1-bit)

For other parts, the Deep Guided Filter (DGF) incorporates a limited number of binarized 1 × 1 convolutions internally. For more detailed specifications, please refer to [40, 1]. The complete network is constructed and trained using PyTorch [50].

8 Additional Visualizations

8.1 Visual Results

We show more visual results in Fig. 6, where we can more clearly see the advantages of our BiMatting over other binarization methods, both in edge details and local region matting. At the same time, we also provide a video (BiMatting.mp4 file in the supplementary material) to show the advantages of our BiMatting in more detail.

8.2 Composited Datasets

We follow [1] as a guide to constructing composite training and test samples. We show some examples of composited training samples from the matting datasets in Fig. 7. The clips contain natural movements when compositing with videos as well as artificial movements generated by the motion augmentation. Motion augmentation was exclusively applied to the foreground and background of the image in the testing samples (Fig. 8). The motion augmentation solely involved affine transforms. Moreover, the strength of the augmentation was deliberately toned down in comparison to the training augmentation, ensuring that testing samples possess a high-degree realism.

Composited Frames

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose the first binarized video matting network, namely BiMatting. They first analyze the bottlenecks of the direct binarization of video matting models and propose an accurate and efficient binarization method. Compared with other full-precision neural networks and other binarization methods, the authors confirm the effectiveness and great potential of BiMatting. It is worth noting this paper constructs a binary backbone that achieves similar acceleration compared to binarized mobilenetv3, a super-lightweight architecture, for the first time and realizes the practical application on video matting. The proposal and application of this binary network mean that the practicality of binarization has been significantly improved in general.\n\nSTRENGTHS:\nThe authors propose the first binary neural network for video matting tasks for reducing computation consumption significantly while retaining practical accuracy. Since video matting tasks usually run on resource-constrained devices, this research is practically significant.\n\nIn terms of method, the authors successfully design an ultra-lightweight video matting network by binarization and make it have a significantly improved accuracy rate. In general, the authors' method tightly combines video matting tasks and architectures.\n\n1. The binary backbone with SBB is interesting, and I think it is the major contribution of this paper. Compared with the existing binarization schemes, the authors create a backbone network with reasonable accuracy and comparable efficiency with the binarization mobilenetv3 through the careful design of the architecture. This is very important for the practical application of binary neural networks.\n\nMoreover, the authors’ motivation (or paradigm) for designing the binary backbone network has a general contribution to the binarization community, that is, \"the crucial paradigm of an accurate binarized encoder is the computation-dense form of binarized block\", this motivation can explain the success of binary networks such as Bi-Real and IR-Net, and may lead to more lightweight binary networks with higher accuracy.\n\n2. Using representations of different scales, SAB successfully uses sparse masks to further reduce the computational load of the binary decoder without significant performance degradation. This demonstrates that the representational ability of the binarized computational unit allows the model to exploit less informative representations.\n\nThe experimental and video results are solid. The accuracy of BiMatting not only far exceeds the binarized version of the existing model but also exceeds the full-precision matting model such as BGMv2 while achieving significant storage savings. The visualization results also show that BiMatting has improved significantly in detail.\n\nWEAKNESSES:\n1. For the backbone with SBB, the authors should provide more detailed information. \nFirst of all, I suggest the authors give a more detailed ablation about the efficiency, including the FLOPs and the number of parameters of direct full-precision/binarized mobilenetv3 and SBB backbone. \nIn addition, since the authors add the pre-training process on the ImageNet dataset as stage 0, they should also provide the pre-training ablation results on this dataset, which has clarified the performance improvement of the binarized backbone alone.\n\n2. For SAB, a strange phenomenon is that the results in Table 1 show that using SAB in the direct binarized matting model will cause a crash. Although SAB does not directly target the accuracy bottleneck, it even causes the model to be worse than direct binarization. Can the authors explain this phenomenon?\n\n3. In Table 2, ReCU and BNN present the same collapse, however, as reported in their original papers, the former usually performs significantly better than the latter on ImageNet, even better than methods such as DoReFa-Net. I suggest the authors discuss this phenomenon and report the ImageNet pre-training accuracy of these binary beckbones.\n\n4. I suggest the authors discuss the possibility of BiMatting's actual deployment, including how to deploy it on edge devices and possible hardware inference performance.\n\nQUESTIONS:\nPlease see weaknesses\n\n--After rebuttal--\nThanks for the detailed response, which has well addressed my concerns. I also read other reviewers' comments and the authors' responses. I am satisfied with the rebuttal and increase the rating score.\n\nLIMITATIONS:\nThe authors discuss the limitations of their works in section 4.3.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper propose an efficient solution that utilizes binarization to achieve real-time video matting on for devices constrained by computational resources. The proposed BiMatting constructs shrinkable and dense topologies of the binarized encoder block to enhance the extracted representation, while sparsifying the binarized units to reduce the low-information decoding computation. Extensive experiments shows that it outperforms SOTA binarized video matting methods by a large margin.\n\nSTRENGTHS:\nThis work claimed itself as the first binarization solution for video matting tasks, which may provide a new effective solution to the real-time matting community.\nThe work is based on reasonable analysis and observation that shown in the Section 3.1. The proposed method also effectively address the pointed challenge. Moreover, although the performance is lower than the full-precision models, the proposed method achieves satisfactory results compared with the binarized video matting models.\n\nWEAKNESSES:\n1. The authors need to check the equations and make sure that all the notations are explained. For example, the ⊗ in Eq. (2) is not mentioned in the passage. The |W| is also remained unexplained.\n\n2. Some key ideas are not fully verified by experiments. For examples, as the author claims that dense connection is very important to recover the performance, this assertion is somewhat unwarranted without the comparison of the proposed SBB and SBB w/o dense skip connections for each BiConv. \n\n3. Since the Matting training phase has 4 stages, and each stage has its own independent training settings, the reviewer doubted that whether the Matting training phase can work well without extensive ablation study of different training settings. It would be better to provide experiments to demonstrate the effectiveness and robustness of the proposed method under different training settings. Otherwise, the robustness of the proposed method is hindered.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new video matting method called BiMatting. It is based on Binary neural networks (BNNs), a more compact network to reduce the computational and storage requirements of video matting. Specifically, the authors addressed the accuracy bottleneck of BNNs by re-designing its encoder and decoder architecture. Performance was evaluated on several video benchmarks and compared with SOTA methods.\n\nSTRENGTHS:\nThis is the first time BNNs was used for video matting application. This problem was well-motivated and several challenges were addressed by careful architecture design and new training procedures. The authors provided a comprehensive comparison with other SOTA methods and demonstrated that this method offers a good tradeoff between performance and storage. It is computationally efficient that BiMatting reduces computational FLOPs by 11 times and storage by 21 times.\n\nWEAKNESSES:\n1. Although this method outperforms existing binarized video matting models, it is not yet on par with its full-precision counterpart in visual quality.\n2. Another potential weakness is the complexity of the training pipeline of Bi-Matting, which contains a pre-training phrase and a matting training phrase with four stages.\n\nQUESTIONS:\n1. The authors claimed that binarizing the encoder causes a significant drop in performance. Why is that the case? Do the authors mean binarizing the activation of the encoder?  \n2. Is there a reason why the paper does not have a related work section? I would be curious to see recent works and other applications of BNNs to image segmentation and other relevant fields. It would help me evaluate the novelty and impact of current methods.\n\nLIMITATIONS:\nThe author addressed the limitation that the method is not yet on par with 32-bit RVM. Quality-wise, it tends to get blurred and prefers simpler backgrounds.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors analyzed the operations inside the deep video matting networks and proposed an efficient binarization method to greatly reduce the computation cost. Specifically, they re-designed the encoder and also sparsified the decoding process. The proposed methods are shown to outperform the existing baselines and achieves reasonable visual quality.\n\nSTRENGTHS:\n- Section 3 shows binarization of the encoder parts bring the most harmful degradation of the accuracy, and the current decoder consumes the most computational resources. The analysis and preliminary experiments demonstrate the reason of the proposed methods. The logic and story-telling of the paper make a lot of sense. \n- The review likes the reasoning of the encoder (lacking of short connection within the blocks) and decoder (many redundancy in computation), and the connections between the analysis and the proposed designs. \n- The design of SAB makes sense for any video applications.\n\nWEAKNESSES:\n- The training pipeline is too complicated. \n- Table 1 is confused. So for BiMatting (Ours), is it the same as RVM (SBB+SAB)? It's better to state the difference of RVM and BiMatting, and show it clearly in the table. Why RVM + SAB has much worse results than the one without any of SBB and SAB? Is it possible the bad performance is due to the bad training or need a better parameter set different from the proposed one?\n\nQUESTIONS:\n- In section 3.2.2, ep(5), the authors proposed to use mean to compute the values for 'short connection'. How did the authors come up with that solution? Why not using another convolution layers to reduce the channel number or taking the addition? Has the authors done some comparison?\n- Any plans to extend the work to other video applications?\n\nLIMITATIONS:\n- The SAB designs may fail when the video is dynamic or having more foreground object movement. \n- Complicated training procedures and vague stopping conditions for each stage. It makes the results in the paper hard to be reproduced.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}