SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Haotong Qin\*1,2, Lei Ke\*2,3, Xudong Ma1, Martin Danelljan2, Yu-Wing Tai4, Chi-Keung Tang3, Xianglong Liu\*1, Fisher Yu2
$^{1}$ Beihang University  $^{2}$ ETH Zürich  $^{3}$ HKUST  $^{4}$ Dartmouth College
Real-time video matting on edge devices faces significant computational resource constraints, limiting the widespread use of video matting in applications such as online conferences and short-form video production. Binarization is a powerful compression approach that greatly reduces computation and memory consumption by using 1-bit parameters and bitwise operations. However, binarization of the video matting model is not a straightforward process, and our empirical analysis has revealed two primary bottlenecks: severe representation degradation of the encoder and massive redundant computations of the decoder. To address these issues, we propose BiMatting, an accurate and efficient video matting model using binarization. Specifically, we construct shrinkable and dense topologies of the binarized encoder block to enhance the extracted representation. We sparsify the binarized units to reduce the low-information decoding computation. Through extensive experiments, we demonstrate that BiMatting outperforms other binarized video matting models, including state-of-the-art (SOTA) binarization methods, by a significant margin. Our approach even performs comparably to the full-precision counterpart in visual quality. Furthermore, BiMatting achieves remarkable savings of  $12.4 \times$  and  $21.6 \times$  in computation and storage, respectively, showcasing its potential and advantages in real-world resource-constrained scenarios. Our code and models are released at https://github.com/htqin/BiMatting.
The success of deep neural networks has led to remarkable advancements in computer vision tasks, including video matting [1, 2, 3, 4, 5, 6, 1, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]. However, many practical applications based on deep networks require real-time processing with minimal latency, which is challenging due to the high computational and storage demands. To address this issue, researchers have developed lightweight video matting algorithms such as Robust Video Matting (RVM) [1] and BGMv2 [10]. While these methods achieve significant speedups and memory reductions, they still rely on expensive floating-point operations, leaving room for further compression.
One of the most promising approaches to improving the efficiency of neural networks is network binarization [17, 18, 19, 20, 21, 22, 23, 24, 25, 26]. Binary neural networks (BNNs) have emerged as a highly effective technique for optimizing
neural networks by reducing parameter bit width to 1-bit. BNNs leverage compact binary parameters that occupy less memory space and use efficient bitwise operations, which are much less computationally expensive than floating-point operations. By employing the binarization approach, researchers can significantly reduce the computational and storage demands of video matting applications.
Figure 1: BiMatting enjoys impressive computation/storage savings while surpassing SOTA 1-bit and some 32-bit models in accuracy.
* equal contribution corresponding author
Figure 2: BiMatting overview. We apply Shrinkable Binarized Block (SBB) for the accurate encoder and Sparse-Assisted Binarization (SAB) for the efficient decoder. Arrows  $\rightarrow$  indicate the flow of features,  $\oplus$ ,  $\oslash$ , and avg indicate concatenation, splitting, and averaging of features, respectively.
Despite that the generic binarization methods have been studied extensively by the binarization community, the direct binarization for existing video matting models still leaves severe accuracy and efficiency bottlenecks. First, to construct an efficient architecture, many video matting methods contain encoders with lightweight full-precision backbones to obtain intermediate features. For example, MobileNet architectures are widely applied as lightweight extractors in several video matting models to provide strong features [1, 11]. However, this practice introduces an accuracy bottleneck in the binarized video matting model. Our analysis shows that the computing units and topology of existing lightweight encoder architectures are unfriendly to binarization, leading to severe degradation in the quality of intermediate features extracted by its binarized counterpart. Second, we identify that the architecture of the existing video matting decoder [1] causes immense redundant computation at the multi-scale decode stage. Intensive computation is still needed in large portions of certain foreground or background regions (spatial regions outside unknown parts of the trimap) of the high-resolution feature maps. This weakens the savings brought by binarization, which hinders efficiency improvements in the context of extremely compressed bit-width.
In this paper, we provide empirical studies of the above-mentioned bottlenecks. This leads us to propose BiMatting, a Binarized model for accurate and efficient video Matting (overview in Fig. 2). To tackle the accuracy bottleneck, we first investigate the limitations of existing binarized encoders in representation extraction. Then, we propose Shrinkable Binarized Block (SBB), which follows a binarization-friendly computation-dense paradigm to construct a flexible block structure. SBB effectively extracts high-quality features with improved topology and operators. In addition, given a reliable binarized encoder, we further develop Sparse-Assisted Binarization (SAB) to effectively reduce the computational consumption of the binarized decoder. The overall burden is thus greatly reduced. SAB removes repeated computation by spatial sparseness [27] while preserving the accuracy of binarized units, resulting in a notable computation reduction without compromising performance.
Our BiMatting is the first binarization solution for video matting tasks, which surpasses 1-bit matting models using existing binarization algorithms by a significant margin [21, 17, 23, 18]. Notably, BiMatting even outperforms some full-precision models [28, 11] in terms of accuracy while being highly efficient. Our extensive experiments on fundamental tasks across VideoMatte240K (VM) [10], Distinctions-646 (D646) [29], and Adobe Image Matting (AIM) [30] datasets demonstrate that the advantages of BiMatting are task-independent. In addition, our SBB and SAB components are highly efficient, allowing BiMatting to achieve  $12.4 \times$  FLOPs and  $21.6 \times$  storage savings compared to the full-precision counterpart, leading a promising way for the video matting on edge scenarios (Fig. 1).
2 A Baseline for Binarized Video Matting
In this section, we first build a baseline to study the binarized video matting model. The baseline is based on straightforward binarization and an existing lightweight video matting architecture.
2.1 Binarization Framework
Binarization applies the sign function to compress weights and activations to 1-bit for computing units of the binarized network [17, 19, 20, 22, 31], and the propagation process can be expressed as
$$
\operatorname {s i g n} (\boldsymbol {x}) = \left\{ \begin{array}{l l} + 1, & \text {i f} \boldsymbol {x} \geq 0 \\ - 1, & \text {o t h e r w i s e} \end{array} , \quad \frac {\partial \mathcal {L}}{\partial \boldsymbol {x}} = \left\{ \begin{array}{l l} \frac {\partial \mathcal {L}}{\partial \operatorname {s i g n} (\boldsymbol {x})}, & \text {i f} x \in (- 1, 1) \\ 0, & \text {o t h e r w i s e} \end{array} , \right. \right. \tag {1}
$$
(a) Accuracy perspective
(b) Efficiency perspective
Figure 3: Analysis of bottlenecks for the binarized video matting baseline respectively in accuracy and efficiency on the VM dataset. We report (a) the MAD metrics of binarizing each component, and (b) the FLOPs and storage consumption of each component in the binarized baseline.
where  $\mathcal{L}$  as the loss function. To build a strong binarized baseline by existing techniques, we apply floating-point scaling factors for weights  $W$  [19, 23] and learnable thresholds for activations  $A$  [23]:
$$
\boldsymbol {B} _ {\boldsymbol {w}} = \operatorname {s i g n} (\boldsymbol {W}), \quad \boldsymbol {B} _ {\boldsymbol {a}} = \operatorname {s i g n} (\boldsymbol {A} - \tau), \quad \boldsymbol {o} = \alpha \boldsymbol {B} _ {\boldsymbol {w}} \otimes \boldsymbol {B} _ {\boldsymbol {a}}, \tag {2}
$$
where  $B_{w}$  and  $B_{a}$  denote the binarized weight and activation, respectively. The channel-wise  $\alpha$  is obtained by mean  $(|W|)$ , and the layerwise threshold  $\tau$  is initialized as 0. Note that the Straight-Through Estimator (STE) [32] is uniformly applied to all experiments and binarization methods, approximating the gradient of the sign function as clip function (as the backward in Eq. (1)), since most other estimators consume unaffordable GPU memory [31, 22]. Binarization is applied to computation layers except for those who access original inputs or produce final outputs.
2.2 Video Matting Architecture
Matting model aims to break down a frame  $I$  into a foreground  $F$  and a background  $B$ , using an  $\alpha$  coefficient to represent the linear combination of the two [1, 7],
$$
I = \alpha F + (1 - \alpha) B. \tag {3}
$$
Compared to image matting [2, 3, 4, 5, 6, 33, 34, 35, 36], matting methods specifically developed for video sequence should be more effective in utilizing spatial-temporal information in videos [1, 7, 8, 9, 10, 11, 37, 38]. Among them, the recent Robust Video Matting (RVM) [1] method achieves the SOTA accuracy of the video matting task and stands out significantly in terms of efficiency. The architecture of RVM mainly comprises a lightweight MobileNetV3-based encoder (including backbone and Atrous Spatial Pyramid Pooling (ASPP) [39] module), a recurrent decoder, and a deep guided filter (DGF) [40] module. The concise architecture brings efficiency, even realizing real-time matting at high resolution. While RVM is one of the lightest video matting models available, there is still significant potential for compression in terms of reducing bit width, given the high cost of both the floating-point parameters and computations involved.
3 The Rise of BiMatting
3.1 Bottlenecks of Binarized Video Matting Baseline
Our goal is to achieve practical and resource-efficient video matting via binarization. However, the encoder and decoder of the binarized baseline pose accuracy and efficiency bottlenecks, respectively.
From an accuracy perspective, Fig. 3 (a) compares the accuracy drops on the VideoMatting (VM) dataset by binarizing every single part in the RVM model. Specifically, we find that binarizing the existing lightweight MobileNetV3 backbone in the encoder causes the most significant drop in accuracy among all parts, with a drop almost equivalent to directly binarizing the entire network (Bi-All 28.49 vs. Bi-Backbone 23.56 for MAD metric). In contrast, binarizing the ASPP, decoder, and DGF parts produce less harm to accuracy (less than a 3.05 MAD increase). Thus, improving the backbone of the encoder to make it more amenable to binarization is of the highest priority in addressing the accuracy drop in the binarized video matting model.
From the efficiency perspective, we show in Fig. 3 (b) the computation and storage usage for each part of the binarized video matting model to demonstrate the individual consumption in efficiency. Our analysis from an efficiency perspective reveals that the decoder consumes a significant amount
of computational resources, accounting for  $71.8\%$  (0.38G) FLOPs with only  $12.1\%$  parameters after binarization. Conversely, the binarized encoder has a higher parameter count  $(81.8\%)$ , but its FLOP consumption is only  $16.6\%$ . Negligible consumption is observed for the other parts. Based on these observations, it is evident that the computational redundancy of the decoder significantly impacts the acceleration performance of the overall model. Thus, there is ample room for computational optimization after direct binarization to further enhance the model's efficiency.
Given the observations from the aforementioned experiments, we find that the current baseline presents two major challenges: 1) the encoder's lightweight backbone is unsuitable for binarization and fails to generate practical features, and 2) the decoder continues to exhibit considerable computational redundancy even after binarization. Thus, we propose a binarization-friendly encoder and a computationally-efficient decoder to construct an accurate and efficient binarized matting model.
3.2 Shrinkable Binarized Block
3.2.1 Binarization-evoked Encoder Degradation
The binarized video matting baseline in Sec. 2 employs a lightweight MobileNetV3 as the encoder's backbone and directly binarizes its convolutional and linear layers. However, the extensive use of groupwise and pointwise convolutions presents significant challenges to binarization [41, 23], resulting in severe accuracy degradation. Fig. 3 shows that directly using binarized MobileNetV3 leads to over  $4 \times$  MAD metric degradation. On the other side, existing binarized networks are also far from the efficiency level of binarized MobileNetV3, making it hard to be transferred straightforwardly to construct the binarized matting model. For example, some binarized MobileNet-based models suffer at least  $3 - 8 \times$  FLOPs than binarized MobileNetV3 [23, 41, 26]. Therefore, a new binarized backbone is necessary for feature extraction to achieve high-quality features for the decoder while keeping the model ultra-lightweight.
We first analyze the degradation of the binarized encoder in existing baselines. We note the binarized convolution as  $\mathrm{BiConv}_n(\cdot)$  or  $\mathrm{GBiConv}_n(\cdot)$ , where  $n$  is the kernel size of the convolution and G mark denotes groupwise. Their superscripts  ${}^{\mathrm{up}}$ ,  ${}^{\mathrm{eq}}$ , and  ${}^{\mathrm{dn}}$  indicate the number of their output channels is greater than, equal to, or less than that of their input channels, respectively. Then the binarized MobileNetV3 block in the baseline's encoder can be expressed as
$$
\begin{array}{l} M B V 3 \text {B l o c k (1)}: \quad \boldsymbol {o} = \operatorname {B i C o n v} _ {1} ^ {\mathrm {e q}} (\operatorname {G B i C o n v} _ {n} ^ {\mathrm {e q}} (\operatorname {B i C o n v} _ {1} ^ {\mathrm {e q}} (\boldsymbol {x}))) + \boldsymbol {x}, \quad s. t. c ^ {\boldsymbol {x}} = c ^ {\boldsymbol {o}} \\ M B V 3 \text {B l o c k (2)}: \quad \operatorname {B i C o n v} _ {n} ^ {\mathrm {d p}} (\operatorname {G B i C o n v} _ {n} ^ {\mathrm {e q}} (\operatorname {B i C o n v} _ {n} ^ {\mathrm {d p}} (\boldsymbol {x}))) + [ \boldsymbol {x}, \boldsymbol {o} ] \end{array} \tag {4}
$$
$$
M B V 3 \text {B l o c k} (2): \quad \boldsymbol {o} = \operatorname {B i C o n v} _ {1} ^ {\mathrm {d n}} \left(\operatorname {G B i C o n v} _ {n} ^ {\mathrm {e q}} \left(\operatorname {B i C o n v} _ {1} ^ {\mathrm {u p}} (\boldsymbol {x})\right)\right) + \left[ c ^ {\boldsymbol {x}} = c ^ {\boldsymbol {o}} \right] \boldsymbol {x},
$$
where  $\pmb{x}$  and  $\pmb{o}$  denote the input and output, respectively, kernel size  $n \in \{3, 5\}$ , and  $[\cdot]$  denotes the Iverson bracket [42], evaluating to 1 if the condition inside the parentheses is true, and 0 otherwise. The batch normalization and the activation layers following convolutions are omitted.
Eq. (4) suggests at least two issues that impede accurate binarization. Firstly, all convolutions are groupwise or pointwise that have fewer parameters than regular convolutions, thus are sensitive to binarization and hard to recover from crashes caused by mutual influence. Secondly, utilizing only per-block short connection is insufficient since convolution-specific shortcuts are critical to achieving performance recovery [22, 23]. Several existing binarization methods aim to address the aforementioned issues, e.g., applying regular convolutions instead of grouped ones [23, 26, 41], or creating shortcuts for channel-constant convolutions [23, 41]. Though these techniques may be effective, they are significantly more computational-expensive than directly binarized MobileNetV3 and still suffer representation loss. Therefore, it is necessary to develop stronger feature extraction encoders for binarized video matting models to address these limitations.
3.2.2 Shrinkable Binarized Block for Accurate Encoder
To mitigate the degradation of the encoder caused by binarization while keeping it lightweight, we present an efficient Shrinkable Binarized Block (SBB) for building a binarized video matting encoder, which is conducive to a flexible and lightweight architecture via channel-shrinkable design while retaining the representations and gradients.
Based on the analysis in Sec. 3.2.1, we determine that the crucial paradigm of an accurate binarized encoder is the computation-dense form of binarized block. In terms of topology, we ensure that every binarized convolution has a corresponding/dense connection to preserve the representation
accurately. In terms of the operator, introducing computation-dense regular convolutions can prevent groupwise and pointwise ones from becoming exclusive. Following this paradigm, SBB recovers the representation in the binarized block such that enables the backbone to effective feature extraction in flexible dimensional space.
Specifically, we first introduce the sub-blocks of SBB. Based on the above analysis, we find that the key element to constructing the flexible binarized architecture is to allow various sub-blocks in the feature extractor to freely adjust (increase, maintain, or shrink) the channel number of the outputs, while the shortcut should accompany with each binarized convolution to maintain representation. Therefore, the channel-shrunk SBB sub-block can be first constructed as follows:
$$
\begin{array}{l} \text {S u b - S B B} (1) \quad \theta^ {\mathrm {d n}} (\boldsymbol {x}): \quad \boldsymbol {o} = \operatorname {B i C o n v} _ {1} ^ {\mathrm {d n}} \left(\boldsymbol {x} ^ {\prime}\right) + \operatorname {m e a n} \left(\boldsymbol {x} ^ {\prime \left(1, \frac {1}{2} c ^ {\infty}\right)}, \boldsymbol {x} ^ {\prime \left(\frac {1}{2} c ^ {\infty}, c ^ {\infty}\right)}\right), \tag {5} \\ \boldsymbol {x} ^ {\prime} = \operatorname {B i C o n v} _ {3} ^ {\mathrm {e q}} (\boldsymbol {x}) + \boldsymbol {x}, \quad s. t. c ^ {\boldsymbol {x}} = 2 c ^ {\boldsymbol {o}}, \\ \end{array}
$$
where  $\pmb{x}^{\prime(m,n)}$  denotes taking the  $m$  to  $n$  channels of  $\pmb{x}^{\prime}$ . In the sub-block, we introduce shortcuts for channel-shrunk binarized convolution with channel splitting and averaging operations, allowing the channel to shrink while constructing a shortcut with negligible overhead. This design enables the binarized encoder to leverage computing units with higher output channels (such as 32 or 64) for extracting low-channel representations (16 or 32, correspondingly), resulting in dependable features for the decoder. Inspired by [23, 41], the channel-maintain and increased sub-blocks are as follows:
$$
\begin{array}{l} \text {S u b - S B B} (2) \quad \theta^ {\mathrm {e q}} (\boldsymbol {x}): \quad \boldsymbol {o} = \operatorname {B i C o n v} _ {1} ^ {\mathrm {e q}} (\boldsymbol {x} ^ {\prime}) + \boldsymbol {x} ^ {\prime}, \quad \boldsymbol {x} ^ {\prime} = \operatorname {B i C o n v} _ {3} ^ {\mathrm {e q}} (\boldsymbol {x}) + \boldsymbol {x}, \quad s. t. c ^ {\boldsymbol {x}} = c ^ {\boldsymbol {o}} \\ \text {S u b - S B B} (3) \quad \theta^ {\mathrm {u p}} (\boldsymbol {x}): \quad \boldsymbol {o} = \left(\left(\operatorname {B i C o n v} _ {1} ^ {\mathrm {e q}} _ {1} \left(\boldsymbol {x} ^ {\prime}\right) + \boldsymbol {x} ^ {\prime}\right) \oplus \left(\operatorname {B i C o n v} _ {1} ^ {\mathrm {e q}} _ {2} \left(\boldsymbol {x} ^ {\prime}\right) + \boldsymbol {x} ^ {\prime}\right)\right), \tag {6} \\ \boldsymbol {x} ^ {\prime} = \operatorname {B i C o n v} _ {3} ^ {\mathrm {e q}} (\boldsymbol {x}) + \boldsymbol {x}, \quad s. t. c ^ {\boldsymbol {x}} = c ^ {\boldsymbol {o}} / 2 \\ \end{array}
$$
where  $\oplus$  denotes the concatenate operation. Benefiting from the above sub-block variants, SBB can implement a flexible feature extraction architecture.
Next, we create SBB by assembling these sub-blocks. Each SBB is made up of three sub-blocks: the head, middle, and tail. The head and tail sub-blocks produce identical output channels, while the middle sub-block extracts feature in higher dimensions with double output channels:
$$
\boldsymbol {S B B}: \quad \boldsymbol {o} = \theta^ {\mathrm {d n}} \cdot \theta^ {\mathrm {u p}} \left(\boldsymbol {x} ^ {\prime}\right) + \boldsymbol {x} ^ {\prime}, \quad \boldsymbol {x} ^ {\prime} = \theta^ {\mathrm {e q}} (\boldsymbol {x}) \left[ c ^ {\boldsymbol {x}} = c ^ {\boldsymbol {o}} \right] + \theta^ {\mathrm {u p}} (\boldsymbol {x}) \left[ c ^ {\boldsymbol {x}} = \frac {1}{2} c ^ {\boldsymbol {o}} \right]. \tag {7}
$$
where  $x'$  is the output of the head sub-block that varies  $\theta^{\mathrm{eq}}$  or  $\theta^{\mathrm{up}}$  depending on the channel constraint in Iverson brackets. The binarized sub-blocks in the middle and at the end of the sequence respectively increase and decrease the feature dimension for thorough extraction, while also incorporating shortcut across sub-blocks that allows the representations to remain intact. Using the block-level crossing shortcut also mitigates the influence of the splitting and averaging procedure on the representations in the trail channel-contracted sub-block.
3.3 Sparse-Assisted Binarization
3.3.1 Computational Decoder Redundancy
In Sec. 3.1, we discover that while binarizing the decoder prevents accuracy performance from deteriorating, it results in significant computational costs (greater than  $71.8\%$ ) despite having a relatively small parameter size (less than  $12.1\%$ ). This inefficiency creates a bottleneck for the binarized video matting model. This is attributed to the decoder where high-resolution features are computationally demanding. To illustrate, consider the decoder's output block, which uses two  $3 \times 3$  convolutions to perform computations on the original scale's features. Since grouped convolutions that are unfriendly to binarization are not utilized here, the accuracy reduction of decoder binarization is not as severe as it is on the encoder. But the computation of this single block in the decoder (the last one in 5 decoder blocks) is even equivalent to  $103\%$  of the entire encoder in a binarized baseline.
However, it is worth noting that not all spatial computations are equally crucial. In the case of video matting, the majority of frames have large, uninterrupted foreground and background areas, which can usually be identified at a lower resolution. Nevertheless, the current decoder architecture repeats these computations at various scales, leading to significant computational overhead, particularly on high-resolution feature maps. This issue renders constructing an efficient decoder challenging, even
with the most aggressive 1-bit binarization approach. As a result, a fundamental redesign of the computation unit is required to create an efficient decoder that minimizes redundant computation while maintaining accurate matting results. Such a revamp should consider the context of binarization and focus on streamlining the computation process while preserving the quality of the output.
3.3.2 Sparse-Assisted Binarization for Efficient Decoder
To address the computational redundancy problem in the decoder of the binarized video matting model, we propose the Sparse-Assisted Binarization (SAB) method in BiMatting.