Real-Time High-Resolution
Background Matting
2022/06/08
M s toshi T teno
@S to L b M1

aaaaa1

Background Replacement
Movie

Video Conference

https://www.famefocus.com/video/amazing-before-afterhollywood-vfx-fantastic-beasts-and-where-to- nd-them/fi

https://www.studiobinder.com/blog/what-is-a-green-screen-video/Ref. [1] S.Lin et al.

Ref. [1] S.Lin et al.

2

Background Matting
Basic Process＝✕
Source image

Alpha matte

Foreground image

Creating Alpha matte = Alpha Matting3Matting with a known backgroundSource image

Encoder

Decoder

Alpha matteBackground image

4

Related Work
Trimap-based matting

Source image

EncoderDecoder

Alpha matte❌ Manual annotation is needed
Trimap (manual annotation)❌ Performance depends on the quality of Trimap
5

Related Work
Matting w/o any external input

Model that
separate fgr&bgr
Source image

Alpha matte❌ often fail to generalize
6

ApproachGiven

I : imageB : background image

composited image I′ over new background B′Predict

I′ = αF + (1 − α)B′

α : alpha matte⚠ Actually, not predict foreground directly
But, predict foreground residual F R( = F − I)
F can be recovered byR

F = max(min(F + I),0)￼￼￼￼

F : foreground✅ Improves learning✅ Allows us to apply low-res F R on tohigh-res I

7

Architecture
Coarse Alpha matte

Re ned Alpha matte
Source image
Encoder

DecoderCoarse Foreground image(Foreground residual)

Base Network

Re nement Network

Background image

Foreground image(Foreground residual)

fifi

Error map
fRe iner

8

Architecture
Ref. [1] S.Lin et al.

9Base Network

Ref. [1] S.Lin et al.

Backbone : ResNet50 (ResNet101, MobileNetV2)
ASPP : Atrous Spatial Pyramid Pooling module from DeepLabV3
Decoder : skip connection, bilinear upsampling, 3*3 convolution, BN, ReLU

10

Re inement Network
Ref. [1] S.Lin et al.

1.

Choose k patches to re ne using Error Map

2. Re ne patches by upsampling, 3*3 convolution, BN, ReLU
fifi

ffi

3. Swap in the respective patches that have been re ned

11Datasets
Introducing 2 new datasets

VideoMatte240K• 484 high-res videos• total of 40,709 unique frames of alpha mattes andforegrounds• 384 videos are 4K, 100 are in HD

PhotoMatte13K/85• collection of 13,665 images• averaging resolution is around 2000 * 2500
• For privacy and licensing issue, only 85 mattes ofsimilar quality is publicly released
12Training
Augmentation
A ine Trans

Horizontal
Flipping

Brightness

Hue&Saturation

Blurring&Sharpening

RandomNoiseForeground

◯

◯

◯

◯

◯

◯

Background

◯

◯

◯

◯

◯

◯

Additional
Noise & Jitter& A ine Trans

Shadow
E ect◯◯ffffff13

Training
Lossℒbase = ℒα + ℒFc + ℒEcℒrefine = ℒα + ℒF
ℒα = ∥α − α*∥1 + ∥∇α − ∇α*∥1 ( α* : ground-truth.

L1 loss over the whole alpha matte and its Sobel gradient)ℒF = ∥(α* > 0) * (F − F*)∥1 (L1 loss only on the foreground pixels where α* > 0 )ℒE = ∥E − E*∥2, E* = | α − α* | (mean squared of error map)
14

Training
Training order1.

Train only the base network with VideoMatte240K

2. Train entire model jointly on VideoMatte240K
3. Train entire model jointly on PhotoMatte13K
4. Train entire model jointly on Distinctions-646 (dataset form Ref.[4] Y. Qiao)

I skipped 3,4 because of datasets’ unavailability

15

Implementation
I implemented :• Background Matting architecture from scratch with PyTorch• Code for training (trained 1 epoch)• training base network• training re nement network• Code for testing image background matting

I referred to :• O cial Implementation with PyTorch• ResNet Implementation from scratch• Pretrained weights of ASPP module from DeepLabV3fiffi

16Quality Analysis

17Speed Analysis
Image matting time : 0.0388 sec/image ≃ 25.78 fps• GPU : NVIDIA Tesla V100 32GB

c.f. original speed measurement

Ref. [1] S.Lin et al.

• Image : 3840 * 2160 (4K)
• Average over 35 images

18

Pull Requests• Code for test video background matting• stock video footage• webcam• Revise test_image.py with tricks to speed up inference time
• Create additional Backbone Network options (ResNet101, MobileNetV2)• Homographic Alignment (see original implementation)

19

Reference[1] S. Lin, A. Ryabtsev, S. Sengupta, B. Curless, S. Seitz, I. Kemelmacher-Shlizerman, “Real-time hithresolution background matting,” CVPR, pp.8762-8771, 2021
[2] L.C. Chen, G. Papandreou, F. Schro , H. Adam, “Rethinking atrous convolution for semantic imagesegmentation,” arxiv:1706.05587, 2017[3] L.C. Chen, G. Papandreou, I. Kokkinos, k. Murphy, A.L. Yuille, “Deeplab: Semantic imagesegmentation with deep convolutional nets, atrous convolution, and fully connected crfs,” IEEE
transactions on pattern analysis and machine intelligtnece, vol.40, no.4, pp834-848, Apr. 2018[4] Y. Qiao, Y. Liu, X. Yang, D. Zhou, M. Xu, Q. Zhang, X. Wei, “Attention-guided hierarchical structure
aggregation for image matting,” CVPR, pp.13676-13685, 2020

ff20

Appendix
ASPP : Atrous Spatial Pyramid Pooling
ASPP module consists of multiple dilated convolution lterswith di erent dilation rates.
“it is e ective to resample features at di erent
scales for accurately and e ciently classifying
regions of an arbitrary scale.”Ref.[2] L.C.Chen

fi

ff

ffi

ff

ff

Ref.[3] L.C.Chen

21