Deep Image Matting韩坤洋1. Deep Image Matting

CVPR 2017

2. Natural Image Matting via Guided Contextual Attention AAAI 2020
3. Context-Aware Image Matting for Simultaneous Foreground andAlpha Estimation
ICCV 2019

4. F, B, Alpha Matting Arxiv
5. Background Matting: The World is Your Green Screen
6. Real-Time High-Resolution Background Matting Arxiv

CVPR 2020

Deep Image Matting
University of Illinois at Urbana-Champaign
Adobe Research
CVPR 2017

Deep Image Matting• Current methods are designed to solve the matting equation• Very small dataset• 27 training images and 8 test images

New matting dataset• 1) Find images on simple or plain backgrounds, create alpha matte• 2) Randomly sample N background images in MS COCO and Pascal
VOC• Training set,
• 493 unique foreground objects and 49,300 images• Testing set,
• 50 unique objects and 1000 images• Trimap• randomly dilated

Matting encoder-decoder stage
VGG-16

Matting encoder-decoder stage
Alpha Prediction Loss

Compositional Loss

C_p = FG * α + BG * (1 - α)Matting refinement stage
Input Image Concat Alpha PredAlpha PredExperimental results
SAD(Sum of Absolute Differences)
MSE(Mean Squared Error)
GradientConnectivity

Natural Image Matting via
Guided Contextual Attention
SJTU
AAAI 2020

Motivation• Affinity-based and sampling-based algorithms• Need both FG and BG information to estimate the alpha matte• Only background and unknown areas in the trimap• Learning-based image matting methods• SampleNet, deep inpainting methods, combination• Propose a novel image matting method• based on the opacity propagation in a neural network• We devise a guided contextual attention module, mimic the affinitybased propagation

Baseline StructureO(c(hw) 2)

Guided Contextual Attention Module
Two different feature flows• High-level Alpha features• Low-level image features

Low-level Image Feature• Known part and unknown part
• Extract 3 × 3 patches (as conv kernels• Correlation measure• Compute similarity by convolution

Kernel size：D_k = 3• Regular：• Input：M x H x W• Output：N x H x W• Param：3 x 3 x M x N• GCA：• Input：M x H1 x W1
• Patch：N = H x W , 3 x 3 x M• Param：3 x 3 x M x N• Output：N x H1 x W1

High-level Alpha features• Extract 3 × 3 patches• Reconstruct alpha features• Element-wise summation, residual connection

Result

Context-Aware Image Matting for
Simultaneous Foreground and Alpha Estimation
Portland State University
ICCV 2019Motivation• Simultaneously estimate the alpha map and the foreground image• Attribute• local visual features and global context information• combination of the Laplacian and feature loss
• various effective data augmentation strategies

Context-Aware Image Matting
Xception 65 architecture

16x2x4x

Context-Aware Image Matting
Laplacian loss

Context-Aware Image Matting
Laplacian loss

Feature loss

• Fˆ, ground truth foreground• α^, ground truth Alpha matte• φ_layer, features output by the layer in a pre-trained VGG16 network.
• Our method uses [conv1 2, conv2 2, conv3 3, conv4 3]

Context-Aware Image Matting
Laplacian loss

Feature loss

L1 lossData Augmentation• Subtle artifacts• misaligned JPEG blocks, compression quantization artifacts, andresampling artifacts• Augmentation• Resizing augmentation• Use re-JPEGing and Gaussian blurF, B, Alpha Matting
Trinity College DublinMotivation• Recent two methods• Show improved results by also estimating the foreground colors,• Significant computational and memory cost

• This paper• low-cost modification to also predict the foreground and backgroundcolours• study variations of the training regime, loss functions

Contributions
1. a comparison of min-batch and stochastic gradient descent andthe use of batchnorm vs. groupnorm
2. a study of the different α-matte losses (L1, gradient, laplacianpyramid, compositing loss).
3. a study of the potential benefit of also predicting F and B
alongside α and the possible losses associated with this (L1 lossand exclusion loss).

Network Arch• Encoder-decoder with Unet style architecture• Main difference, also predicts F and B from single encoder-decoder• Extending output channels from one to seven (1 for α, 3 for F and 3for B)

Encoder• ResNet-50• increase the number of input channels from 3 to 9
• encode the trimap• using Gaussian blurs• of the definite foreground and background masks• at three different scales• remove striding, add dilation, [‘layer 3’, ‘layer 4’ ]Batch Normalisation vs. Group Normalisation• A mini-batch size of one can greatly increase the networkaccuracy• Use Group Normalisation (32 channels per group)F, B, α Losses

Fˆ, Bˆ, αˆ Fusion• Predictions for αˆ, Fˆ and Bˆ are decoupled• Equation 1 is not explicitly enforced• Propose a fusion mechanism based on maximum likelihood estimate

Fˆ, Bˆ, αˆ Fusion• Assuming Gaussian distributions

Fˆ, Bˆ, αˆ Fusion• Adopt an iterative block solver approachTest Time Augmentation• We use a comprehensive test-time augmentation, combining
rotation, flipping and scaling

Batch-Size and BN vs. GN
Loss Function and Activation

Evaluating the Impact of Jointly Estimating F, B, αResult

Background Matting:
The World is Your Green Screen
University of Washington
CVPR 2020

Motivation• To extracting (pulling) a good quality matte, require either agreen screen studio, or the manual creation of a trimap• Paper propose• Take an additional photo of the (static) backgroundSupervised Training on the Adobe Dataset

Supervised Training on the Adobe Dataset• Input• An image I with a person in the foreground,• An image of the background B• A soft segmentation of the person S,• A stack of temporally nearby frames M, (optionally for video)• Generate S• Apply person segmentation• Erode (5 steps), dilate (10 steps)• Apply a Gaussian blur (σ = 5)

• Set M to be the concatenation of the two frames before and after• converted to grayscale, focus more on motion cuesSupervised Training on the Adobe Dataset• Residual-block-based encoderdecoder, doesn’t work
• Reason, domain gap,
• trusting the background B’ too much andgenerating holes• Instead, we propose a new Context
Switching blockSupervised Training on the Adobe Dataset• Separately produce 256 channels offeature maps
• Combines the image features from I,
producing 64-channel features for each
• Combines 3*64 and 256 channel, 1x1
conv BN and ReLU

Supervised Training on the Adobe Dataset

Adversarial Training on Unlabelled Real data
• Still fails to handle all difficulties present in real data
1. traces of background around fingers, arms, hairs copied into matte
2. segmentation failing
3. foreground color matching the background color
4. misalignment between the image and the backgroundAdversarial Training on Unlabelled Real data
• Problem• 1. G_Real could settle on setting α = 1 everywhere• 2. Initializing with G_Adobe and fine-tuning with a low learning rate,
not allow significant changes to generate good mattes on real data

Adversarial Training on Unlabelled Real data
• Use G_Adobe for teacher-student learning.
• Obtain (F , ˜ α˜) = G(X; θ_Adobe) to serve as “pseudo ground-truth”

Adversarial Training on Unlabelled Real data
• Adversarial loss
• Loss on the output of G(X; θReal) compared to “pseudo ground-truth”

Adversarial Training on Unlabelled Real data
• For the discriminator, we minimize:Result

Real-Time High-Resolution
Background Matting
University of Washington

Motivation• While many tools now provide background replacement
functionality• yield artifacts at boundaries• higher quality results, but do not run in real-time, at high resolution• In this paper, we introduce the first fully-automated, real-time,
high-resolution matting technique.

Dataset• VideoMatte240K• 484 high-resolution green screen• generate a total of 240,709 unique frames• 384 videos are at 4K resolution and 100 are in HD• PhotoMatte13K/85• 13,665 images shot with studio-quality lighting and cameras in front of agreen-screen• narrow range of poses• high resolution, averaging around 2000×2500

DatasetNetworkBase net
Down sample

Resnet/MobileNet• Alpha matte• Foreground Residual• Error Map• Hidden Layer(32 channel)

Refinement Network• select patch based on error map• Input: [alpha, fgr, hid, src, bg] 1/2• Crop 8 x 8 patch• 3x3 conv, 3x3 conv, 0 pad• 4 x 4 patch, upsample 8 x 8• Concat [src, bg] 1• 3x3 conv, 3x3 conv, 0 pad• 4 x 4 patch, replace [alpha, fgr] 1[32 + 1 + 3 + 6, 24, 16 + 6, 12, 4]

Loss• Alpha loss
• Foreground Residual loss
• Error map loss
• Loss function

Result