TransFill: Reference-guided Image Inpainting byMerging Multiple Color and Spatial Transformations
Yuqian Zhou 1 , Connelly Barnes 2 , Eli Shechtman 2 , Sohrab Amirghodsi 2
1 IFP, UIUC, 2 Adobe Research

CVPR 2021

Kang LiaoMotivation

Photo 1

Photo 2

It’s hard to capture all perfect faces in one shotMotivation

Photo 1

Photo 2

Replace the naughty girl from other frames

Motivation

Photo 1

Photo 2

Everybody looks good in one shotReference-guided Image
InpaintingChallenges
Reference-guided image inpainting is challenging due to different views,different lights, different scales, etc.

Related Technologies

Image
Inpainting

VideoInpainting

Image
Alignment

Image
Harmonization

Proposed Framework

Source/Reference Image
Color-Spatial
Transformer

Target Image with Mask
Single-Proposal
Fusion

Mask

Multi-homography Proposal Module

Source/Reference Image
Color-Spatial
Transformer

Target Image with Mask
Single-Proposal
Fusion

Mask

Multi-homography Proposal Module

Multi-homography Proposal Module
First, Let’s go to a simple case: one homography

Homography Transformation

Homography Matrix
Different Transformations

Homography Estimation

SIFT Detector

Feature PointsHomographyPoints Matching

RANSAC: Control OutlierImage Alignment

Homography
WarpImage Sequences withDifferent Views

View Normalization

Multi-homography Proposal Module
First, Let’s go to a simple case: one homography

feed all the extracted feature points and their descriptors intoa pre trained OANet for outlier rejection

Multi-homography Proposal Module
Then, Why multi-homography

Multi-homography Proposal Module
Then, Why multi-homography

Baseline method with single
homography

Misalignments

Parallax

A simplified illustration of the parallax

Objects from different depths have different
relative motions. Closer objects providelarger parallax

Images from Wikipedia

Multi-homography Proposal Module

Multiplane Image (MPI)MPI of a scene

Stereo Magnification: Learning View Synthesis using Multiplane Images. Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, Noah Snavely. In SIGGRAPH 2018

Multi-homography Proposal Module
…
H4
H3
H2
H1
H1-H5: Alignment results from five clustering depths One depth layer for one homography(agglomerative clustering method).
H*H6: a homography estimated using all the points

Multi-homography Proposal ModuleThis strategy is very common in image alignment and image stitching fields

Gao, Junhong, Seon Joo Kim and M. S. Brown. “Constructing image panoramas using dual-homography warping.” In CVPR 2011.

Single Image Inpainting Module

Source/Reference Image
Color-SpatialTransformer

Target Image with Mask
Single-Proposal
FusionMask

Single Image Inpainting Module

ProFill (ECCV 2020)
Yu Zeng, et al. ProFill: High-Resolution Image Inpainting with Iterative Confidence Feedback and Guided Upsampling, ECCV 2020

Color-Spatial Transformer Module

Source/Reference Image
Color-Spatial
Transformer

Target Image with Mask
Single-ProposalFusion

Mask

Color-Spatial Transformer Module

Target Image with Mask
Mask
Aligned source image fromi homography
Color Transformer
Spatial Transformer

CSTColor-Spatial Transformer Module

Deep Bilateral Filtering(Siggraph 2017 )Spatial Transformer Network(STN, NIPS 2015)

Optical FlowSingle-Proposal Fusion Module

Source/Reference Image
Color-Spatial
Transformer

Target Image with Mask
Single-ProposalFusion

Mask

Single-Proposal Fusion Module

SPF
Target Image with Mask
Confidence MapMask

Refined Source Image

Merged Refined Source Image

Inpainting Result
from ProFill
Packed Features

Single-Proposal Fusion Module

SPF
It is easier to understand the SPF
from the formulation

Single-Proposal Fusion Module

RecallIntermediate Results

Multi-Proposal Image Fusion Module

Source/Reference Image
Color-Spatial
Transformer

Target Image with Mask
Single-ProposalFusion

Mask

Multi-Proposal Image Fusion Module

MPF
Target Image with Mask

Mask

Refined Source Image

Packed Features

Final Result

Inpainting Result
from ProFill

Visual Comparison

Ablation Study

More Intermediate Results

More Intermediate Results

Conclusion• TransFill, a multi-homography estimation pipeline to obtain multiple transformations of
the source image, where each aligns a specific region to the target image.
• Propose to learn a color and spatial transformer to simultaneously perform a color
matching and make a per-pixel spatial transformation to address any residual differences
after the initial alignment.
• Learn weights suitable for combining all final proposals with a single image inpainting
resultDiscussion• The model is too large, not very compact, not end-to-end. Used pre-trained models ortraditional methods: (1) SIFT, (2) OANet, (3) RANSAC, (4) Agglomerative Clustering
Method, (5) MonoDepth Estimation, (6) ProFill, (7) Deep Bilateral Filtering

Multi-homography Proposal Module

Discussion
End-to-end, address the parallax using multi-grid homography, fast alignment

Nie, L., Lin, C., Liao, K., Liu, S., & Zhao, Y. (2021). Depth-Aware Multi-Grid Deep Homography Estimation with Contextual Correlation. ArXiv,
abs/2107.02524.

Discussion

Nie, L., Lin, C., Liao, K., Liu, S., & Zhao, Y. (2021). Depth-Aware Multi-Grid Deep Homography Estimation with Contextual Correlation. ArXiv,
abs/2107.02524.

Discussion• Merge the multiple fusion strategies

Thanks
Email: kang_liao@bjtu.edu.cn