Diffusion-based Holistic Texture
Rectification and Synthesis
Guoqing Hao1,2, Satoshi Iizuka1, Kensho Hara2,Edgar Simo-Serra3, Hirokatsu Kataoka2, Kazuhiro Fukui1
1University of Tsukuba
2AIST
3Waseda UniversityBackground• Texture represents visual characteristics of an object’s surface• Application: 3D modeling and image editing• Derivation: hand-drawn images; real images• Difficult to obtain planar texture• Distortions, perspective issues, and occlusions in real images
PerspectiveIssue

Original Image

PlanarTexture

Edited ImageDistortion andOcclusion

Real Image

2

Our Goal• Obtaining planar textures from degraded textures in real images• By rectifying distortions, perspective issues and occlusions• Preserving visual characteristics• Extending applicability of texture synthesis

Proposed
Framework
DegradedTexture Sample
Real Image

Planar Texture3Related Work:
Exemplar-based Texture Synthesis

Real Image• Synthesizing larger texture from small texture sample[Efros and Freeman 2001; Liu+ 2020; Mardani+ 2020]• Texture samples must be square and undistorted

Texture Sample• Difficult to obtain such texture samples:• Perspective issues, geometric distortions, and occlusions• Easy to synthesize disappointing texture

Synthesized Texture

4Related Work:
Texture Scraping [Li+ 2022]• Texture scraping from real images• Texture grouping with convolutional networks and graph networks• Texture synthesis by completing missing regions• Handle occlusions but ignore geometric distortions• Generate unsatisfactory results

Input Image

GroupingTexture

Input Image

GroupingTexture
5Methodology Overview• First diffusion-based framework for rectifying distortions and occlusionsin textures• Occlusion-aware latent transformer• effectively compute guidance for the generation process• Novel mechanism for synthetic training dataset construction
Latent Diffusion ModelVQ
Encoder

Input

Occlusion-aware
Latent Transformer

Diffusion
Process

VQ
Decoder

Cross-attention

OutputNovel6

Synthetic Training Dataset Construction• Collect 22,043 planar texture images from various sources• Apply transformations and masking on planar textures:• Homography transformation; Thin Plate Spline transformation; Free-from mask

• Generate various paired texture degradation dataDi�usion-based Holistic Texture Rectification and Synthesis

SA Conference Papers ’23, December 12–15, 2023, Sydney, NSW, Australia

Homography
Frozen duringTransformation
training C Channel-wise concatenation

Filtering

Free-fromThin Plate Spline
Mask
Transformation
Forward Diffusion Process
Reverse Sampling Process
Cross
attention

Planar ∈ ℝ.×%&'×%&'
VQ-Encoder

TextureDatasets

PlanarTexture!* ∈ ℝ+×.%×.%!) ∈ ℝ+×.%×.%

Add Noise!" ∈ ℝ+×.%×.%...

Homography
Transformation

Degraded Textures
TrainingRectified∈ ℝ.×%&'×%&' Paired Data!̃ ∈ ℝ+×.%×.%7!̃

∈ ℝ+×.%×.%Qualitative Results on Synthetic Test Images

Input Texture

MAT [Li+ 2022]

VQGAN[Esser+ 2021]

Ours

Ground Truth

8

Qualitative Results on Real Images

Real Image

MAT[Li+ 2022]

VQGAN[Esser+ 2021]

Ours

9

Failure Cases• Produce imperfect results under varying lighting conditions and extremedistortions• Possible solutions:• Masking regions with significant lighting changes• Introducing more training data with extreme distortions
Varying Lighting Condition

Input Image

Rectified Texture

Extreme Distortion

Input Image

Rectified Texture

10Conclusion• The first framework for rectifying distortions and occlusions intextures• Introduce a new occlusion-aware latent transformer• Propose a novel mechanism for synthetic training dataset
construction• In-depth evaluation that demonstrates the superior performance ofthe proposed framework
11