Stereoscopy: from XY to Z
2D/3D Conversion for Cartoon(2011/08/10)
OLM Digital, Inc.
Salvati MarcVideoP12_408-422_L

2

Overview• 2D/3D conversion, generalities– Rotoscope– The 3 methods (HIT, Camera Map, Depth)– Displace, fill hole and final adjustment– Automatic conversion• 2D/3D conversion for cartoon

3

From 2D to 3D• Add parallax to images– From movie shots (with normal camera)– From cel animation• Adding parallax method– HIT(Horizontal Image Translation)• Shift images manually– Add 3D information, process images• Depth and displacement• Camera mapping and projection4

2D/3D conversion workflow
Create 2D version
Rotoscope

Create LRimages

Create 3D Scene
Camera map

Create depth per layer
Paint, 3D Scene…

Render with2 camera

Displace

HIT

Fill the holes caused by Rotoscope or Displace
Adjust depth sensation

5

Rotoscope• Cutout an image in multiple layers– Separate BG/Characters• Animation– For each frame, cutout the image– Really time consuming• Process each parts– Adjust parallax– Use appropriate method• Camera map, HIT, Depth…

6

Rotoscope sample

7

HIT method• HIT:Horizontal Image Translation– Shift horizontally pair of images– From rotoscope result– Cardboard effect

Display8

Camera Map Method• Create 3D scene– Project 2D image– Render with two cameras• See in other course part– Parallel type, off-axis, cross type (keystone effect)– Use camera rig to optimize depth range– Render with asymmetric frustum
Cameramap

Render

Display9

Depth Method: Overview• Create a depth image– 3D position and volume information– Use rotoscope result to adjust depth per parts– Displacement according to depth– Can be build directly in 2D, or with 3D scene
DisplaceDisplay

10Depth Method: Depth from Rotoscope

Rotoscope

Depth per part

Gather and blur edges

11Depth Method: Local and Global Depth• Local depth from mask(Medial axis, distancetransform, mask edge blur)• Combine with global depth

12Depth Method: 3D Depth• 3D Depth from 3D scene• Re-use existing model– Render depth– Attention! Use appropriate shader!

13

Depth Method: 3D Depth Shader• Use the right depth shaderz1/zLinearDisplace

LinearDisplace

14

Depth Method: Displace• Use displacement map– Adjust displace: comfort/impressive• Example: 40px, divergence limit: 9px (cinema)
Displace（＋10）

1131

20
HIT

920
Displace（－10）
Divergence!

11Adjustparallax

15

Fill the hole
• Ex: when cutting out character from BG– Parallax result in some hole appearance– Need to fill the holes• Paint• Plug-ins

Rotoscope

Displace

16

Adjusting Stereopsis• Comfortable stereopsis– Respect disparity angle range• Use HIT
• Adjust displace values• Adjust camera map inter-axial distance, near/far• Impressive stereoscopic image– Choose the appropriate cut– Adjust the depth• Parallax is related to projection environment– Difficult to check17

Summary• Rotoscope• Add parallax, combine different methods– HIT– Depth– Camera map• Final adjustment– For safety/comfort– For the show

• Want automatic conversion!?
18

What About Automatic Conversion?
• Still a research problem• Rotoscope– Use of color, region recognition– Use shape feature points• Depth / 3D model generation– Color saturation, blue, lightness…
– Parallax to resolve front/back relations– Mesh from silhouette– Find perspective lines• In general semi-automatic methods are more
affordable AND preferable– Designers want control!
19

2D/3D Conversion of Cartoon20

2D/3D conversion for Anime• What we have
– Background– Cel– Animation• How to convert it?
– Background– Cel– Animation21

Detailed BG• Non-photorealistic• 2D or/and 3D

3D BG

2D BG

22

Cel
• Flat colors• Main color• Shadow Color• Outline Color• White Background

Cel animation
23

Animation• 2-3 frames stop rule– 24 frame  8-12 drawings

24

Converting BG• Anime specific colors– Cannot rely on color for depth generation– Simple colors may be easier to rotoscope• Only one still image, not animated– Take time to create 3D model, paint depth– Will be used in many cuts, for long cuts
– Don’t need tracking!

• Anime specific perspective– Automatic generation of BG model challenging– Difficult to use camera map
25

Difficult to Use Color for Depth Generation

BlueRGBSaturation

26Difficult to Use Color for Depth GenerationBlueRGBSaturation27

Normal Perspective

28

Anime Perspective29

Converting Cel• Cel and BG are separated• Multiple characters may be drawn on differentlayers– Need more work from 2D• Color are flat, separated by outline– Part /Region recognition is easy

• Color are flat
– Difficult to see in 3D!

30

Easy Rotoscope: use the outline• Region separated by outline (no tracking)– Gather region by color– Add the outline• Region can be gathered easily (easy tracking)31

Easy Rotoscope: use the outline• Use feature points for smarter rotoscope– Easy to detect (outline)32

Difficult Depth Sensation of Flat colors• Whatever precise depth, flat colors remain flat

Non textured

DepthTextured

33

Difficult Depth Sensation of Flat colors

34

Difficult Depth Sensation of Flat colors

35

Converting Animation• Motion is not continuous– Creates tracking problem• Need to animate rotoscope and depth– Get a rotoscope result that need few tracking– Generate depth from rotoscope– Use tracking tools• Anime specific tracking– Avoid using manual mask creation• Need lot of manual tracking

36Rotoscope with Few Tracking• Color based rotoscope• Region based rotoscope– If simple region specification

37

Generate Depth from Rotoscope• Distance transform, medial axis

• Blur alpha channel• Generate 3D model from silhouettes

38Tracking Ideas• Use connection map• Region tracking

39Tracking Ideas• Use connection map• Region tracking
HeadTorso

ArmLeg40

Tracking Ideas• Use character sheets as connections references• Could be used for automatic rotoscope too41

Conclusion• Live action movie trick can not be used
• Rotoscope maybe easier for cartoon 2D/3Dconversion• Tracking has to be avoided or done by hand!
– Give the possibility to edit to the user

• Flat colors in 3D: still an issue42

VideoP14_470-489_L

43

Want more?
• OLM Open Tools– After Effects plug-ins– Maya plug-ins– And more…
– http://www.olm.co.jp/rd/?lang=en• 3D movies projection (9th,10th,11th)– West Building, Exhibit Hall C, booth 667

44