Research Activities in
Computational Media Group
Yuichi Ohta, Yoshinari Kameda, Itaru Kitahara

Computational Media Group

Division of Computational Informatics
Center for Computational Sciences
University of Tsukuba
2007.10.31

Network Transmission andInteractive Display of Live 3D Video

Mixed Reality

Privacy Considering Video Surveillance

Topics Presentedby Itaru Kitahara

1NetworkTransmission andInteractive Displayof Live 3D VideoPurpose• To realize a 3D free-viewpoint video system– Multiple audiences– Live sports events in a distant large-scale space(e.g., a soccer stadium)– Freely moving viewpoints• To create an advanced contents technology.
– Developing an interactive 3D video display.
– An ordinary person can enjoy without any special
knowledge about video capturing and editing.

2

Demonstration
The world first live 3D free-viewpoint video system

１２Key Technology• Small-Scale Space Large-Scale Space

Ordinary Approach

Player Billboard

Soccer Games:
Stadium is Large-Scale.
More than 22 players

A single plane andThe dynamic texture3

Physically Correct vs.
Perceptually Correct
These two 3D videos is perceived
almost equally in a large-scale space.

Live 3D Video System

4Experiments in a Real Stadium• Conducting more than 20 experiments– Tokyo National Olympic Stadium– Yoyogi National Gymnasium• An unexemplified system has been developed.

Remarkable Achievements• International Journal of Computer Vision (IJCV)– The flagship journal in computer vision• “2006 Image Electronics Technology Award”– The Institute of Image Electronics Engineering of Japan5

Soccer Ball Tracking• The 3D position of the ball is calculated by thetwo camera images.
• Kalman Filter predicts the lost 3D ball position.
• Tracking the ball in 3D at almost on-line speed.

Back Ground Modeling
Developing a 3D background modeling methodthat needs only simple manual⎛ m⎞⎛ u ⎞ operations.
p’p⎜ ⎟⎜ ⎟⎜1⎟⎝ ⎠⎜1⎟⎝ ⎠λ⎜ v ⎟ = H ⎜ n ⎟q’q

P' = ( xP ' , yP ' , z P ' ) = ( xP , yP , h + dα )Q' = ( xQ ' , yQ ' , zQ ' ) = ( xQ , yQ , h + dα )Q’P’
Homegraphy between theground plane and an input image⎛ u ⎞ ⎛ C11 C12 hC13 + C14 ⎞⎛ x ⎞⎟⎜ ⎟⎜ ⎟ ⎜λ ⎜ v ⎟ = ⎜ C21 C22 hC23 + C24 ⎟⎜ y ⎟⎜1⎟ ⎜C C
hC33 + C34 ⎟⎠⎜⎝ 1 ⎟⎠32⎝ ⎠ ⎝ 31h+dαzyxdPQ

Ground region plane6

Advance Contents Providing
An empathic 3D video system that counts users’
personal preference.
Typical 3 viewpoints wheremany users prefer are
estimated by interview survey.
γβ
A receiverα
A passer

Mixed Reality

7

Purpose• Shared Mixed Reality Supports
Collaboration Tasks.
– Augmenting Visual Information by CG Objects– Smooth Communication by Sharing RealObject’s Appearance

MR Face• MR Face– A mosaic face with real and virtual facial parts• Restoring the eye expressions lost whenwearing HMDs8

Nested Marker• The distance between each user andan object is different.
• Nested Marker is effective for enlarging
the visual range of observers.
Single Marker

Nested Marker

Remote Face-to-Face Mixed Reality• Advanced Tele-Communication– Two people in distantplaces to share MRspace– Real-time computing
is key issue9

Remarkable Achievements• “University President Award 2005”– received by a PhD student (2006.3)• “Microsoft Innovation Award 2007 (IT Division) ”– Innovation Japan 2007 (2007.9)

Privacy Considering
Video Surveillance Systemby Combining theAdvantages of Mobile andEnvironmental Cameras

10Purpose• An advanced video surveillance system– Combining the advantages of mobile andenvironmental cameras• A Privacy Considering Video Media– Appropriately capture, record and display thevideo information with considering about theprivacy information.

Advanced Video Surveillance
Advantage

Weak Point

Env. Cam.

Accurate Object Tracking by Limitation of FOVeasy camera calibration

Mob. Cam.

Wide Field of View (FOV)

Camera calibration issue

Environmental Camera
MobileCamera

MobileCameraOccluded by Obstacles

Out of the Field of View

Video Surveillance System by Combining
Mobile Cameras and Environmental Cameras.

11

Mobile Camera Calibration
λ (u , v,1)T = P3×4 ( X , Y , Z ,1)TEnv.-Cam

Mobile Cam.

SIFT

SIFT

Projective matrix P is estimated by
the result of SIFT-matching.
3D (X,Y,Z) estimatedby stereo vision with SIFT

Privacy Considering Video Media
Effective
Utilizationof Surveillance
Information
Street surveillancecameraPrivacy protected region

Capturingspace
Captured image

Calibrated
mobile camera

3DImageprocessing

MobileVideo Surveillancemodel
3D Modeling Unit

Privacy Protecting Unit

12

Privacy Level Control
3D Space

Demanding Privacy Protection
Undemanding Privacy ProtectionProjection

2D Space
Captured Image

Our method can control the privacy level by using 3D model.

External Review on Center for Computational Sciences, University of Tsukuba

Computational Informatics
Research Division
Computational Media Group
Yuichi Ohta
Yoshinari Kameda
Itaru Kitahara
2007/10/31

13

Visual Augmentation for Pedestrians
Visual Augmentation for Drivers
Massive Sensing
Walking

Mixed Reality

Driving

Many cameras here and there
Computer Vision

Outdoor Mixed Reality / See-Through Vision

Visual Augmentation for Pedestrians(Outdoor Mixed Reality)
Pedestrian + MR gadget + Surveillance cameras• See-Through Vision• Smooth Video HoppingReal Camera ACameraFl
ySmart PDA

VirtualGPSInertia Sensor

CameraReal Camera B14

Outdoor Mixed Reality / See-Through Vision

See-Through Vision
Map (top view)"See"See-Through" on MR Display

Outdoor Mixed Reality / See-Through Vision

Image Synthesis for MR Display

V15

Outdoor Mixed Reality / Smooth Video Hopping

Smooth Video Hopping• Problem on monitoring via many cameras– On switching cameras, images are instantly changed• You need to understand how camera images change.
• You have to learn the locations and directions of the surveillancecameras in the system beforehand.

Surveillance camera images

Outdoor Mixed Reality / Smooth Video Hopping

Idea of Smooth Video Hopping• Visualize camera transition between cameras bypseudo-3D transition video sequence– Virtually show images seen from intermediate viewpointon switching cameras

Camera A

Pseudo-3D transition video(Intermediate viewpoint)

Camera BV16ITS

Visual Augmentation for Drivers(Mixed Reality in ITS)
Vision enhancement for drivers with support of roadside cameras• Navigation by Mixed Reality• Virtual Slope, Floating Virtual MirrorITS / WSD and MR Navigation

Windshield Display andMR Navigation
Windshield Display (WSD)
Projector

MR Navigation
LandmarkMirror
WindshieldDisplayθDriverRetro-reflectivescreen(GPS + Inertia sensor)

Nagivation icons on WSD

V17

ITS / Virtual Slope

Virtual Slope• Safety check on making right turn
• Visualize hidden vehicle on virtual slope
Driver’s viewV

ITS / Floating Virtual Mirror

Floating Virtual Mirror• Safety check on making left turn
• Visualize rear and surrounding area in form of mirror
Driver’s view
Tracking the vehicle byvirtual camera rotation

Floating virtual mirror

Driver’s vehicle

V18

ITS / Environment and Simulator

Environment and Simulator
Cameras on real road commonly used in our campus

Simulator (1st, 2nd)

Massive Sensing

Massive Sensing
Sensing and understanding of human behaviors with a massive numberof sensors (cameras, microphones, etc.)

CAM
MIC
PSD

19

Massive Sensing / Address

We use cameras.
MANY cameras.
• As so many cameras are coming…
– Hard to see/understand what is going on– Hard to maintain camera positions/directions

Massive Sensing / Address

Address of Massive Sensing• Many cameras (+ microphones, etc)
– No calibration in advance– No special/uniform layout (scattered arbitrarily)– Various sensing ability• Detection/Recognition– Relies heavy on data analysis– On-the-fly training

20

Massive Sensing / Surveillance

Visual Surveillance Using
Multiple Cameras• Monitoring videos– Inter-camera Region of Interest (ROI)– Dominant cameras• No layout knowledge in advance– On-the-fly training– No heuristics of the scene

Massive Sensing / Surveillance

Visual Surveillance Using
Multiple Cameras• Monitoring videos– Inter-camera Region of Interest (ROI)– Dominant cameras?

21

Massive Sensing / Surveillance

Visual Surveillance Using
Multiple Cameras• Monitoring videos– Inter-camera Region of Interest (ROI)– Dominant cameras!

Find ROIs that share thesame target space

Massive Sensing / Surveillance

Visual Surveillance Using
Multiple Cameras• Monitoring videos– Inter-camera Region of Interest (ROI)– Dominant camera set

Suggest cameras you expect
to see objects simultaneously.

22

Massive Sensing / SurveillanceVideo Process• Web cameras (roughly synchronized)
×320[pix]×240[pix]×3[rgb]×15[fps]×60[sec] ×60[min]×8[hour]×36[unit] ≒ 3.6TB△8[block]×8[block]×1[gray] ×10[fps] ×60[sec] ×60[min]×8[hour]×36[unit] ≒ 660MB• Frontal Region Detection

N-D vector for framex(t ) = {x1 (t ),..., xi (t ),..., xN (t )}N = C×R
C : # of cameras

R : # of blocks

Massive Sensing / Surveillance

Redundant Block Elimination
Shrink the N-D space•

Step1: Foreground Detection– Eliminate blocks that never find foregrounds•

Step2: Principal Component Analysis– Eliminate less important blocks that do not contribute to the calculation of valuesof principal axes

Example of 12 cameras:
ROIs/all blocks = 253/768

23

Massive Sensing / Surveillance Result 1"Suggestion of Associative Blocks"• Evaluate the co-occurrence degree between two
ROIs for all the ROI pairs• List up the highly correlated ROIs

Massive Sensing / Surveillance Result 2"Dominant Camera Set"
• Evaluate the co-occurrence degree between two
ROIs for all the ROI pairs• Count the amount of correlation for any pairs ofcameras• Select cameras that belong to higher correlationpairs

24