NICOGRAPH International 2012, pp. 94 - 101

Forensic CG Video Generation
with Augmented Reality Technology
Atsushi Sugiura

Masahiro Toyoura

Xiaoyang Mao

University of Yamanashi{g12dhl02, mtoyoura, mao} (at) yamanashi.ac.jp

Abstract
This paper presents a technology for supporting in-house forensic CG video generation by police andprosecution. For confidentiality, it is usually not allowed to generate criminal CG video throughsubcontracting, whereas existing CG modeling and video editing systems are too complicated to use for police orprosecution. The proposed technique solves this problem with Augmented Reality (AR) technology. We propose
to deploy multiple cameras and AR markers on criminal site. While a user of the police or the prosecution plays
a role of the criminal wearing a video see-through head mount display, video is automatically generated. CG
objects, such as the virtual victim, are displayed on the head mount display and rendered in the video. Our main
contributions in this research are concurrent displaying of first-person and third-person views for visualizing
spatial relationship between the user and its surroundings, and complementary use of marker information frommultiple cameras for solving the occlusion problem inherently caused by operating at the actual criminal site.

1. Introductionof the criminal on site. Modeling of human motion and actual

The lay judge system has been introduced all over the world.

environment can be omitted in this situation. Although the AR

The lay judges are chosen from citizens in general. They discussmarkers may be occluded by the user and real objects,

about guilt or innocence and the appropriate punishment withcomplementing marker information among multiple cameras

professional judges. The system makes the judge to follow thecan solve the problem. Displaying multiple views from thesense of ordinary people. It also makes the period much shorter.

multiple cameras will help the user to grasp the spatial

Since the lay judges are amateur, the materials for them are

relationship between the user and its surroundings, even if therequired to be easily understandable without expert knowledge.

user is not well trained for using AR system.

We focus on the demonstration CG video as a part of forensic

In the next section, we first introduce related works. Section 3

materials, which enables the judges to understand the criminal

describes how the proposed system generates CG video. The

situation intuitively. Although it is already generally used in TV

results of a subject study are described in Section 4, and Section

programs, it has not been frequently used in the court. This is

5 concludes the paper.

because subcontracting is inappropriate for confidentiality,

2. Related works

while police or prosecution is not expertise for generating
movie. We propose to support in-house CG video generation

Several research groups have proposed to use CG models aswith Augmented Reality (AR) technology.

forensic materials. Section 2.1 introduces the existing work on

AR technology support to display CG objects in video

creating CG models based on the investigation data ofcaptured real world. Supplementary information can be added incriminals.. Section 2.2 briefly reviews the medical application

the video. We suppose multiple cameras and multiple markers

of AR.

are deployed on the criminal site. A user of the police or the

We use multiple views to help the user understand the spatialprosecution plays the role of the criminal wearing a video

relationship between himself/herself and surroundings. The

see-through head mounted display. CG objects are rendered insystem consists of multiple cameras and multiple PC connected

the video captured criminal site and displayed to the headthrough a client/server network. An existing work on multiple

mounted display of the user. All things the police or theview AR is described in Section 2.3. The research work which

prosecution should do for generating CG video is playing a role

using AR technology in a network environment is introduced in
94

NICOGRAPH International 2012, pp. 94 - 101
2.4.

The video generated by interpolating multiple views gives
virtual views, which is not suitable for being used as forensic

2.1 CG Modelingmaterials since the reliability is most important for the judge.

Creating the 3D CG model of criminal site requires technical

Our system generates third-person views from real cameras.

skills. It also costs much time. March et al. proposed a

Viewers can change the camera, not the viewpoint, freely. They

technique for converting forensic pathology descriptions into

can examine the situation from the viewpoints of different

3D CG models [1]. Thali et al. generated individual models ofcameras.

corpuses using markers [2]. Captured images, MRI (Magnetic

2.4 Practical use of AR and network

Resonance Imaging) and CT (Computed Tomography) data areintegrated to create the 3D CG model with the markers.

There is a research [7] which combines AR technology and a

Compared to the corpuses, criminal site and the animation

network. It transmits and receives the information on the real

depicting the process of a crime are much more difficult toworld between remote places. It aims at deleting physicalmodel since they varies case by case and there are no standard

restriction, and can perform all operations in real-time.

models are available. We solve the problem by using AR

Our system also performs the data communications between

technology. In our system, the CG model of victim, which canpersonal computers using a network. To be immersive, real time

usually be obtained by modifying a standard model, is rendered

processing is important. However, there is a delay problem by ainto the captured image of crime site, and a video depicting the

network. In order to solve the problem, the data volume forprocess of crime can be automatically generated simply bycommunication was reduced by sending thumbnails of images.

having a user playing the role of criminal. CG objects can be

3. Proposed methodarranged in the crime site intuitively by using any markers.

Our method generates a video for investigating a crime scene

2.2 Medical applications of AR

with AR technology. The video consists of the images obtained

Augmented Reality technology has been adopted to an

by rendering the virtual objects, such as the 3D model of aendoscope surgery supporting system [3][4]. A small hole on

victim, weapon, over the real camera captured images ofthe surface of a patient's body is made for the endoscopecriminal site. The knowledge about CG modeling is not needed.

surgery. An endoscope and forceps are inserted into the hole.

When using the actual criminal site as the background, we

The operation is done with a narrow view of the endoscope. The

cannot expect large equipment such as position sensors or asurgery is skill demand and requires advanced training. AR

good lighting condition. We should detect AR markers in suchtechnology supports the surgery. Virtual internal organs are

situation. The problem is solved by the complement of multiple

generated from CT images of the patient before the operation.

marker information with multiple cameras. The multiple

They are projected on the surface of the patient's body during

cameras also support a user to act in the space. The user wears a

the operation. The deformable registration between the patient's

HMD, and multiple views of the cameras are displayed on the

body and the virtual organs is given with optical markers. Sincemonitor of HMD.

human body is a non-rigid object, the registration is usually very

Section 3.1 describes the overview of our system, and Sectiondifficult.

3.2 describes the complement of multiple marker information.

Portable system is desired for our system. Easy calibration ofSection 3.3 is about the user support by displaying multiple

multiple cameras is very important in case of in-house movie

views, and Section 3.4 introduces the configuration ofgeneration. We achieve this by adopting ARToolKit markers [5]server/client network for transferring multiple views.

for calibration and registration. Our system is portable and easy

3.1 Overview

to handle.

We arrange multiple cameras and multiple AR markers on the

2.3 Displaying multiple views for AR

actual crime site. The lightings and the position sensors are not

How two or more views are displayed in AR has been

available since the space is not a managed studio. The markers

actively discussed. Sport is a good application of AR. The 3D

are often occluded by the user or the other objects in realsoccer observation system developed by Inamoto and Saito

world. When multiple cameras are placed in the space, we cangenerates free viewpoint video from images captured with a

expect some of them can detect the marker. Then, marker

large number of cameras [6]. A viewer can specify an arbitrary

information

viewpoint and the system generates the corresponding image by

information between cameras. In addition, the multiple

interpolating multiple images from different cameras. Cameras

cameras provide multiple views for the user to grasp theare calibrated in advance.

spatial relationship of the virtual objects and real objects
95

can

be

complemented

byintegrating

theNICOGRAPH International 2012, pp. 94 - 101
including the user himself/herself.

the mark is in an inappropriate angle relative to the camera, themarker becomes undetectable. We solve the problem by using

The configuration of our proposed system is shown in Figure
1. As shown in Figure 1(a), the user wears a head mounted

the information from multiple cameras complementally.

display with cameras (HMD). The video of first-person view isFigure 2 shows the flow of the complement of marker

displayed on the HMD. Multiple cameras which are

information.

connected to the network are arranged in the space. They
capture the video from the third-person viewpoints. The video istransferred to the user's HMD. A composite video with both of
the first-person view and the third-person views is provided
to the user.
Multiple markers are deployed in the space. The individual
markers correspond to individual virtual objects. The marker
for a virtual victim is set on the floor, and one for a virtual
weapon grabbed in the user's hand. The virtual victim and thevirtual weapon are displayed on the positions of the markers in
the video. It means the video captured in the actual crimesite is augmented with the virtual objects by AR technology.
Web camera

HMD with camera

Figure 2. Complement of undetectable marker information.
When camera 1 fails to detect a mark, a geometric
AR marker

transformation matrix is calculated using the information from(a)Before overlaid. The user wears HMD. The multiple cameras andcamera 2.

markers are deployed in the space.
Virtual weapon

When a marker is detectable, a geometric transformation
matrix P can be calculated with the function of ARToolKit forcomputing the marker coordinate system. Then a virtual object
can be displayed on the marker in the marker coordinate system.
Without loss of generality, let`s consider a system consists of
two cameras and two markers as shown in Figure 3. Denote thegeometric transformation matrix of marker A and B detectedfrom camera 1 are P1A and P1B, the geometric transformation
matrix of marker A and B detected from camera 2 are P2A and

Virtual victim

P2B. Assume camera 1 can detect both marker A and B, but(b)After overlaid. Video captured with multiple cameras. The virtual

camera 2 can detect marker B only. Because marker A is

victim and the virtual weapon are displayed on the positions of theundetectable from camera 2, geometric transformation matrix

markers in the video.

P2A cannot be calculated. Assume the camera coordinate system

Figure 1.Configuration of the proposed system.

of camera 1 viewed from camera 2 is Φ 2 → 1, than thetransformation matrix of marker B viewed from camera 2 via
camera 1 becomes Φ 2→ 1･P1B. That is, we can obtain the

3.2 Complement of multiple marker information
ARToolKit is a library for creating AR environment easily. It

following equations:detects a marker with one camera, estimates the position andposture of the marker, and displays a virtual object on the

P2B = Φ2→1 ・P1B

position indicated by the marker. However, when a part of the

Similarly, the transformation matrix for maker A viewed

marker is absent from the field of view of the camera or whenfrom camera 2 via camera 1 is
96

(1)NICOGRAPH International 2012, pp. 94 - 101

P2A = Φ2→1 ・P1A

3.4 Network by socket connection(2)

A network is used in order to transmit images of two or more

Combining equation (1) and (2) we obtain the following

viewpoint between personal computers. A server/client model

equation:using TCP socket connection is used. The image of the third

P2B = Φ2→1 ・P1B

person view mode acquired with another camera is sent to a(3)

server using a network. However, the images captured with theSince P1A, P1B, and P2B are available, P2A can be computed

cameras usually are large, and the system can become slow

with equation (3) and the marker coordinate system of marker A

depending on the traffic of network. However, real time

viewed from camera 2 can be calculated from P2A.

processing is very important in case of an AR system. We solve

The principle behind the above calculation is that we can firstthe problem by using small images for the third-person view.

obtain the spatial relationship between the two cameras throughThe downsizing is executed before the images are transferred.

the marker which is detectable by both camera (Equation(1))

4.

and then compute the coordinated system of the occluded

Experiments

marker from its coordinate viewed from another camera.

We examined how much subjects grasp the spatial

Therefore, to an occluded marker, at least one more marker

relationship between the subjects and virtual objects bywhich is not occluded from both cameras is required. Moreover,displaying

there is no way to complement a marker which is occluded frominformation between multiple cameras and multiple markers is

all cameras.

also examined through another subject study.

third-person

views.

ComplementofmarkerIn the proposed system, a network of client/server model is

4.1 Enviroment

constructed for communicating among cameras. Marker
information is detected by the client and sends to a server. A

Experimental environment is shown in Figure 4. Three PCs

server receives data from a client, when there is a marker which

are deployed in a space. A HMD with cameras (Wrap920AR,

was undetectable, the server complements marker information

Vuzix Corporation) is connected to a server PC. The resolution

by using the data from other clients.

of cameras is 640x480 and frame rate is 30fps. The

2to1

Camera 1

P1B

resolution of display monitors is 800x600.
Camera 2

P2A(undetectable)

P1B

P2B

Client 1

Client 2

Figure 3. Marker information complement.. The projection metrics by
the two cameras and two markers can be calculated by thecomplement of marker information.
Server

3.3 Displaying multiple views to the user
When creating a synthetic image, the spatial relationship ofFigure 4. Environment for experiments. A server PC with HMD

reality and an virtual object becomes important. It may be

and two clients PCs with web camera are deployed in a space.

difficult for a performer to grasp the whole spatial relationshipThe cameras capture the video. The marker information are

only with the image of a first person viewpoint. We improveacquired from the cameras.

the situation by providing the performer the image of a third
person viewpoint. Since the performer can see the image of

The HMD provides the video of the first-person view to avarious viewpoints, it becomes easy for him/her to understandsubject. The remaining two PCs are clients with web cameras

the spatial relationship among virtual objects and actual(Logicool C920 and QCam-200SX) connected. They capture

criminal site, and to set up optimal positions and numbers ofvideo from the third-person views. The video were 640x480 incameras and markers, which is usually very difficult forresolution and 30fps in frame rate.

un-specific environment.

Two markers are deployed in the space. One of them
represents the position of a virtual victim, and the other the97

NICOGRAPH International 2012, pp. 94 - 101
position of a virtual weapon. Figure 5(a) shows the marker forprocess. Since the weapon marker went out of sight of HMD

the virtual victim, and Figure 5(b) the marker for the virtual

cameras, the virtual weapon could not be displayed. Figure 7(b)weapon. The size of both markers is 80 square mm. When theshows the result with marker complement process. The virtual

marker for the virtual victim was put on the floor, it rarely came

weapon could be displayed even if the marker went out of theinto sight of the subject. Therefore, the marker was put on a bar

view. The marker information was transformed to the server PC.

as shown in Figure 5(a). We designed the weapon as shown inThe

Figure 5(b) to correctly calculate occlusion via the virtualcomplemented by integrating marker information detected in

victim. The edge of the weapon was rendered as a virtual object.

other cameras.

informationofundetected

weaponmarkerwas

Figure 8(a) represents the first-person view, and Figure 8(b)the third-person view. Based on the victim marker, which could
be detected in both two cameras, the weapon marker iscomplemented in its position although the marker could not be
detected in the camera of the first-person view. The informationof the relationship between the victim marker and the weapon
marker is transferred through the network. As the result, the(a) Victim marker(b) Weapon marker

virtual weapon could be displayed in the first-person view.

Figure. 5 Markers for experiments. The size of both markers is
80 square mm. The height of the victim marker is 1.1m. A
handle is attached to the weapon marker.

4.2 Complement of marker information
In this experiment, server and client processing is correctly
performed among two or more PCs. We examined whether thecomplement of marker information was realized.
An operation scene is shown in Figure 6. The user wore a
HMD and grabbed the weapon marker in one's hand. The
cameras on the HMD captured the video of first-person view.
The video of the third-person view from the other cameras
were transferred to the server PC.
(a) Result without complement of marker information. Since theweapon marker went out of sight of HMD cameras, the virtual
weapon could not be displayed.

Figure 6. Operation scene. The user wore a HMD and grabbed
the weapon marker in one's hand. Multiple cameras capture thevideo.
(b) Result with complement of marker information. The
The victim marker was arranged in the detectable space frominformation of undetected weapon marker was complemented

all cameras. In several frames, the weapon marker was out of

by integrating marker information detected in other cameras.

sight in first-person view. The result of marker complement is

The virtual weapon could be displayed using the information

shown in Figure 7.

even if the marker went out of the view.

Figure 7(a) shows the result without marker complement

Figure 7. Marker information complement.
98

NICOGRAPH International 2012, pp. 94 - 101
We examined the scene that the subject stabbed the virtual
victim with the virtual weapon. Whether the virtual weapon was
detected or not is shown in Figure 9. The marker could not be
detected in 765 frames without complement of marker
information as shown in Figure 9(a). Comparing with the result,the marker could be detected in all frames with complement ofmarker information as shown in Figure 9(b). We conclude thatthe complement was effective for displaying the marker for CG
video generation in our proposed system.

4.3 User support by displaying multiviews
In this experiment, the ease of grasping spatial relationship(a) First-person view.

between the virtual and real objects by displaying video ofadditional third-person views was examined.
This system was used by three male subjects and a female
subject in their 30's. Figure 10 shows captured images of oursystem. The third-person views are not displayed in Figure10(a), and they are displayed in Figure 10(b).

(b) Third-person view.
Figure 8. Complement of marker information based on thevictim marker. The marker information calculates using thevictim marker of both first-person view and third-person view.

No complement

Object(a) No third-person view. Grasping spatial relationship isdifficult with first-person view only.

On

Off

0

100 200 300 400 500 600 700 800 900[frame](a) The result without the complement of marker information.
Complement

No complement

ObjectOnOff
0

100 200 300 400 500 600 700 800 900[frame](b) With third-person views. The third-person views are added
to the upper right of the image displayed to the HMD. Grasping(b) The result with the complement of marker information.

spatial relationship become easier by referring to the

Figure 9. The result of marker detection with/without

third-person views.

complement of marker information through the network.

Figure 10. Images displayed to the HMD of a user.
99

NICOGRAPH International 2012, pp. 94 - 101
From the comments of all subjects, displaying third-person
views helped to easily grasping the spatial relationship between
the virtual objects and real objects including the subjects
themselves. We had the subjects put the virtual weapon at a
specified point to evaluate how they could grasp the spatial
relationship. A cube was displayed on the victim marker. The
first task was to put the end of edge of the virtual weapon on thesurface of the cube. The second task was to put the end at 50mm
distance away from the surface of the cube. We examined theaccuracy and time to complete the tasks with and without
third-person views displayed. When the virtual objects areplaced on 300mm distance from each other, we start to measure
the time for the tasks. We stop the time when the subjects(a) With third-person view

stopped the motion of the virtual weapon. The scene of the tasks
is shown in Fig. 12
300mm(a) Before Task
0mm

Move

(b) A example of generated image of third-person view(b) Task 1

Figure 12. Experiment scene. We evaluated how they could
grasp the spatial relationship. The virtual weapon is brought

50mm

close to a cube. The lapsed time and distance at that time aremeasured.
MoveAnother problem was that the planer weapon marker as(c) Task 2

shown in Figure 15(a) could not be detected from many

Figure 11. Tasks for evaluating the accuracy and time bydirections. When the marker was rotated, it was often missed

displaying third-person views.

from third-person view cameras. We improved the marker asshown in Figure 15(b). Five markers are arranged on a cube. It

Results are shown in Figure 13 and Figure 14. The error of

was robustly detected from cameras when the relative position

distance between virtual objects became smaller by displaying

between the markers and cameras was changed. The subject

the third-person views. The time for the tasks also became

could play a role of the criminal without special attention to theshorter. Visual support of the third-person views was confirmed

marker direction.

to be valid for the tasks.

We also considered about the position of the third-person

Based on the problems found through experiments, theviews in the display monitor. Windows on the lower right often

system has been improved. The data amount of captured images

overlapped with the right hand of the users. Therefore, for thewas large to transfer through the network in real time. The

right-handed users, the virtual weapons may be occluded by theoriginal size of captured images is 640x480. The image size ofwindows. We changed the position of the windows to the upper

160x120 is used for The third-person views. Before transferring,right of the monitor.

the image size of 640x480 is reduced to 160x120. We solve theproblem by transferring only small images (160x120) to theserver PC. It reduced the data amount on the network and realtime display of multiple views was realized.
100

NICOGRAPH International 2012, pp. 94 - 101
Server/client network provide both of the first-person view andTime (s)Error of distance (mm)the third-person views on the HMD. It supports users to grasp

40

80

the spatial relationship between themselves and surroundings.
The marker information detected from multiple markers are

60

30

40

20

integrated in the server PC. When a marker is occluded from acamera, the information of the marker can be given by anothercamera.
As a future work, the interface for augmenting additional
information to the video is desired. It should support the20

10visualization of information in the video. Recently, CT and MRI
have been introduced to judicial autopsy. Such data will

0

With
views

Without
views

0

facilitate the generation of precise 3D models. We plan toWith views Without
views

provide the technologies for calibrating CG models to real(b) Time

As another potential application, the lay judges can not only(a) Error of distanceobjects.

Figure 13. Result of task1.

watch the video, but experience the criminal site themselves byusing the AR system themselves, so as to understand theTime (s)Error of distance (mm)situation of the crime better.

40

80

References30

60[1] J. March, D. Schofield, M. Evison, and N. Woodford,"Three-Dimensional

40

Computer

Visualizationof

Forensic

Pathology Data," The American Journal of Forensic Medicine

20

and Pathology, vol.25, pp.60-70, 2004.
[2] M. Thali, M. Braun, U. Buck, and E. Aghayev, "Scientific20

10Documentation, Reconstruction and Animation in Forensic: Individual
and Real 3D Data Based Geo-Metric Approach Including Optical

0

With
views

Without
views(a) Error of distance0

Body/Object Surface and Radiological CT/MRI Scanning," Journal ofWith views Without
views

Forensic Sciences, vol.50, No.2, 2005.
[3] H. Moriguchi, Y. Kuroda, H. Takiuchi, and O. Oshiro, "The(b) Time

calibration-less estimation of camera model of oblique-viewing

Figure 14. Result of the task2.

endoscope for AR surgery," IEICE Technical Report, vol.110, no.108,
pp.43-46, 2010.
[4] Y. Kuroda, H. Kanamori, H. Takiuchi, M. Tanooka, M. Imura, T.
Kuroda, O. Oshiro, "Inside-body Registration with Pair-line Matchingfor AR Surgical Navigation," Journal of The Virtual Reality Society ofJapan, vol.14, pp.435-444, 2009.
[5] H. Kato and M. Billinghurst, "Marker tracking nad hmd calibration
for a video-based augmented reality conferencing system,"(a) A single planar marker(b) Markers on a cube

International Workshop on Augmented Reality, pp.85-94, 1999.

Figure 15. Marker for the virtual weapon. A single planar marker was[6] N. Inamoto and H. Saito, "Fly-Through Observation System for

rotated, it was often missed from third-person view cameras. Five

3D Soccer Movie Based on Viewpoint Interpolation," Journal of The

markers are arranged on a cube. The subject could play a role of theInstitute of Image Information and Television Engineers, vol.58, No.4,

criminal without special attention to the marker direction.

pp.529-539, 2004.
[7] Y. Okamoto, I. Kitahara, Y. Ohta, "3D Human Body Display by

5. Conclusions

Using Depth Information for Remote Shared Mixed Reality," IEICE

In this paper, in-house 3D CG video generation supporting

Technical Report, vol.94, pp.830-838, 2011.

system using AR technology is proposed. The system focuseson the reenactment of the crimes causing injury or death, which
are impossible to reenact actually. PCs connected with a101