The Journal of the Society for Art and Science, Vol. 20, No. 1, pp. 55-71 (2021)

View-dependent Projection Mapping Enhanced by Real Background
Khuslen Battulga1) (Nonmember)

Tadahiro Fujimoto1) (Member)

1) Graduate School of Engineering, Iwate University
khuslenb@gmail.com

fujimoto@cis.iwate-u.ac.jp

Abstract“Spatial augmented reality (SAR)” or “projection mapping” projects an image of a virtual object on the surface ofa real object. A “view-dependent” display shows the virtual object with its correct appearance for an arbitrary
viewer’s position. If the virtual object and the real object have different shapes, the virtual object’s image toproject needs to be correctly distorted according to the viewer’s position. Besides, the difference causes the real
object’s surface to have some empty areas onto which the virtual object is not projected. Such empty areas seriously degrade the viewer’s feeling that the virtual object merges into the real world. We propose a method toeliminate undesired empty areas by projecting the real background behind the real object in a view -dependent way.
Our method converts a real background’s image captured by a fixed camera to an appropriate image for a viewer’s
position based on homography. This image conversion approximates the background ’s shape by a plane adjusted
for an arbitrary viewer’s position. The plane is determined by practical parameters interpolated on a 3D grid space.
Consequently, the viewer does not feel the presence of the real object and feels the virtual object merging into thereal world.

55

The Journal of the Society for Art and Science, Vol. 20, No. 1, pp. 55-71 (2021)

1. Introduction

problems in SAR. Many view-dependent methods to solve this problem
have been proposed. On the other hand, as far as we know, there is no

1.1 Background

existing method to treat the “empty area” problem. In this paper, we pro-

Augmented reality (AR) [1][2] is a computer-based technology to gen-pose a method to eliminate undesired empty areas by projecting the realerate a virtual world by adding a variety of virtual things, such as virtual

background behind the real object in a view-dependent way. In order toobjects, to the real world in order to augment the real environment. The

treat a far background in a large space, even outdoors, the real back-virtual world is usually displayed on various kinds of devices, such as aground is not measured by an RGB-D camera with a depth range limit

monitor, a tablet, a smart phone, a head-mounted display, and special

but captured by a usual RGB camera and approximated by a plane ac-

glasses. A viewer usually has his/her own display device to see an image

cording to a viewer’s position. Then, the captured image is converted toof the virtual world generated for only him/her according to his/her view-

an appropriate image for the viewer’s position based on homography.

ing position. This means that multiple viewers see different images re-

Although the plane is determined manually, a practical way is proposed

spectively. The viewer feels as if a virtual object actually existed in theto adjust it easily, intuitively, and efficiently by devising effective back-real world if the appearance of the object is observed correctly from anground parameters. A simple grid-based interpolation mechanism forarbitrary position around it. The display to show a virtual object’s correct

the parameters provides an appropriate plane to an arbitrary viewer’s

appearance according to a viewer’s position is usually called view-de-position. Consequently, the projected real background shows its appear-

pendent display, which is so important in AR. One typical way for view-

ance as correctly as possible so as to match the real background directly

dependent display is to use a marker to indicate the exact 3D position ofseen by the viewer seamlessly along the real object’s contour. Our

a virtual object to obtain a correct image of the object seen from amethod makes the viewer not feel the presence of the real object but feelviewer’s position. Some systems use a screen made from special material

the virtual object merging into the real world.

to reflect a different correct image according to a viewer’s position. “Par-Many view-dependent SAR methods have been developed to allow

allax” is a visual effect caused by the difference of the positions of left

multiple viewers to see a virtual object’s correct appearance simultane-

and right eyes. In a typical stereoscope system, a viewer wears special

ously by relying on a specialized device with multiple projectors and a

glasses that give the respective eyes different images generated by thereal object with specialized material. On the other hand, an existing

parallax effect. In AR, usually, a displayed image depends on a viewer’s

method using a non-specialized configuration with an off-the-shelf pro-position.

jector and an ordinary real object sacrifices the important merit of SAR;

Spatial augmented reality (SAR) [1][2][3][4] augments the real world

only a single viewer is allowed to see the correct appearance. Our

without display devices described above. SAR usually uses a projector tomethod has the same situation. However, the simplicity of our method

display an image by projecting it onto the surface of a real object, such as

enables the integration with an existing method developed for multiple

a screen, a wall, a building, a house, a car, and even a human in the realviewers.

world. Projection mapping is a well-known technology to achieve SAR.

In Section 2, we describe related works and arguments for our method.

In this paper, we use “SAR” and “projection mapping” as the same

In Section 3, we explain our method. In Section 4, we show experi-

meaning if the usage causes no problem. In SAR, a viewer sees an image

mental results. Finally, in Section 5, we conclude this paper.

projected on a real object’s surface by the naked eyes; that is, multiple
viewers see the same image simultaneously. In this sense, the displayed

2. Related Work and Argument

image does not depend on the viewer’s position [2]. Actually, an SAR
system sometimes, or often, does not need the view-dependent display

2.1 Related Workof a virtual object. If it is needed, one easy way is to project an image of

One of pioneering technologies with respect to SAR is “the office ofa virtual object onto the surface of a real object whose shape is the samethe future” [5] proposed in 1998. The representative works proposed inas the virtual object’s shape. In this case, the virtual object’s appearance

recent 20 years are surveyed intelligibly in [6][7]. The view-dependent

projected on the real object’s surface is observed correctly from an arbi-

projection needs an appropriate projection image to project on a real ob-

trary viewer’s position. However, it is practically difficult to prepare

ject surface by a projector. Such a projection image depends on a pro-such a real object for an arbitrary virtual object. Then, the difference be-jector’s position, pose, and optical properties as well as a real object sur-tween their shapes causes a problem of “incorrect appearance” of theface’s geometry and material properties such as texture and reflectance.

virtual object. The difference between the shapes causes another prob-

The projector and the real object surface need to be calibrated in advance.

lem. The real object’s surface has “empty areas” on which the virtual

A calibration in SAR consists of geometric calibration and photometricobject is not projected. Such empty areas seriously degrade a viewer’s

calibration [6][7]. A geometric calibration makes a projected image on

feeling that the virtual object merges into the real world.

a real object surface geometrically correct without distortion. Whenmultiple projectors are used, projected images by the respective projec-

1.2 Objective

tors are partially overlapped so as to reconstruct a virtual object by their

The above “incorrect appearance” problem has been one of maincorrect arrangement. A photometric calibration makes a projected image
56

The Journal of the Society for Art and Science, Vol. 20, No. 1, pp. 55-71 (2021)
photometrically correct such that the projected image’s colors become

virtual object image on the surfaces. The projected image is seen by an-

as similar as possible to its original image’s colors according to a realother viewer. The viewers perceive the virtual object existing in the same

object surface’s material properties. When multiple projectors are used,

3D position. The remaining pair measures and projects the environment

the color intensities of the respective projected images are adjusted on

between the viewers. Even in this system, each projected image gives

the overlap areas. A typical calibration uses a camera to capture a pro-the virtual object’s correct appearance to only a single viewer. The above

jected image for automatic geometric and photometric calibration. Suchsystems can work in only a non-large room-scale environment because

a calibration is usually called projector-camera calibration [6][7].

the real-time measuring by a Kinect sensor, generally an RGB-D cam-Our method involves geometric calibration. Therefore, we mention it

era with a depth range limit, is necessary. On the other hand, many SAR

below. A geometric calibration estimates the geometry of a real object

systems to allow multiple viewers to see the correct appearance of asurface and the intrinsic and extrinsic parameters of a projector. A pro-common virtual object simultaneously have been proposed [27][28][29]jector-camera calibration also estimates the intrinsic and extrinsic pa-[30][31][32][33]. Such SAR is often called light field projection. These

rameters of a camera. Some methods estimate unknown projector pa-

systems use specialized configurations with special devices and real ob-

rameters by using known camera parameters while others estimate allject surfaces; typically, they use multiple projectors and an anisotropic

unknown projector and camera parameters. Besides, some methods es-or lenticular surface made from special material to reflect different im-

timate an unknown real object surface’s geometry while others estimate

ages to different directions. The development of view-dependent projec-

unknown projector and camera parameters by using a known surface’stion technologies for multiple viewers using non-specialized configura-geometry. “Shader Lamps” [8] is an early well-known SAR work. First,

tions is an important future work in SAR.

the 3D shape of a real object is measured by a 3D touch probe scanner

2.2 Argument

to obtain its 3D model as a virtual object. Then, multiple projectors arecalibrated by projecting markers from each projector onto the real ob-

The main purpose of AR is to augment the real world by adding aject’s surface and matching the projector’s marker pixels with 3D points

virtual object and making a viewer feel the object merging into the realon the surface. Geometric projector-camera calibrations are categorizedworld. Multiple viewers can experience the augmentation simultane-

into semi-automatic calibration and self-calibration [7]. A semi-auto-

ously by using their respective display devices. On the other hand, SAR

matic calibration uses a specialized apparatus [9][10][11][12] [13]. Typ-has mainly two purposes. One purpose is to decorate a real object’s sur-

ically, a projector-camera pixel correspondence is estimated by project-

face by projecting various images. Multiple viewers can enjoy the dec-

ing structured light patterns, such as a chessboard pattern, onto a planar

oration simultaneously by the naked eyes [8][19][20][21][22][23][34].

surface, such as a screen, by a projector and capturing it by a camera. A

This purpose needs a 3D model of the real object as a virtual object.

homography matrix is often used to relate the planar surface with theAnother purpose is to display a virtual object on the surface of a real

image planes of the projector and the camera. A self-calibration does not

object from which the virtual object does not originate, such as a screen

use a specialized apparatus [14][15][16][17][18]. Typically, structured

and a wall. In this case, the real object’s shape has no relation with thelight patterns are projected onto a non-planar surface whose geometry isvirtual object’s shape. A typical method aims to provide the view-de-

unknown. Recently, various dynamic projection mapping methods havependent projection of the virtual object without any consideration for the

been proposed to calibrate a moving real object, such as a human facerelation between the shapes. When the virtual object is desired to merge

and clothes, and give an appropriate projection in real time. For example,

into the real world, there is no problem if the real object should be shown

an object silhouette [19], a rigid surface [20], and a non-rigid surface

as an element of the real world [26]. On the other hand, there are many[21][22][23] have been treated.

cases in which the presence of the real object should not be shown. Con-An existing SAR method using a non-specialized configuration basi-

sequently, the “empty area” problem happens. When a viewer moves

cally enables view-dependent projection for only a single viewer. For

freely within a wide range around the real object, this problem becomesexample, “HeatSpace” [24] and “OptiSpace” [25] systems use projec-

more serious.

tors and Kinect sensors. Each system measures and analyzes the envi-

As far as we know, there is no method to solve the empty area prob-ronment in a room and the movement of a single viewer during a certain

lem intendedly. The system of [29] uses a specialized mirror rotating attime, and determines the optimal surface on which a virtual object image

high speed to reflect a virtual object image to a viewer. The real back-should be projected in the environment. The viewer can see the correct

ground behind the mirror is seen through it intermittently, which resultsappearance of the virtual object within an analyzed small area. In “dy-in causing a visual effect by which the virtual object is floating in front

adic SAR system” [26], two viewers stand face-to-face with each otherof the background without feeling empty areas. However, the viewer

near the opposite walls in a room and interact with a common virtual

unavoidably perceives the presence of the mirror. Our method solves theobject. This system uses three projector-Kinect pairs mounted on theempty area problem without any specialized device.

ceiling. The two pairs face the viewers respectively. Each pair measures

Our method uses not only a projector but also a camera, which is used

the surfaces of one viewer and his/her back environment and projects a

to capture a real background. We assume that the intrinsic and extrinsic
parameters of the projector and the camera and the geometry of a real
57

The Journal of the Society for Art and Science, Vol. 20, No. 1, pp. 55-71 (2021)
object are already known by geometric calibration. An arbitrary photometric calibration method can be used for our method. However, in theexperiments, photometric calibration was not applied and a white object
with a diffuse reflection surface was used as a real object. Besides, a
tracking sensor is used to track a moving viewer in real time. Our
method obtains a color image of the background by the camera, approximates the shape of the background by a plane, and converts the image
to an appropriate image for a viewer’s position based on homography.
This enables our method to treat a far background in a large space, evenoutdoors. The plane is determined by effective background parameters,
which are interpolated on a 3D grid space to adjust the plane for an ar(a1)bitrary viewer’s position. Our method uses the “two-pass algorithm” [5](a2)[35][36] to obtain a projection image containing a virtual object and areal background for view-dependent projection. The projection of thisimage results in displaying the virtual object’s correct appearance on areal object’s surface whose empty areas are eliminated by the projected
real background according to a viewer’s position.

(b1)

(b2)

(b3)

(c1)

(c2)

(c3)

(d1)

(d2)

(d3)

(e1)

(e2)

(e3)

(f1)

(f2)

(f3)

(g1)

(g2)

(g3)

3. View-dependent projection mapping enhanced by real background
3.1 Outline
If the surface of a real object is given the projection of a virtual object
and their shapes are different from each other, the virtual object’s correct
appearance cannot be seen from an arbitrary viewer’s position. This is
explained in Figure 1, in which projection mapping is simulated virtually by a graphic library OpenGL. In the following, the notation “(a1,2)”
means “(a1) and (a2)”, and “(a,b1)” means “(a1) and (b1)”. In this simulation, a virtual object is a teapot shown in (b1,2,3), and a real object isa cube shown in (c1,2,3). The images (a1,2) are top views to show thepositions of the teapot, the cube, a projector, and a viewer on the horizontal plane. The images (b,c2) are seen from the projector’s position
while the images (b,c1) and (b,c3) are seen from the left and rightviewer’s positions. The images (b1,2,3) are obtained by rendering theteapot from the respective positions; they have the teapot’s correct appearances. The gray colors in (c1,2,3) mean depth values from the respective positions. If the image (b2) is used as a projection image andprojected onto the cube’s surface by the projector, the surface is seenfrom the respective positions as shown in (d1,2,3); the red part on the
cube’s surface in (a2) is given the projection of the teapot. The teapot’s
appearance is correct in (d2), while it is incorrect in (d1,3). Our method
solves the “incorrect appearance” problem by the “two-pass algorithm”[5][35][36], which “correctly distorts” an image with correct appearance

Figure 1. Correct/incorrect appearances of projected images and enhancement by real background.

to obtain an appropriate projection image to project by a projector according to the positional relation between the projector and a viewer.
In (d1,2,3), the gray-colored empty areas on the cube’s surface are notdegrade the viewer’s feeling that the teapot merges into the real world.

given the projection of the teapot. The colored boxes in (e1,2,3) simulate

In order to avoid this problem, our method captures an image of the real

the real background behind the cube seen from the respective positions

background behind a real object by a camera and projects the image ontoin the real world. Then, the actual views seen from the positions becomesuch empty areas. This results in making a viewer not feel the presence(f1,2,3). In addition to the incorrect appearance, the empty areas greatlyof the real object. The view-dependent projection of the teapot and the58The Journal of the Society for Art and Science, Vol. 20, No. 1, pp. 55-71 (2021)
real background with their correct appearances provides the ideal views

Step 2: The viewer composite image (d1) is used as a texture and pro-in (g1,2,3), in which the teapot looks as if it were floating in the air in

jectively mapped on the real object’s surface. The textured surface isfront of the real background.

rendered from the projector’s position to generate a projection image

We need to consider how to obtain the background image seen from(d2), as shown in (a2). The image (d2) consists of (b2) and (c2), which

the viewer’s position. One solution is that the viewer carries a camera toare generated from (b1) and (c1) respectively.

directly use a captured background image. However, this imposes a bur-

Step 3: The projection image (d2) is projected on the real object’s sur-den on the viewer. Practically, the camera should be fixed at a certain

face, which results in showing the view of the virtual object’s correct

position. Then, a background image captured from the camera’s posi-

appearance (d3), as shown in (a3). The view (d3) consists of (b3) andtion needs to be converted into an image seen from the viewer’s position.

(c3). The view (e3) shows the background directly seen by the viewer

An easy solution is to use an RGB-D camera to obtain not only a color

in the real world. Then, the actual view seen by the viewer is (d3’).

image but also a depth image of objects in the background. The depth

The view (c3’) contains only the background.

image is used to obtain the 3D shapes of the background objects, which

The details of these steps are explained below. The main novelty ofare rendered from the viewer’s position to obtain the objective image.

our method is in Step 1-2. The viewer background image is generated

However, the quality of such an image obtained by using a reasonablefrom the camera background image by a background plane defined by

RGB-D camera, such as a Kinect sensor, is not so high because of theappropriate background parameters interpolated on a 3D grid space. In

measurement errors of depth values and the pixel correspondence errors

the following, we use “background” instead of “real background” if it

between a color image and a depth image. Besides, as a crucial problem,

does not mislead a reader.

the depth value that the RGB-D camera can measure is limited to a certain range; for example, the practical depth range of a Kinect sensor is

3.2 Practice environment

limited from 0.5 to 4.0 meters. This means that a background far from

We need to prepare the same practice environment in both of the real

the camera cannot be measured. A traditional solution in computer vi-

space and the virtual space. In the real space, a real object, a projector,

sion to estimate depth values from color images captured by multiple

and a camera are prepared. In the virtual space, an accurate 3D geomet-

RGB cameras can treat a far background. However, it is troublesome

ric model of the real object is made. Then, the registration between theand difficult to set up and calibrate the cameras accurately. Besides, thespaces is done by placing the real object, the projector, and the camera

estimation is not always exact and stable. Therefore, our method usesin the same positions and orientations in both spaces. Their positions andonly a color image captured by a single RGB camera to treat a far back-orientations are fixed during a viewer’s SAR experience.

ground in a large space, even outdoors, easily and stably. Our method

The world coordinate system [𝑥, 𝑦, 𝑧] by the rectangular coordinate

approximates the 3D shape of the background, that is, the 3D shapes ofsystem is given in the practice environment. The coordinates 𝑥 and 𝑦

the background objects by a background plane according to a viewer’s

define the horizontal plane, and the coordinate 𝑧 defines the vertical

position. Then, the captured image is converted into an image seen fromdirection. A viewer usually moves around the real object. Therefore, thea viewer’s position based on homography. The converted backgroundreal object is put on the origin 𝑂 of the world coordinate system, andimage by the approximation needs to be similar to the real backgroundthe polar coordinate system [𝑟, 𝜃, 𝜑], which is defined bydirectly seen by the viewer as exactly as possible. Particularly, in order𝑥 = 𝑟 cos 𝜃 sin 𝜑,(1)

to avoid the viewer’s strange feeling, the converted background pro-𝑦 = 𝑟 sin 𝜃 sin 𝜑,(2)jected on the real object’s surface should match the directly-seen real𝑧 = 𝑟 cos 𝜑,(3)background seamlessly along the real object’s contour.

is used for the viewer’s position.

The algorithm of our method executes the following steps, as shownin Figure 2. Steps 1 and 2 are done in the virtual space of a computer

3.3 Generation of viewer composite image

while Step 3 is done in the real space, that is, the real world. Step 2 isA viewer composite image is generated for an arbitrary viewer’s po-

known as the two-pass algorithm [5][35][36]. These are executed in realsition by Step 1, which consists of the following three sub-steps.

time for the sake of the movement of a viewer and background objects.
Step 1: A viewer composite image seen from a viewer’s position is gen-3.3.1 Viewer virtual object image

erated by the next sub-steps, as shown in Figure 2 (a1).

A viewer virtual object image is generated in Step 1-1. In the virtual

Step 1-1: A virtual object is rendered from the viewer’s position to gen-space, a 3D geometric model of a virtual object is made. It is scaled anderate a viewer virtual object image (b1).

positioned so as to fit inside the real object. Then, it is rendered from theStep 1-2: A background image captured by the camera, called camera

viewer’s position to generate a viewer virtual object image.

background image (c1*), is converted into a background image seenfrom the viewer’s position, called viewer background image (c1).
Step 1-3: The viewer virtual object image (b1) and the viewer background image (c1) are combined into a viewer composite image (d1).
59The Journal of the Society for Art and Science, Vol. 20, No. 1, pp. 55-71 (2021)(a1) Step 1(a2) Step 2(a3) Step 3

Step 1

Step 2

Real space

Step 3

Virtual space

Real space

Virtualobject

(b1)

(b2)

(b3)

(e3)

(c1*)

(c1)

(c2)

(c3)

(c3’)

Camera’s view(d1)
Viewer’s view

Background

Composite(d2)
Projector’s view
Figure 2. Our algorithm.

3.3.2 Viewer background imageℎ11
𝐻 = [ℎ21ℎ31

A viewer background image is generated in Step 1-2. It is obtained
from a camera background image by using a background plane to approximate the actual shape of a background. Our method uses a homog-

(d3)

(d3’)
Viewer’s view

ℎ12
ℎ22
ℎ32

ℎ13
ℎ23 ],ℎ33(5)where 𝜆 ≠ 0 is a constant. The symbol 𝑇 means a transposed matrix.

raphy matrix.

With respect to the camera 𝐶𝑚 , 𝑚 = 1, 2, the intrinsic parameter

When an image on a plane 𝑃 is seen from two cameras, the images

matrix 𝐴𝑚 and the extrinsic parameter matrix [𝑅𝑚 𝒕𝑚 ] defined byof these cameras are directly related to each other by a 3 × 3 homog-

the rotation matrix 𝑅𝑚 and the translation vector 𝒕𝑚 are given byraphy matrix. The 2D pixel coordinates 𝑼1 = [𝑢1 , 𝑣1 ]𝑇 of a camera
𝐶1 and 𝑼2 = [𝑢2 , 𝑣2 ]𝑇 of a camera 𝐶2 have the relationship̃ 1 = 𝜆 [𝑼1 ] = 𝐻 [𝑼2 ] = 𝐻𝑼
̃2
𝜆𝑼
1
1

(4)

by a homography matrix

60

𝑓𝑥𝑚
𝐴𝑚 = [ 0
0

𝑠𝑚
𝑓𝑦𝑚
0

𝑚
𝑟11
𝑚
𝑅𝑚 = [𝑟21
𝑚
𝑟31

𝑚
𝑟12
𝑚
𝑟22
𝑚
𝑟32

𝑐𝑥𝑚
𝑐𝑦𝑚 ],
1
𝑚
𝑟13
𝑚
𝑟23
],
𝑚
𝑟33

(6)

(7)

The Journal of the Society for Art and Science, Vol. 20, No. 1, pp. 55-71 (2021)
𝐷𝑃 from the point 𝑄0 to the plane 𝑃 as follows:𝑡𝑥𝑚
𝒕𝑚 = [𝑡𝑦𝑚 ].
𝑡𝑧𝑚

𝑑 = −𝑵𝑇 𝑿0 − 𝐷𝑃 ,

(8)

(17)

where the vector 𝑵 has the same direction as the direction from theIn the matrix 𝐴𝑚 , the parameters 𝑓𝑥𝑚 and 𝑓𝑦𝑚 are the focal lengths

point 𝑄0 to the plane 𝑃. By using the origin 𝑂 with world coordi-

in terms of pixels, 𝑐𝑥𝑚 and 𝑐𝑦𝑚 are the coordinates of the principalnates 𝑶 = [0, 0, 0]𝑇 as the point 𝑄0 , we obtainpoint, and 𝑠𝑚 is the skew coefficient. The 2D pixel coordinates 𝑼𝑚

𝑑 = −𝐷𝑃 .

and the 3D world coordinates 𝑿 = [𝑥, 𝑦, 𝑧]𝑇 have the relationship̃ 𝑚 = 𝜆𝑚 [𝑼𝑚 ] = 𝐴𝑚 [𝑅𝑚 𝒕𝑚 ] [𝑿] = 𝐴𝑚 [𝑅𝑚 𝒕𝑚 ] 𝑿
̃,
𝜆𝑚 𝑼
1
1

to (3) as follows:(9)
where 𝜆𝑚 ≠ 0 is a constant. Besides, the plane 𝑃 is defined by
𝑵𝑇 𝑿 + 𝑑 = 𝑛𝑥 𝑥 + 𝑛𝑦 𝑦 + 𝑛𝑧 𝑧 + 𝑑 = 0,

(10)

𝑇

(19)
(20)

𝑛𝑧 = cos 𝜑𝑃 .

(21)

the three parameters 𝐷𝑃 , 𝜃𝑃 , and 𝜑𝑃 to define the plane 𝑃 cannot

is a constant. Then, the homography matrix 𝐻 is represented by
𝐻 = 𝐴1𝑛𝑥 = cos 𝜃𝑃 sin 𝜑𝑃 ,𝑛𝑦 = sin 𝜃𝑃 sin 𝜑𝑃 ,Among the above-described elements to define the matrix 𝐻, only

where 𝑵 = [𝑛𝑥 , 𝑛𝑦 , 𝑛𝑧 ] , |𝑵| = 1, is a unit normal vector, and 𝑑
{(𝑵𝑇

(18)

The world coordinates of the vector 𝑵 can be defined by using two
coordinates 𝜃𝑃 and 𝜑𝑃 and setting 𝑟 = |𝑵| = 1 in Equations (1)𝑅2−1 𝒕2 − 𝑑)𝑅1 + (𝒕1 − 𝑅1 𝑅2−1 𝒕2 )𝑵𝑇 }𝑅2−1 𝐴−1
2 .

be determined automatically. We call them background parameters. We(11)adjust them manually to make the plane 𝑃 approximate the shape of abackground optimally such that a resulting viewer background image

The derivation of Equation (11) is described in Appendix A.
If the angles of view 𝛼𝑥𝑚 and 𝛼𝑦𝑚 , and the numbers of pixels

becomes as similar as possible to the actual background view seen from

𝑀𝑥𝑚 and 𝑀𝑦𝑚 of the camera 𝐶𝑚 are known, then the parameters

the viewer’s position. However, it is not practically easy to obtain the

𝑓𝑥𝑚 , 𝑓𝑦𝑚 , 𝑐𝑥𝑚 , and 𝑐𝑦𝑚 are obtained by

optimal plane by using the above three parameters. We improve them

𝑓𝑥𝑚 = 𝑀𝑥𝑚 ⁄{2 tan(𝛼𝑥𝑚 ⁄2)},
𝑓𝑦𝑚 = 𝑀𝑦𝑚 ⁄{2 tan(𝛼𝑦𝑚 ⁄2)},(12)

𝑐𝑥𝑚 = 𝑀𝑥𝑚 ⁄2,
𝑐𝑦𝑚 = 𝑀𝑦𝑚 ⁄2.

(14)

into other parameters in Section 3.6.1.

(13)

3.3.3 Viewer composite image
In Step 1-3, a viewer composite image is obtained by overlaying the(15)

viewer virtual object image on the viewer background image. The vir-

Besides, it is often assumed that𝑠𝑚 = 0.

tual object can be made semitransparent by alpha blending so that the(16)

background can be seen through the virtual object.

We consider the camera 𝐶1 as a camera to capture a background, thecamera 𝐶2 as a viewer, and the plane 𝑃 as a background plane to ap-

3.4 Generation of projection image

proximate the shape of the background. Then, a viewer background image is obtained from a camera background image by determining the

In Step 2, a projection image is generated from the viewer composite

matrix 𝐻 in Equation (11) and using the correspondence between theimage by using the two-pass algorithm [5][35][36]. First, the viewer

coordinates 𝑼1 and 𝑼2 in Equation (4). This 2D computation using

composite image is used as a texture and projectively mapped on thethe 3 × 3 matrix 𝐻 is so efficient compared to the usual 3D compu-

surface of the geometric model of the real object from the viewer’s po-tation using a 4 × 4 matrix by considering the two image planes andsition. Then, the textured surface is rendered from the projector’s posi-

the plane 𝑃 in the 3D space.

tion to generate a projection image. The resulting projection image is

The matrix 𝐻 in Equation (11) consists of the intrinsic and extrinsic“correctly distorted” so that the virtual object and the background in thematrices of the cameras 𝐶𝑚 , 𝑚 = 1, 2 , and the parameters of theoriginal viewer composite image can be seen with their “correct appear-

plane 𝑃. These are obtained as follows. The camera 𝐶1 is a camera toances” by the viewer in Step 3. If a graphic library or tool for projective

capture a background. The matrix 𝐴1 can be obtained by camera cali-

texture mapping is available, such as OpenGL, the above can be easily

bration; it can be also obtained by Equations (12) to (16) although it isrealized.

influenced by the errors of 𝛼𝑥𝑚 and 𝛼𝑦𝑚 in the specification and the

3.5 Projection of projection image

assumption by Equation (16). The matrix 𝑅1 and the vector 𝒕1 are
obtained from the position and viewing direction of the camera 𝐶1 . The

In Step 3, the projection image obtained in Step 2 is projected on thecamera 𝐶2 is a viewer. The matrix 𝐴2 can be given arbitrarily andsurface of the real object by the projector in the real space. The viewer

Step 2 needs to use the same matrix. The matrix 𝑅2 and the vector 𝒕2

sees the virtual object and the background correctly.

are obtained from the viewer’s position and viewing direction, which

3.6 Adjustment of background parameters

are obtained by tracking the viewer in real time. Finally, with respect tothe plane 𝑃, the constant 𝑑 is obtained from the vector 𝑵, the world

The background parameters to define a background plane need to becoordinates 𝑿0 of an arbitrary reference point 𝑄0 , and the distanceadjusted as easily as possible to approximate a real background’s shape
optimally. We present a practical adjustment method as follows.
61

The Journal of the Society for Art and Science, Vol. 20, No. 1, pp. 55-71 (2021)

3.6.1 Practical background parameters
The three background parameters 𝐷𝑃 , 𝜃𝑃 , and 𝜑𝑃 to define abackground plane 𝑃 are presented in Section 3.3.2. The parameter 𝐷𝑃
is the distance from the origin 𝑂 to the plane 𝑃. The parameters 𝜃𝑃and 𝜑𝑃 define the unit normal vector 𝑵 of the plane 𝑃. The real object is put on the origin 𝑂. Practically, the plane 𝑃 needs to be adjusted
to obtain an optimal viewer background image by changing the parameters and observing the projected image on the real object’s surface. Thisadjustment is not done efficiently if the parameters 𝐷𝑃 , 𝜃𝑃 , and 𝜑𝑃
are used directly. In the following, the “viewer” means a person to prepare a projection mapping event and adjust the plane 𝑃 in advance. In(a)Figure 3 (a), the viewer adjusts the plane 𝑃 by seeing the real object
and the real background toward the origin 𝑂 from the viewer’s eye position 𝑄𝑣 ; the viewer’s viewing direction is represented by the red longarrow. In the following, we mean “viewer’s eye position” by “viewer’s
position”. The planes 𝑃1 and 𝑃2 have the unit normal vectors 𝑵1
and 𝑵2 defined by the parameter sets [𝜃𝑃1 , 𝜑𝑃1 ] and [𝜃𝑃2 , 𝜑𝑃2 ] .
The two planes have the same distance 𝐷𝑃 from the origin 𝑂 in the
directions of their normal vectors. However, the distances 𝐷1 and 𝐷2
in the viewer’s viewing direction are different. That is, when the viewer
changes 𝜃𝑃 and 𝜑𝑃 to adjust the normal vector, the distance in theviewing direction also changes against the viewer’s intention. This
causes a serious difficulty for the viewer to obtain an optimal plane 𝑃.

(b)
Figure 3. Background parameters.

Therefore, we present another practical way, as shown in Figure 3 (b).
𝑇First, the base normal vector 𝑵0 = [𝑛𝑥0 , 𝑛𝑦0 , 𝑛𝑧0 ] , |𝑵0 | = 1 , isobtained from the coordinates 𝑹𝑣 = [𝑟𝑣 , 𝜃𝑣 , 𝜑𝑣 ]𝑇 of the viewer’s po-

The two parameter sets [𝐷𝑃 , 𝜃𝑃 , 𝜑𝑃 ] and [𝐿𝑃 , ∆𝜃𝑃 , ∆𝜑𝑃 ] are related by Equations (27), (28), and (30). The set [𝐿𝑃 , ∆𝜃𝑃 , ∆𝜑𝑃 ] ena-

sition 𝑄𝑣 :𝜃𝑃0 = 𝜃𝑣 + 180°,(22)

Therefore, we use the parameters 𝐿𝑃 , ∆𝜃𝑃 , and ∆𝜑𝑃 as background

𝜑𝑃0 = 180° − 𝜑𝑣 ,(23)

parameters practically. These parameters determine an arbitrary plane

𝑛𝑥0 = cos 𝜃𝑃0 sin 𝜑𝑃0 ,𝑛𝑦0 = sin 𝜃𝑃0 sin 𝜑𝑃0 ,

(24)
(25)

𝑃 for a viewer’s position 𝑄𝑣 with coordinates 𝜃𝑣 and 𝜑𝑣 . The unit

𝑛𝑧0 = cos 𝜑𝑃0 .

(26)(28). The constant 𝑑 is determined by Equation (29), in which the vec-

Next, the base plane 𝑃0 is defined such that it has the normal vector

tor 𝑵0 is determined by Equations (22) to (26). The plane 𝑃 always

defined; its direction is the same as the viewer’s viewing direction. It isbles the viewer to adjust the plane 𝑃 more easily than [𝐷𝑃 , 𝜃𝑃 , 𝜑𝑃 ].

normal vector 𝑵 is determined by Equations (19) to (21), (27), andhas the base point 𝑄𝑃 , which is seen by the viewer at the center of𝑵0 and a distance 𝐿𝑃 from the origin 𝑂 . An arbitrary unit normal

his/her visual field. The point 𝑄𝑃 is away from the viewer by the dis-

vector 𝑵 defined by parameters 𝜃𝑃 , 𝜑𝑃 , and Equations (19) to (21)tance 𝐿𝑃 + 𝑟𝑣 . For fixed values of ∆𝜃𝑃 and ∆𝜑𝑃 , the change of theis obtained based on the vector 𝑵0 using difference parameters ∆𝜃𝑃distance 𝐿𝑃 moves the plane 𝑃 forward and backward in the viewing

and ∆𝜑𝑃 by
𝜃𝑃 = 𝜃𝑃0 + ∆𝜃𝑃 = 𝜃𝑣 + 180° + ∆𝜃𝑃 ,(27)

direction with the normal vector 𝑵 unchanged. For a fixed value of

𝜑𝑃 = 𝜑𝑃0 + ∆𝜑𝑃 = 180° − 𝜑𝑣 + ∆𝜑𝑃 .

(28)

𝐿𝑃 , the change of the angles ∆𝜃𝑃 and ∆𝜑𝑃 causes the 3D rotation ofThen, the viewer can obtain an objective plane 𝑃 based on the plane

the plane 𝑃 around the point 𝑄𝑃 , which is fixed as a rotational center,𝑃0 by changing the three parameters 𝐿𝑃 , ∆𝜃𝑃 , and ∆𝜑𝑃 . The baseby moving the normal vector 𝑵 apart from the base normal vector 𝑵0 .
The above way enables the viewer to obtain an optimal plane 𝑃 easily,

point 𝑄𝑃 is the intersection of the plane 𝑃0 and the viewing direction;

intuitively, and efficiently.

it has the world coordinates 𝑿𝑃 = 𝐿𝑃 𝑵0 . The point 𝑄𝑃 is on an arbitrary plane 𝑃 defined by 𝐿𝑃 , ∆𝜃𝑃 , and ∆𝜑𝑃 . This means that the co-3.6.2 Interpolation of background parameters

ordinates 𝑿𝑃 satisfy Equation (10), which gives𝑑 = −𝑵𝑇 𝑿

𝑇
𝑝 = −𝐿𝑃 𝑵 𝑵0 .

An optimal background view projected on the real object’s surface by(29)

using a background plane 𝑃 should be as similar as possible to the ac-

Equations (18) and (29) give the relation between 𝐷𝑃 and 𝐿𝑃 :𝐷𝑃 = 𝐿𝑃 𝑵𝑇 𝑵0 .

tual background view directly seen by a viewer. Such a plane 𝑃 de-

(30)

62

The Journal of the Society for Art and Science, Vol. 20, No. 1, pp. 55-71 (2021)
pends on the viewer’s position. Our method provides optimal back-space. The tracking sensor is fixed near the real object and directed to-

ground parameters 𝐿𝑃 , ∆𝜃𝑃 , and ∆𝜑𝑃 according to the viewer’s po-

ward the area in which the viewer can move. Consequently, the tracking

sition. They are separated as follows:

sensor and the camera to capture a real background are fixed near the

𝐿𝑃 = 𝐿𝐺𝑃 + 𝐿𝐿𝑃 ,∆𝜃𝑃 = ∆𝜃𝑃𝐺 + ∆𝜃𝑃𝐿 ,∆𝜑𝑃 = ∆𝜑𝑃𝐺 + ∆𝜑𝑃𝐿 .

real object and directed toward the sides opposite to each other.

(31)
(32)

3.7.2 Generation of viewer virtual object image(33)

The global parameters 𝐿𝐺𝑃 , ∆𝜃𝑃𝐺 , and ∆𝜑𝑃𝐺 are common for every

Our system has two options to obtain a viewer virtual object image inviewer’s position to adjust the plane 𝑃 roughly. The local parameters

Step 1-1. The first option renders the 3D geometric model of a virtual

𝐿𝐿𝑃 , ∆𝜃𝑃𝐿 , and ∆𝜑𝑃𝐿 change according to the viewer’s position to adjust

object in real time during a viewer’s SAR experience. This option can

the plane 𝑃 in detail. For an arbitrary viewer’s position with coordi-

generate an image for an arbitrary position of a moving viewer with lessnates 𝑹𝑣 = [𝑟𝑣 , 𝜃𝑣 , 𝜑𝑣 ]𝑇 , the following functions of 𝑹𝑣 provide thememory than the second one. Besides, a desired rendering and shading

local parameters:
𝐿𝐿𝑃 = 𝐹𝐿𝐿𝑃 (𝑹𝑣 ) = 𝐹𝐿𝐿𝑃 (𝑟𝑣 , 𝜃𝑣 , 𝜑𝑣 ),∆𝜃𝑃𝐿 = 𝐹∆𝜃𝐿 (𝑹𝑣 ) = 𝐹∆𝜃𝐿 (𝑟𝑣 , 𝜃𝑣 , 𝜑𝑣 ),algorithm can be used. However, the algorithm needs to be implemented(34)

so as to work efficiently in real time in our system, and the quality of the(35)

image is restricted by the real-time processing time. The second option∆𝜑𝑃𝐿 = 𝐹∆𝜑𝑃𝐿 (𝑹𝑣 ) = 𝐹∆𝜑𝑃𝐿 (𝑟𝑣 , 𝜃𝑣 , 𝜑𝑣 ).

(36)

renders the model in advance for predetermined 𝑀𝑟 × 𝑀𝜃 × 𝑀𝜑 dis-𝑃

𝑃

These functions are defined in the ranges

crete viewer’s positions and saves the rendered images to a storage de-𝑟𝑚𝑖𝑛 ≤ 𝑟𝑣 ≤ 𝑟𝑚𝑎𝑥 ,(37)

vice such as a hard disk. In a viewer’s experience, all the saved images

𝜃𝑚𝑖𝑛 ≤ 𝜃𝑣 < 𝜃𝑚𝑎𝑥 , 𝜃𝑚𝑖𝑛 = 0, 𝜃𝑚𝑎𝑥 = 360,(38)

are stored on a memory, and the image for the discrete position nearest

𝜑𝑚𝑖𝑛 ≤ 𝜑𝑣 ≤ 𝜑𝑚𝑎𝑥 ,(39)

to the actual position of a moving viewer is used. This option can use

by using grid points 𝐺[𝑖,𝑗,𝑘] with coordinates [𝑟𝑖 , 𝜃𝑗 , 𝜑𝑘 ], 0 ≤ 𝑖 <high-quality images generated by time-consuming advanced rendering

𝑁𝑟 , 0 ≤ 𝑗 < 𝑁𝜃 , 0 ≤ 𝑘 < 𝑁𝜑 , given by

and shading algorithms and graphic tools. However, it needs large stor-

𝑟𝑚𝑖𝑛 (= 𝑟𝑚𝑎𝑥 )𝑟𝑖 = {

𝑟𝑚𝑎𝑥 − 𝑟𝑚𝑖𝑛
𝑟𝑚𝑖𝑛 + ∆𝑟 ∙ 𝑖, ∆𝑟 =𝑁𝑟 − 1age/memory; in particular, an amination of a virtual object needs huge

(𝑁𝑟 = 1),

storage/memory to treat all frame images for all discrete positions.

(𝑁𝑟 ≥ 2),

4. Experiment(40)
𝜃𝑗 = ∆𝜃 ∙ 𝑗,
𝜑𝑘 = {(41)∆𝜃 = 360⁄𝑁𝜃 ,𝜑𝑚𝑖𝑛 (= 𝜑𝑚𝑎𝑥 )(𝑁𝜑 = 1),

𝜑𝑚𝑎𝑥 − 𝜑𝑚𝑖𝑛
𝜑𝑚𝑖𝑛 + ∆𝜑 ∙ 𝑘, ∆𝜑 =𝑁𝜑 − 1(𝑁𝜑 ≥ 2).

We made some experiments in virtual and real environments. The
demo videos of the experiments can be accessed on the journal website.
Besides, the additional experiments and demo videos can be accessedon the following website: http://www-cg.cis.iwateu.ac.jp/~fujimoto/demo/projmap_3d_back-(42)
Each grid point is given local parameters 𝐿𝐿𝑃 [𝑖, 𝑗, 𝑘], ∆𝜃𝑃𝐿 [𝑖, 𝑗, 𝑘], and∆𝜑𝑃𝐿 [𝑖, 𝑗, 𝑘], which are interpolated linearly to define the functions. In

ASv20n1/projmap_3d_back.html

order to obtain an optimal plane 𝑃 for an arbitrary viewer’s position,

4.1 Experiment in virtual environment

optimal global and local parameters 𝐿𝐺𝑃 , ∆𝜃𝑃𝐺 , ∆𝜑𝑃𝐺 , 𝐿𝐿𝑃 [𝑖, 𝑗, 𝑘] ,∆𝜃𝑃𝐿 [𝑖, 𝑗, 𝑘], and ∆𝜑𝑃𝐿 [𝑖, 𝑗, 𝑘] need to be given in advance.

by OpenGL. It is shown in Figure 4. The top view (a) shows the hori-

Our current system provides an interface to give the global and local

zontal 𝑥𝑦 plane, and the side view (b) shows the 𝑥𝑧 plane. A cube isparameters by trial and error manually. In the real space, the interface

a real object, which is fixed at the origin 𝑂; it is denoted by the violet

enables us to stand at a position near each grid point, check a backgroundsquare. A viewer sees it at a position 𝑄𝑣 with coordinates 𝑹𝑣 =view projected on the real object’s surface, and adjust the background[𝑟𝑣 , 𝜃𝑣 , 𝜑𝑣 ]𝑇 . The three small round red dots denote typical viewer’s

plane by changing the parameters interactively to obtain an optimal

positions and the long red arrows denote their viewing directions; inbackground view in real time. It is an important future work to develop

each of (a) and (b), the middle dot denotes the viewer’s initial position

an efficient way to determine the optimal parameters automatically or𝑖𝑛𝑖 𝑖𝑛𝑖𝑖𝑛𝑖
𝑄𝑣𝑖𝑛𝑖 with 𝑹𝑖𝑛𝑖𝑣 = [𝑟𝑣 , 𝜃𝑣 , 𝜑𝑣 ] . A projector is fixed at a posi-semi-automatically.

tion 𝑄𝑝𝑟 with 𝑹𝑝𝑟 = [𝑟𝑝𝑟 , 𝜃𝑝𝑟 , 𝜑𝑝𝑟 ] ; it is directed toward the cube.

We firstly made an experiment in a virtual environment constructed𝑇

𝑇

A camera is fixed at a position 𝑄𝑐 with 𝑹𝑐 = [𝑟𝑐 , 𝜃𝑐 , 𝜑𝑐 ]𝑇 ; it is di-

3.7 Practicerected toward the side opposite to the cube. The green and blue dots de-

The following are necessary for the practice of our method.

note the positions 𝑄𝑝𝑟 and 𝑄𝑐 , and the green and blue long arrows
denote their viewing directions. The viewer, the projector, and the cam-

3.7.1 Tracking of viewer

era are given the values in Table 1. The height of the floor is 𝑧 = −1.2.

We use a sensor to track a moving viewer in the real space in real time

In this experiment, the viewer’s coordinate 𝜑𝑣 moves from 80 toto obtain his/her position, which is used in Steps 1 and 2 in the virtual

100 degrees; the coordinate 𝑧 relative to the floor’s height moves
63

The Journal of the Society for Art and Science, Vol. 20, No. 1, pp. 55-71 (2021)

(c)

(a)

(b)
Figure 4. Virtual environment.

Viewer: 𝑄𝑣𝑖𝑛𝑖
Projector: 𝑄𝑝𝑟
Camera: 𝑄𝑐

Table 1. Values for experiment in virtual environment.
Position
Position
Gaze point
Angles of Numbers of[𝑟, 𝜃, 𝜑][𝑥, 𝑦, 𝑧][𝑥, 𝑦, 𝑧]
view
pixels(meters)(meters)(𝑟: meters)[𝛼𝑥 , 𝛼𝑦 ][𝑀𝑥 , 𝑀𝑦 ](𝜃, 𝜑: degrees)(degrees)
3.0, 180.0, 90.0 −3.0, 0.0, 0.0 0.0, 0.0, 0.0 20.0, 15.0 640, 480
4.2, 180.0, 74.0 −4.0, 0.0, 1.2 0.0, 0.0, 0.0 38.0, 23.0 640, 400
0.8, 0.0, 140.0 0.5, 0.0, −0.6 10.0, 0.0, 0.0 57.0, 43.0 640, 480

roughly from 1.7 to 0.7 meter for 𝑟𝑣 = 3.0 meters. This range ofboxes are rotated randomly. Their centers are arranged regularly with𝑧 simulates the height of an actual viewer’s eyes. The size of the cube

random displacements within ±10 % of the interval between neigh-is 0.43 meters. The cube’s top and bottom faces are parallel to the floor,

boring boxes horizontally and vertically on the planes and within ±0.1

while its four vertical edges are put toward ±𝑥 and ±𝑦 directions.

meter perpendicularly to the planes. The size of each box is 0.553 me-

The angles of view [𝛼𝑥 , 𝛼𝑦 ] and the numbers of pixels [𝑀𝑥 , 𝑀𝑦 ] ofters. We call the planes “box planes”. The distribution of the boxes close

the projector and the camera are the same as those of our actual projector

to the two box planes needs to be approximated by a single backgroundand camera used in the experiments in Section 4.2, while those of theplane 𝑃. Each image in Figure 5 has two areas. The dark central area isviewer were selected to appropriately evaluate the performance of ourthe cube’s surface onto which the teapot and the background are pro-method.
The result of the experiment is shown in Figure 5. The images arejected by the projector; the cube’s top face is bright and its bottom face“augmented” views seen from twenty-five viewer’s positions given bybackground directly observed. Our method aims to remove the presence𝜃𝑣 = 160, 170, 180, 190, 200 and 𝜑𝑣 = 80, 85, 90, 95, 100 atof the cube by projecting the background on its surface. Thus, the back-a constant distance 𝑟𝑣 = 𝑟𝑣𝑖𝑛𝑖 = 3.0. An orange teapot is a virtual ob-

ground plane 𝑃 should be adjusted such that the projected backgroundject. The colored boxes simulate objects in a real background. The boxes

matches the directly-observed background along the boundary of theare arranged close to two planes vertical to the 𝑥𝑦 plane; the browncube’s contour as seamlessly as possible. The background parameter in-lines in Figure 4 (c) represent the planes. The two planes intersect per-terpolation provides the “boundary match” for an arbitrary viewer’s po-

pendicularly, and they are 10 meters away from the origin 𝑂 . The

sition.

is black due to the position of the projector. The remaining area is the64The Journal of the Society for Art and Science, Vol. 20, No. 1, pp. 55-71 (2021)

Table 2. Global background parameters for Figure 5.
Figure 5
𝐿𝐺𝑃
∆𝜃𝑃𝐺
∆𝜑𝑃𝐺
(a)
12.8
0.0
0.0
(b)
10.0
0.0
0.0

Table 3. Definition of grid points for Figure 5 (b).
Minimum
coordinate

Maximum
coordinate

Number ofpointsInterval𝑟𝑖𝑟𝑚𝑖𝑛 = 2.5

𝑟𝑚𝑎𝑥 = 3.5

𝑁𝑟 = 3∆𝑟 = 0.5𝜃𝑗𝜃𝑚𝑖𝑛 = 0𝜃𝑚𝑎𝑥 = 360

𝑁𝜃 = 18∆𝜃 = 20

𝜑𝑚𝑎𝑥 = 100

𝑁𝜑 = 3∆𝜑 = 10𝜑𝑘 𝜑𝑚𝑖𝑛 = 80

Table 4. Local background parameters given to grid points for Figure 5 (b). Other grid points are given [𝐿𝐿𝑃 , ∆𝜃𝑃𝐿 , ∆𝜑𝑃𝐿 ] = [0.0, 0.0, 0.0].
𝑟𝑖 = 3.0
𝜃𝑗
160
180
200
𝜑𝑘
𝐿𝐿𝑃
∆𝜃𝑃𝐿
∆𝜑𝑃𝐿
𝐿𝐿𝑃
∆𝜃𝑃𝐿
∆𝜑𝑃𝐿
𝐿𝐿𝑃
∆𝜃𝑃𝐿
∆𝜑𝑃𝐿
1.203993
−25.0
−10.0
2.997460
0.0−10.0
1.203993
25.0
−10.0
80
1.033779
−25.0
0.0
2.8
0.0
0.0
1.033779
25.0
0.0
90
1.203993−25.0
10.0
2.997460
0.0
10.0
1.203993
25.0
10.0
100
The images in Figure 5 (a) were obtained when the interpolation was

two box planes by the single plane 𝑃. The projected background in each

not used. Local parameters were not used, and the global parameters inof the remaining six images for 𝜃𝑣 = 160 and 200 has the boxes,

Table 2 were carefully selected so as to provide the best boundary match

all of which are close to only one box plane. In this case, the single plane

along the cube’s contour for the viewer’s initial position 𝑄𝑣𝑖𝑛𝑖 . The re-

𝑃 worked quite well. On the other hand, the images other than thesulting plane 𝑃 was parallel to the 𝑦𝑧 plane and 12.8 meters away

above nine images were obtained from interpolated parameters. Thesefrom the origin 𝑂. The central image for 𝑹𝑖𝑛𝑖𝑣 = [3.0, 180, 90] has

images also have appropriate boundary matches although there are also

an appropriate boundary match although there are some noticeable

some gaps. The two images for 𝜃𝑣 = 180 have some noticeable gaps

“gaps” between the projected background and the directly-observed

and the four images for 𝜃𝑣 = 160 and 200 have less gaps by the

background; the gaps were caused by the approximation of the two box

same reason as described above. The ten images for 𝜃𝑣 = 170 andplanes by the single plane 𝑃 as well as the approximation of the 3D

190 have noticeable gaps although the boxes in their projected back-shapes of the boxes by the plane 𝑃. However, the other images havegrounds have the same situations as those for 𝜃𝑣 = 160 and 200.

worse boundary matches with serious gaps due to the same parameters

The parameters for 𝜃𝑣 = 170 are given by interpolating the parame-

selected only for 𝑄𝑣𝑖𝑛𝑖 .

ters for 𝜃𝑣 = 𝜃𝑗 = 160 and 180. This means that the plane 𝑃170 is

The images in Figure 5 (b) were obtained when the interpolation was

the intermediate between the two planes 𝑃160 and 𝑃180 , where 𝑃𝜃𝑣

used. The grid points 𝐺[𝑖,𝑗,𝑘] were given by Table 3, and the global andmeans a plane 𝑃 for 𝜃𝑣 . Thus, the noticeable gaps for 𝜃𝑣 = 170

local parameters in Tables 2 and 4 were selected so as to provide the best

were caused by the plane 𝑃170 that does not coincide with the box

boundary matches for the viewer at the respective grid points. The gridplane. The gaps for 𝜃𝑣 = 190 have the same cause. Compared to Fig-

point with 𝑹𝑖𝑛𝑖was given the same parameters in total as those given𝑣

ure 5 (a), the boundary matches are greatly improved in Figure 5 (b) byin the “without-interpolation” case. Consequently, the central images inthe background parameter interpolation although there are some gaps,(a) and (b) are the same by the same plane 𝑃. The two grid points with[𝑟𝑖 , 𝜃𝑗 , 𝜑𝑘 ] = [3.0, 180, 80] and [3.0, 180, 100] were given thepart of which are noticeable.

parameters to make the same plane 𝑃 remain. The three grid points
with 𝜃𝑗 = 160 in Table 4 were given the parameters to make the plane

to appropriately eliminate undesired empty areas on a real object’s sur-

𝑃 coincide with the right box plane in y ≤ 0 seen from the origin 𝑂;

justed easily, intuitively, and efficiently by practical background param-a viewer at 𝜃𝑣 = 160 sees mainly the boxes close to the right plane.
In the same way, the parameters given to the three grid points with 𝜃𝑗 =eters, which are interpolated to make the boundary match between a

200 make the plane 𝑃 coincide with the left box plane in y ≥ 0. The

as possible for an arbitrary viewer’s position. This ability achieves thenine images with black frames were obtained directly from the parame-

effective view-dependent projection to make a viewer not feel the pres-ters given to the grid points. These images have appropriate boundary

ence of a real object but feel a virtual object merging into the real world.

The above result shows that our method has the fundamental ability
face by projecting a real background. A background plane can be ad-

projected background and a directly-observed background as seamlessmatches although there are some gaps. The three images for 𝜃𝑣 = 180

4.2 Experiment in real environment

have more noticeable gaps than the others have. The projected background in each of the three images has the boxes, some of which are

We secondly made some experiments in a real environment. We usedclose to the right box plane and others of which are close to the left one.

a white cube made from styrofoam as a real object; its size is 0.43 me-

The noticeable gaps were caused by the imperfect approximation for theters. We used an off-the-shelf projector, RICOH IPSiO PJ WX5150. We
65

The Journal of the Society for Art and Science, Vol. 20, No. 1, pp. 55-71 (2021)

80

85

90

95

100
𝜑𝑣
𝜃𝑣

160

(b) 𝜃𝑣

160

(a)

170

180

190

200

80

85

90

95

100
𝜑𝑣
170
180
190
Figure 5. Augmented views for different viewer’s positions in virtual environment. 𝑟𝑣 = 𝑟𝑣𝑖𝑛𝑖 = 3.0.
(a) Background parameters are not interpolated. (b) Background parameters are interpolated.
66

200

The Journal of the Society for Art and Science, Vol. 20, No. 1, pp. 55-71 (2021)

Figure 7. Augmented views in real environment with near planar background.
Figure 6. Virtual object.
used two Kinect sensors. One was used as a tracking sensor to track aviewer. Another was used as a camera to capture a real background. This
Kinect sensor captured only color images of the background and did not
capture depth images. We used it instead of a usual RGB camera because it has a wide angle of view to capture the background widely. We
used old-type Kinect sensors, Kinect for Windows, because we needed
to connect them to one personal computer; more than one new-type Kinect sensor cannot be connected to one personal computer. The two Ki-

Figure 8. Augmented views in real environment with near non-planarbackground.

nect sensors were fixed near the cube and directed toward the sides opposite to each other. The angles of view and the numbers of pixels of the
projector and the Kinect sensors are shown in Table 1. We used a per-Figure 8 shows some augmented views obtained in the second exper-

sonal computer with OS: Windows 10, Chipset: Intel(R) Z370 Express,

iment, in which a rack containing some boxes was put as a background

CPU: Intel(R) Core i7-8700K (3.7-4.7GHz), GPU: NVIDIA(R) Ge-

object in front of the wall used in the first experiment. The correct ap-

Force GTX 1060 6GB GDDR5, Mem: 32GB, SSD: 480GB, HDD:

pearance and the appropriate boundary match were also obtained. Some

2TB. We used the 3D geometric model of a virtual object shown in Fig-

gaps between the projected background and the directly-observed back-ure 6. Viewer virtual object images were obtained by the second option

ground were caused by the difficulty of the approximation for the non-in Section 3.7.2; the model was rendered in advance for 𝑀𝑟 × 𝑀𝜃 ×
𝑀𝜑 = 1 × 360 × 1 discrete viewer’s positions, that is, for every one

planar background by the background plane.

degree around the model. We gave global background parameters andment, in which the background was far from the cube and the back-local parameters for grid points manually in advance. All the projection

ground objects had complicated shapes. The black jaggy-shaped wall

processes of our method were executed in real time during a viewer’s

was about 10 meters and the white wall behind it was about 15 me-

SAR experience.

ters away from the cube. There were many background objects in front

Figure 9 shows some augmented views obtained in the third experi-

Figure 7 shows augmented views seen from some viewer’s positionsof the walls, such as desks and chairs. Figure 10 shows some projectionin the first experiment, in which the background was a planar wall withimages of the virtual object. Our method worked well to treat such a far

three figures near the cube. The views show that our method worked

complicated-shaped background in a large space. The correct appear-

well. The correct appearance of the virtual object was always seen fromance and the appropriate boundary match for the complicated back-different viewer’s positions. The appropriate boundary match between

ground were obtained even if the viewer moved within a wide angle

the projected background and the directly-observed background was

range around the cube. The boundary match over the wide range wasobtained by adjusting the background plane so as to fit the wall. The

achieved by the interpolation of background parameters to obtain an op-

virtual object looked as if it actually existed in the real world. It was dif-

timal background plane for each viewer’s position.

ficult to eliminate the presence of the cube perfectly due to the difference

5. Conclusionof the brightness of the projected background and the directly-observed
background. As a result, the cube looked like a transparent box contain-

In SAR, when a virtual object’s shape is different from a real object’s

ing the virtual object inside. This visual effect was also yielded in the

shape, the difference of the shapes causes undesired empty areas onto

experiments below.

which the virtual object is not projected on the real object’s surface. We
proposed a view-dependent method to eliminate such empty areas by67

The Journal of the Society for Art and Science, Vol. 20, No. 1, pp. 55-71 (2021)

Figure 10. Projection images used in experiment of Figure 9.
Figure 9. Augmented views in real environment with far complicatedshaped background.

Applications for Mixed-Reality Experiences, Handbook of Research on Emerging Technologies for Digital Preservation and In-

projecting the real background behind the real object. In order to treat aformation Modeling, pp.256-287, IGI Global, 2017.

far background in a large space, the real background’s image is captured[3] Microsoft Research, User Interfaces for Spatial Augmented Real-by an RGB camera and converted to an image for the viewer’s position

ity, 2016.

based on homography. The image conversion uses the approximation of

https://www.youtube.com/watch?v=qZWpQ8Tuweo

the background’s shape by a background plane, which is defined by[4] Ramesh Raskar, Spatial Augmented Reality: Inserting in the Realpractical background parameters interpolated on a 3D grid space. ThisWorld, 2018.

adjusts the projected background so as to match the directly-observed

https://www.youtube.com/watch?v=4tRmDDM8lTs

background according to the viewer’s position. Consequently, our[5] Ramesh Raskar, Greg Welch, Matt Cutts, Adam Lake, Lev Stesin,

method makes the viewer not feel the presence of the real object but feeland Henry Fuchs, The Office of the Future: A Unified Approach tothe virtual object merging into the real world.

Image-Based Modeling and Spatially Immersive Displays, Proc.

Currently, the presence of a real object is not eliminated perfectly dueof ACM SIGGRAPH 1998, pp.179-188, 1998.

to the difference of the brightness of a projected background and a di-[6] Oliver Bimber, Daisuke Iwai, Gordon Wetzstein, and Anselm

rectly-observed background. This should be appropriately improved by

Grundhöfer, The Visual Computing of Projector-Camera Systems,

photometric calibration. Our method has a serious problem; the view-

Computer Graphics Forum, vol.27, 8, pp.2219-2245, 2008.

dependent display is available to only a single viewer and not available[7] Anselm Grundhöfer and Daisuke Iwai, Recent Advances in Pro-to multiple viewers simultaneously. However, the simplicity of ourjection Mapping Algorithms, Hardware and Applications, Com-method enables the integration with an existing method developed forputer Graphics Forum, vol.37, 2, pp.653-675, 2018.

multiple viewers. This is an important future work. Currently, a back-[8] Ramesh Raskar, Greg Welch, Kok-lim Low, Deepak Bandyo-ground plane is adjusted by giving background parameters manually.

padhyay, Shader Lamps: Animating Real Objects With Image-The development of an efficient way to determine optimal parameters

Based Illumination, Proc. of Eurographics Workshop on Render-

automatically or semi-automatically is also a future work. In the current

ing Techniques 2001, pp.89-102, 2001.

practice, a camera to capture a background and a tracking sensor to track[9] Ramesh Raskar and Paul Beardsley, A Self-Correcting Projector,a viewer are fixed near a real object and directed toward the sides oppo-

Proc. of IEEE Computer Society Conference on Computer Visionsite to each other. This configuration allows a viewer to see only one sideand Pattern Recognition (CVPR) 2001, pp.626-631, 2001.

of a virtual object. Actually, two Kinect sensors are used as the camera[10] Samuel Audet and Masatoshi Okutomi, A User-Friendly Method

and the tracking sensor respectively. If each Kinect sensor works as bothto Geometrically Calibrate Projector-Camera Systems, Proc. ofa camera and a tracking sensor, the viewer can see two opposite sides ofIEEE Computer Society Conference on Computer Vision and Pat-

the virtual object. By expanding this configuration, we are planning totern Recognition Workshops (CVPRW) 2009, pp. 47-54, 2009.

develop a system to allow a viewer to enjoy a 360-degree view of a vir-

[11] Tuotuo Li, Feng Hu, and Zheng Geng, Geometric Calibration of atual object merging into its real background by using multiple Kinect

Camera-Projector 3D Imaging System, Proc. of the 10th Interna-sensors.

tional Conference on Virtual Reality Continuum and Its Applications in Industry (VRCAI 2011), pp.187-194, 2011.
[12] Daniel Moreno and Gabriel Taubin, Simple, Accurate, and Robust

References

Projector-Camera Calibration, Proc. of the 2012 Second Joint[1] Oliver Bimber and Ramesh Raskar, Spatial Augmented Reality:

3DIM/3DPVT Conference: 3D Imaging, Modeling, Processing,Merging Real and Virtual Worlds, A K Peters/CRC Press, 2005.

Visualization & Transmission, pp.464-471, 2012.

[2] Alessandra Meschini, Daniele Rossi, Enrica Petrucci, and Filippo[13] I. Din, H. Anwar, I. Syed, H. Zafar, and L. Hasan, Projector Cali-

Sicuranza, Expanded Cultural Heritage Representation: Digital
68

The Journal of the Society for Art and Science, Vol. 20, No. 1, pp. 55-71 (2021)
bration for Pattern Projection Systems, Journal of Applied Re-(UIST 2017), pp.611-621, 2017.

search and Technology, vol.12, 1, pp.80-86, 2014.

[25] Andreas Fender, Philipp Herholz, Marc Alexa, and Jörg Müller,[14] Tyler Johnson and Henry Fuchs, Real-Time Projector Tracking on

OptiSpace: Automated Placement of Interactive 3D Projection

Complex Geometry Using Ordinary Imagery, Proc. of IEEE Com-

Mapping Content, Proc. of the 2018 CHI Conference on Human

puter Society Conference on Computer Vision and Pattern Recog-

Factors in Computing Systems (CHI 2018), paper no.269, pp.1-11,

nition (CVPR) 2007, pp.1-8, 2007.

2018.

[15] Shuntaro Yamazaki, Masaaki Mochimaru, and Takeo Kanade,[26] Hrvoje Benko, Andrew D. Wilson, and Federico Zannier, Dyadic

Simultaneous Self-Calibration of a Projector and a Camera Using

Projected Spatial Augmented Reality, Proc. of the 27th Annual

Structured Light, Proc. of IEEE Computer Society Conference on

ACM Symposium on User Interface Software and Technology

Computer Vision and Pattern Recognition Workshops (CVPRW)(UIST 2014), pp.645-655, 2014.

2011, pp.60-67, 2011.

https://www.youtube.com/watch?v=D6yWWMwjkcg[16] Behzad Sajadi, Mahdi Abbaspour Tehrani, Mehdi Rahimzadeh,

https://www.youtube.com/watch?v=Df7fZAYVAIE

and Aditi Majumder, High-Resolution Lighting of 3D Reliefs Us-[27] Wojciech Matusik and Hanspeter Pfister, 3D TV: A Scalable Sys-

ing a Network of Projectors and Cameras, Proc. of 2015 3DTV-

tem for Real-Time Acquisition, Transmission, and Autostereo-

Conference: The True Vision - Capture, Transmission and Display

scopic Display of Dynamic Scenes, Proc. of ACM SIGGRAPH

of 3D Video (3DTV-CON), pp.1-4, 2015.

2004, pp.814-824, 2004.

[17] Oliver Fleischmann and Reinhard Koch, Fast Projector-Camera[28] Tibor Balogh, Péter Tamás Kovács, and Zoltán Megyesi,

Calibration for Interactive Projection Mapping, Proc. of 2016 23rd

HoloVizio 3D Display System, Proc. of 2007 3DTV Conference,

International Conference on Pattern Recognition (ICPR), pp.3798-

pp.1-4, 2007.

3803, 2016.

[29] Andrew Jones, Ian McDowall, Hideshi Yamada, Mark Bolas, and[18] Simon Willi and Anselm Grundhöfer, Robust Geometric Self-Cal-Paul Debevec, Rendering for an Interactive 360° Light Field Dis-ibration of Generic Multi-Projector Camera Systems, Proc. of

play, Proc. of ACM SIGGRAPH 2007, article no.40, pp.1-10,

2017 IEEE International Symposium on Mixed and Augmented

2007.

Reality (ISMAR), pp.42-51, 2017.

[30] Joel Jurik, Andrew Jones, Mark Bolas, and Paul Debevec, Proto-[19] Jaewoon Lee, Yeonjin Kim, Myeong-Hyeon Heo, Dongho Kim,

typing a Light Field Display Involving Direct Observation of aand Byeong-Seok Shin, Real-Time Projection-Based Augmented

Video Projector Array, Proc. of IEEE Computer Society Confer-

Reality System for Dynamic Objects in the Performing Arts, Sym-ence on Computer Vision and Pattern Recognition Workshops

metry, vol.7, 1, pp.182-192, 2015.

(CVPRW) 2011, pp.15-20, 2011.

[20] Christian Siegl, Matteo Colaianni, Lucas Thies, Justus Thies, Mi-[31] Toshiyuki Amano and Kensuke Minami, Structural Color Display

chael Zollhöfer, Shahram Izadi, Marc Stamminger, and Frankon Retro-Reflective Objects, Proc. of the 25th International Con-

Bauer, Real-Time Pixel Luminance Optimization for Dynamic

ference on Artificial Reality and Telexistence and 20th Eu-

Multi-Projector Mapping, ACM Transactions on Graphics, vol.34,

rographics Symposium on Virtual Environments (ICAT-EGVE),6, article no.237, pp.1-11, 2015.

pp.37-44, 2015.

[21] Yuichiro Fujimoto, Ross T. Smith, Takafumi Taketomi, Goshiro[32] Andrew Jones, Jonas Unger, Koki Nagano, Jay Busch, Xueming

Yamamoto, Jun Miyazaki, Hirokazu Kato, and Bruce H. Thomas,

Yu, Hsuan-Yueh Peng, Oleg Alexander, Mark Bolas, and Paul

Geometrically-Correct Projection-Based Texture Mapping onto a

Debevec, An Automultiscopic Projector Array for Interactive Dig-

Deformable Object, IEEE Transactions on Visualization and Com-

ital Humans, Proc. of ACM SIGGRAPH 2015 Emerging Technol-

puter Graphics, vol.20, 4, pp.540-549, 2014.

ogies, article no.6, 2015.

[22] Gaku Narita, Yoshihiro Watanabe, and Masatoshi Ishikawa, Dy-[33] Shunsuke Yoshida, fVisiOn: Interactive Glasses-Free Tabletop 3D

namic Projection Mapping onto Deforming Non-Rigid Surface

Images Floated By Conical Screen and Modular Projector Arrays,

Using Deformable Dot Cluster Marker, IEEE Transactions on Vis-

Proc. of ACM SIGGRAPH Asia 2015 Emerging Technologies, ar-ualization and Computer Graphics, vol.23, 3, pp.1235-1248, 2017.

ticle no.12, pp 1-3, 2015.

[23] Amit H. Bermano, Markus Billeter, Daisuke Iwai, and Anselm[34] Min Ki Park, Kyu Je Lim, Myoung Kook Seo, Soon Jong Jung,

Grundhöfer, Makeup Lamps: Live Augmentation of Human Faces

and Kwan H. Lee, Spatial Augmented Reality for Product Appear-

via Projection, Computer Graphics Forum, vol.36, 2, pp.311-323,

ance Design Evaluation, Journal of Computational Design and En-

2017.

gineering, vol.2, 1, pp.38-46, 2015.

[24] Andreas Fender, David Lindlbauer, Philipp Herholz, Marc Alexa,[35] Ramesh Raskar, Michael S. Brown, Ruigang Yang, Wei-Chaoand Jörg Müller, HeatSpace: Automatic Placement of Displays byChen, Greg Welch, Herman Towles, Brent Seales, and Henry

Empirical Analysis of User Behavior, Proc. of the 30th Annual

Fuchs, Multi-Projector Displays Using Camera-Based Registra-

ACM Symposium on User Interface Software and Technology

tion, Proc. of IEEE Visualization, pp.161-168, 1999.
69

The Journal of the Society for Art and Science, Vol. 20, No. 1, pp. 55-71 (2021)[36] Ruigang Yang and GregWelch, Automatic and Continuous Projec-

The next equations are obtained from Equation (a.7):

tor Display Surface Calibration Using Every-day Imagery, Proc. of−1𝒕′𝑚 ] [𝑿𝑚 ],̃ = [𝑿] = 𝑀
̃𝑚 −1 𝑿̃ 𝑚 = [ 𝑅𝑚𝑿1
1
0 0 0 1the 9th International Conference in Central Europe on Computer
Graphics, Visualization and Computer Vision (WSCG 2001),
pp.328-335, 2001.

Appendix
A. Derivation of homography matrix
The homography matrix 𝐻 of Equation (11) is obtained as follows.
In the following, for convenience, some equations used in Section 3.3.2
are written again.
Each of two cameras 𝐶𝑚 , 𝑚 = 1, 2, has an intrinsic parameter matrix 𝐴𝑚 and an extrinsic parameter matrix 𝑀𝑚 = [𝑅𝑚 𝒕𝑚 ] definedby a rotation matrix 𝑅𝑚 and a translation vector 𝒕𝑚 as follows:𝑓𝑥𝑚
𝐴𝑚 = [ 0
0

𝑠𝑚
𝑓𝑦𝑚
0

𝑚
𝑟11
𝑚
𝑅𝑚 = [𝑟21
𝑚
𝑟31

𝑚
𝑟12
𝑚
𝑟22
𝑚
𝑟32

𝑐𝑥𝑚
𝑐𝑦𝑚 ],
1
𝑚
𝑟13
𝑚
𝑟23
],
𝑚
𝑟33

𝑡𝑥𝑚
𝒕𝑚 = [𝑡𝑦𝑚 ].
𝑡𝑧𝑚(a.12)

̃=𝑀
̃𝑚 −1 𝐴̃𝑚 −1 𝑼̃ ∗𝑚 .
𝑿
The following is obtained from Equations (a.11) and (a.13):
0
−1
−1
0]
𝒕′ 𝑚 ] [ 𝐴𝑚̃𝑚 −1 𝐴̃𝑚 −1 = [ 𝑅𝑚𝑀0
0 0 0 10 0 0 1(a.14)

𝒕′𝑚 ].
1(a.13)(a.15)

Equation (a.9) gives the camera 𝐶1
̃ 1∗ = 𝐴̃1 𝑀
̃1 𝑿
̃.
𝑼

(a.16)(a.2)

Equation (a.14) gives the camera 𝐶2
̃=𝑀
̃2 −1 𝐴̃2 −1 𝑼̃ ∗2 .
𝑿(a.3)

̃ 1∗ = 𝐴̃1 𝑀
̃1 𝑀
̃2 −1 𝐴̃2 −1 𝑼̃ ∗2 .
𝑼
The following is obtained from Equations (a.10) and (a.15):(a.17)
Then, the next equation is obtained from Equations (a.16) and (a.17):

−1
−1
̃1 𝑀
̃2 −1 𝐴̃2 −1 = [ 𝐴1 𝑅1 𝐴1 𝒕1 ] [𝑅2 𝐴2
𝐴̃1 𝑀
0 0 0
1
0 0 0

For a point 𝑄 in the 3D space, its 2D pixel coordinates 𝑼𝑚 =[𝑢𝑚 , 𝑣𝑚 ]𝑇 and 3D camera coordinates 𝑿𝑚 = [𝑥𝑚 , 𝑦𝑚 , 𝑧𝑚 ]𝑇 of the−1−1= [𝐴1 𝑅1 𝑅2 𝐴2
0 0 0

camera 𝐶𝑚 and its 3D world coordinates 𝑿 = [𝑥, 𝑦, 𝑧]𝑇 have the relationships(a.18)

𝒕′ 2 ]
1

𝐴1 𝑅1 𝒕′ 2 + 𝐴1 𝒕1 ].
1(a.19)

Then, the following is obtained from Equations (a.12), (a.18), and𝑿𝑿̃,𝑿𝑚 = [𝑅𝑚 𝒕𝑚 ] [ ] = 𝑀𝑚 [ ] = 𝑀𝑚 𝑿
1
1

(a.4)

̃ 𝑚 = 𝜆𝑚 [𝑼𝑚 ] = 𝐴𝑚 𝑿𝑚 ,
𝜆𝑚 𝑼
1

(a.5)

(a.19):
̃ 1 = [𝐴1 𝑅1 𝑅2 −1 𝐴2 −1
𝜆1 𝑼

̃
𝜆 𝑼
𝐴1 𝑅1 𝒕′ 2 + 𝐴1 𝒕1 ] [ 2 2 ]1̃ 2 + 𝐴1 𝑅1 𝒕′ 2 + 𝐴1 𝒕1= 𝜆2 𝐴1 𝑅1 𝑅2 −1 𝐴2 −1 𝑼
−1 ̃
−1
= 𝜆2 𝐴1 𝑅1 𝑅2 𝐴2 𝑼2 + 𝐴1 (𝒕1 − 𝑅1 𝑅2 −1 𝒕2 ).

where 𝜆𝑚 ≠ 0 is a constant. The symbol 𝑇 means a transposed matrix. These are unified as follows:̃ 𝑚 = 𝐴𝑚 𝑀𝑚 𝑿
̃.
𝜆𝑚 𝑼

𝒕′𝑚 = −𝑅𝑚 −1 𝒕𝑚 .
The next equation is obtained from Equation (a.8):0𝑼−10] [𝜆𝑚 [ 𝑚 ]].
̃ 𝑚 = [𝑿𝑚 ] = 𝐴̃𝑚 −1 𝑼̃ ∗𝑚 = [ 𝐴𝑚
𝑿
1
1
0
1
0 0 0 1
The next equation is obtained from Equation (a.9):−1−1= [𝑅𝑚 𝐴𝑚
0 0 0

(a.1)

(a.11)(a.20)

The world coordinates 𝑿 of a point 𝑄 on a plane 𝑃 seen from thecamera 𝐶𝑚 satisfy(a.6)

𝑵𝑇 𝑿 + 𝑑 = 𝑛𝑥 𝑥 + 𝑛𝑦 𝑦 + 𝑛𝑧 𝑧 + 𝑑 = 0,

Equations (a.4), (a.5), and (a.6) are given other representations using̃𝑚 and 𝐴̃𝑚 as follows:
4 × 4 matrices 𝑀(a.21)

𝑇

where 𝑵 = [𝑛𝑥 , 𝑛𝑦 , 𝑛𝑧 ] , |𝑵| = 1, is a unit normal vector and 𝑑 is

𝒕𝑚 𝑿
𝑀𝑚
𝑿
̃ 𝑚 = [𝑿𝑚 ] = [ 𝑅𝑚
̃𝑚 𝑿
̃,
𝑿
][ ] = [][ ] = 𝑀
0 0 0 1 1
0 0 0 1 1
1

a constant. From Equation (a.11), the world coordinates 𝑿 and the

(a.7)
0
𝑼𝑚
̃𝑚
𝐴
0] [𝑿𝑚 ] = 𝐴̃ 𝑿
̃ ∗𝑚 = [𝜆𝑚 𝑼̃𝑼] = [𝜆𝑚 [ 1 ]] = [ 𝑚
𝑚 𝑚,
0 1
1
1
0 0 0 1(a.8)
̃ ∗𝑚 = 𝐴̃𝑚 𝑀
̃𝑚 𝑿
̃.
𝑼
(a.9)

camera coordinates 𝑿𝑚 of the point 𝑄 have the relationship
𝑿 = 𝑅𝑚 −1 𝑿𝑚 + 𝒕′𝑚 .

(a.22)

Then, from Equation (a.21), the camera coordinates 𝑿𝑚 satisfy
𝑵𝑇 (𝑅𝑚 −1 𝑿𝑚 + 𝒕′𝑚 ) + 𝑑 = 0.

(a.23)

From Equation (a.12), Equation (a.23) is arranged into
𝑵𝑇 𝑅𝑚 −1 𝑿𝑚 = 𝑵𝑇 𝑅𝑚 −1 𝒕𝑚 − 𝑑.

The following is obtained from Equations (a.7) and (a.8):
0
𝒕𝑚
𝐴 𝑅
𝐴𝑚 𝒕𝑚
𝐴𝑚
0] [ 𝑅𝑚
̃̃
𝐴𝑚 𝑀𝑚 = []=[ 𝑚 𝑚].
0 0 0
10 0 0 0 1
0 0 0 1(a.10)(a.24)

Then, the following is obtained from Equations (a.5) and (a.24):
̃ 𝑚 = 𝑵𝑇 𝑅𝑚 −1 𝒕𝑚 − 𝑑.
𝜆𝑚 𝑵𝑇 𝑅𝑚 −1 𝐴𝑚 −1 𝑼
(a.25)
From Equation (a.22), the origin 𝑂𝑚 of the camera coordinates 𝑿𝑚
has world coordinates
𝑶𝑚 = 𝑅𝑚 −1 [0 0 0]𝑇 + 𝒕′𝑚 = 𝒕′𝑚 .
70(a.26)

The Journal of the Society for Art and Science, Vol. 20, No. 1, pp. 55-71 (2021)
Then, from Equations (a.22) and (a.26), the viewing direction vector
𝑉𝑚 from the origin 𝑂𝑚 to the point 𝑄 has world coordinates
𝑽𝑚 = 𝑿 − 𝑶𝑚 = 𝑅𝑚 −1 𝑿𝑚 .
If 𝑵𝑇 𝑽(a.27)−1𝑇
𝑚 = 𝑵 𝑅𝑚 𝑿𝑚 = 0, the vector 𝑉𝑚

is parallel to the plane

𝑃 . In this case, the plane 𝑃 cannot be seen from the camera 𝐶𝑚 .
Therefore, it is assumed that
𝑵𝑇 𝑅𝑚 −1 𝑿𝑚 ≠ 0.
(a.28)
Thus, the following is obtained from Equations (a.24) and (a.28):
𝑵𝑇 𝑅𝑚 −1 𝒕𝑚 − 𝑑 ≠ 0.

(a.29)

Then, the following is obtained from Equation (a.25):𝜆𝑚
𝑵𝑇 𝑅𝑚 −1 𝒕𝑚 −𝑑̃ 𝑚 = 1.
𝑵𝑇 𝑅𝑚 −1 𝐴𝑚 −1 𝑼

Khuslen Battulga received a Bachelor and a Master of Business Administration in Management Information Systems from Mongolian(a.30)

University of Science and Technology in 2011 and 2012, respectively.
She worked at Millennium Challenge Corporation from 2012 to 2013.

Equation (a.30) is given 𝑚 = 2 and substituted in Equation (a.20)
to obtain
̃ 1 = 𝜆2 𝐴1 𝑅1 𝑅2 −1 𝐴2 −1 𝑼
̃2
𝜆1 𝑼
+𝐴1 (𝒕1 − 𝑅1 𝑅2 −1 𝒕2 )𝜆2
𝑵𝑇 𝑅2 −1 𝒕2 −𝑑

She was a research student from 2013 to 2014 and a Ph.D. student from2014 to 2020 in Graduate School of Engineering, Iwate University. Her
̃ 2.
𝑵𝑇 𝑅2 −1 𝐴2 −1 𝑼

research interests include projection mapping, 3D modeling, animation,(a.31)and graphic design.

Equation (a.31) is arranged as follows:̃ 1 = 𝜆2 𝐴1 {(𝑵𝑇 𝑅2 −1 𝒕2 − 𝑑)𝑅1
𝜆1 (𝑵𝑇 𝑅2 −1 𝒕2 − 𝑑)𝑼̃ 2.
+(𝒕1 − 𝑅1 𝑅2 −1 𝒕2 )𝑵𝑇 }𝑅2 −1 𝐴2 −1 𝑼

(a.32)

Then, the following is obtained from Equation (a.32):
̃ 1 = 𝐴1 {(𝑵𝑇 𝑅2 −1 𝒕2 − 𝑑)𝑅1
𝜆𝑼
̃ 2,+(𝒕1 − 𝑅1 𝑅2 −1 𝒕2 )𝑵𝑇 }𝑅2 −1 𝐴2 −1 𝑼

(a.33)

where
𝜆 = 𝜆1 (𝑵𝑇 𝑅2 −1 𝒕2 − 𝑑)⁄𝜆2 ≠ 0
is a new constant. Equation (a.33) is arranged as follows:̃ 1 = 𝐻𝑼̃ 2,𝜆𝑼
𝐻 = 𝐴1{(𝑵𝑇(a.34)(a.35)

𝑅2−1 𝒕2 − 𝑑)𝑅1 + (𝒕1 − 𝑅1 𝑅2−1 𝒕2 )𝑵𝑇 }𝑅2−1 𝐴−1
2 .

Tadahiro Fujimoto is currently a professor in the Department of Com-(a.36)

puter and Information Sciences at Iwate University. His research inter-

Equations (a.35) and (a.36) are the same as Equations (4) and (11).

ests include computer graphics and computer vision. He received a BE
in electrical engineering, and an ME and a Ph.D. in computer sciencefrom Keio University in 1990, 1992, and 2000, respectively. He worked
at Mitsubishi Research Institute from 1992 to 1995. He was a research
associate in the Department of Computer and Information Sciences at
Iwate University from 1999 to 2002, a lecturer from 2002 to 2005, andan associate professor from 2005 to 2016. He is a member of SAS Japan,
IEICE Japan, IPS of Japan, IEEE and ACM.

71