Dallas, August 18-22

Volume 20, Number 4, 1986

A MONTAGE METHOD: THE OVERLAYING OF THE COMPUTER
GENERATED IMAGES ONTO A BACKGROUND PHOTOGRAPH
Eihachiro Nakamae, Koichi Harada and Takao Ishizaki
Hiroshima University
Saijo, Higashi-Hiroshima
Hiroshima 724,
Japan
Tomoyuki Nishita
Fukuyama University
Higashimura, Fukuyama
Hiroshima 729-02, Japan

ABSTRACTI. Introduction

A system of computer programs has
beenestablished to generate
highquality montage
image of considerable usefulness in architectural
simulation which combine computer-generated images
and photographed background pictures.
Traditionally,
there are two methods ofcreating architectural montages: (I)anartistpaintsnew buildings onto a background scene
usually generated photographically,and (2) athree-dlmensional scale modeliscreatedtosimulate the whole landscape, and this model isthen photographed.
The montage method describedhere combines aspects of both traditional montage
methods with significant improvementin accuracy
and reduction of time and cost of preparation.
Specifically, a digitized photograph is used as a
background scene onto which is superimposed a 3D
computer-generated image of a new building.
The
outstanding points of the new method are:(i) The shading and shadows of each computer
generated image are calculatedwithhigheraccuracy,(ii) the fog effect is taken into
account,
and (iii) a new anti-aliasing technique
improves the quality of the final montage image.

In order to assess the potential impact of a
new construction on local environments, it is very
important to understand what is going to happen toviewing angles and local lighting/shadows if newbuildings are constructed.
Assessment grows more
important and more difficult as the number andscale of the new buildings increases.
Traditionally,
two
methods
have
beenused for attacking this problem: (I) A landscapewhere new construction occurs is photographed, andan artist paints the
new building(s) onto thispicture. (2) A 3D scale model is made to gauge theinfluence
of
the
new
buildings ontheirsurroundings.
The major difficulty of the second method isthat it is expensive eventhough it offers
three-dimensional investigation.
And it should be
noted that every part
of
the
model
is
artificial.
Though the first method
is less
costly,the montage is totally affected by theartist's personal preference.
Modern computer graphics techniques offer a
viable solution to this problem.
In 1979, Uno etal.[1] proposed a general purpose graphics systemthat was used to create simple image compositions.
In 1980, Feibush et al.[2] overlaid an imaginary
house onto a background using a complex database
composed of many polygons and textures.
In 1984,
Porter et al.[3] presented an interesting approach(Alphachannelmethod)tothedigitized
composition problem that allows processing ofcomplicated three-dimensional images.
Image compositions in these earlier studies
have been investigated chiefly from the viewpointof antl-aliasing.
The montage system in thispaperincludesthefollowing
uniquecharacteristics
alongwith the anti-aliasing
processing:(I) The ratio between the maximum illuminancedue to the sun beam [4]andtheambient
illuminanceof each computer generated image
described in an early paper by the authors is
calculated by using information contained in the
backgroundpicture.
This
enablessuperior
simulation of natural shading and shadows.
(2) Atmospheric moisture effects (fogginess)is added to simulate various weather conditions.
In the new method, the fog effects are processedbyusing
exponentialfunctions[5].
Thesefunctions are determined with a sophisticatedalgorithm based on the illuminance information

CR
Categories
andSubject
Descriptors:
1.3.3[Computer Graphics]:Picture/Image Generation-- display algorithm;
1.3.5[Computer Graphics]:
Computational Geometry and Object Modelling -curve, surface, solid and object representations;1.3.7[Computer
Graphics]:
Three-Dimensional
Graphics and Realithm -- color, shading, shadowing
and texture; 1.4.3[Image Processing]:
Enhancement-sharpening
anddeblurring;
1.4.6[Image
Processing]:
Segmentation -region
growing,
partitioning;
1.4.8[Image
Processing]:
Scene
Analysis -- depth cues; J.4 [Social and Behavioral
Sciences]: Sociology
General Terms: Algorithms, Measurement

Permission to copy without fee all or part of this material is granted
provided that the copies are not made or distributed for directcommercial advantage, the ACM copyright notice and the title of the
publication and its date appear, and notice is given that copying is bypermission of the Association for Computing Machinery. To copy
otherwise, or to republish, requires a lee a n d / o r specific permission.
@ 1986

ACM 0-89791-196-2/86/008/0207$00.75207

~. S I G G R A P H

contained in the
backgroundpicture.
This
makes the montage more reliable and applicable forwider range of environment impact assessment.

2. Overlaying of Computer-Generated Images onto a
Background Photograph
2.1 Schematic Description of Proposed Method
The method consists of the following five
steps:(I) The data for computer-generated building
images are prepared.
(2) The land where the buildings wall be
constructed is photographed.
This picture isprocessed
withan
A/D converter;
geometrictransformation (translation, rotation,
etc.)
are
carried out to fit the photographic image on a CRT
screen.
(3) The computer-generated
images and thephotographed background scene are matchedfromtheviewpointofgeometry and adjusted by
calculating
thecontrolparameters(thecoordinates of the camera position,
the view
angle,
theambient
illuminanceandthefogginess).
(4) The computer
generated
imagesareoverlaid on the background scene and displayed on
the CRT.
Hidden-surface removal,
anti-aliasing
and fog calculations are implemented at thisstage.
(5) The foreground scene in front of thesuperimposedcomputer-generated
image
is
re-displayed.
Further explanation of Steps (3), (4) and (5)is given below.
2.2 Geometric Matching Between Computer-Generated
Images and Background Scene
The view reference point (a point existing on
the object at the center of the photograph)andtheviewpoint(the camera location)must bedesignated to carry out projection mapping.
The
object point is easily known by tracing theposition of this point on a geophysical map.
However,theviewpointis
not easily
determined if the camera is set on an airplane for
instance.
The least square method (see Appendix)
may be used to estimate the viewpoint comparing
several pointsonthe photograph with theiractual position coordinates on a map.
The size of the perspective plane is alignedwith that of the photographbyselectingapproximate
horizontal
and verticalangles;these angles are calculated based on the focal
length of the camera lens and the size of thephotograph.
They can also be determinedbytracing
the objects existingat the edge ofthe photograph on a map.
The photographs are
assumed to have been taken with a standard (55mm)
or telephoto lens to avoid the distortion caused
by wide angle lenses.
2.3 Shading
The position of the sun must be determinedfrom the photograph's time and date as well as thelongitude and latitude of the camera position,
in

208

'86

order
to correctly calculate the shading ofbuildings and the shadows they cast inthesimulated montage image.
Let the angle between
the x-axis (that defines the computer-generated
image) and the southern direction be 6 and thehour angle be T(T=0 deg. corresponds to thenoon
and T =15
deg.
indicates one
hour later).
Furthermore, let $ and 6 denote the longitude and
the
solar
declination,
respectively.
The
direction toward the sun is determined by:[0[-cos~sin~×

L cos6cos~

co,
0
0

@
,

COST]
s~nT

sin6cos~

cos6

@

0

sin6sin$(i)

Once the direction of the sun is determined,shadows and shading are calculated by traditional
methods.
The shading on each object is describedby:

E = Icos~

+ Eh

(2)

where I, cos a and E h stand for the intensity ofsun light, the cosine of the angle between thesurface normal and the direction vector toward
the
sun
and
the
ambient
illuminance,
respectively.
For simplicity, specular reflection
is neglected.
For montage generation, the ratio between E handI,
called H, is important because theultimate aim is to match computer-generated images
to the background scene. First, H is calculated
for the backgroundscene by using the method
explained below.
Then,
the computer generated
images are designed so that their H values areequal to that of the background image.
Two walls of a building in the backgroundscene are chosen (one is preferred to face theshadow side),and their intensity
level
isdetected(this quantity is proportional to theilluminance and the reflectance).
H is thencalculated as:H= (D2cos@ 1 - DlCOS@2)/(Dcosei(±=l,2 ) => 0

l - D2) ,<3)

where Di, D 2 are illuminanoe on each wall.
@ 1 and@2 denote the angle between the direction vector
toward the sun and the surface normal of each wall
surface, respectively.
If there is no such building, a pilot box (awhite box is useful for this purpose) may be set
in the vicinity of the camera to serve as areference for ambient illuminance.
Since the ambient illuminance is affected by
the surrounding objects, H varies depending on theobject position.
H should he calculatedforDallas, August 18-22

Volume 20, Number 4, 1986

several points and the averagecalculation results is used.

value of these

2.4 Fog Effects
In general an object far from the viewpoint
seems to lose its color and tone because of
atomospheric moisture effects.
If these effects
are not taken into account, the new buildings,
computer-generated
images, do not
match
the
photographed background pictures.
The reduction rate of color and tone depends
on the various factors such as seasons,weather
conditions and time.
This fact means that the
informations of the reduction rate should beobtained from the background picture, even though
it is already known that the color and tone ofeach object changes exponentially with respect toits distance from the viewpoint [5].
A new methodhasbeen
established todetermine this exponential function based on the
background picture: The color and tone become grayas the distance becomes large. Let red-, greenor blue-component of the illuminance at a point in
the picture be C e (e=r,g,b, respectively). If thepoint moves from the viewpoint to the infinite
point,
the apparent color C'e of it approaches
exponentially to a color which depends on the
background picture.
This color is called thestandard color F e.
The apparent color C'e istherefore given by:

!

Ce

=

(-[-exp(-x/Re))(F

e

- Ce

)

+ Ce,

(4)

e= r,g,bR(standard distance) and F e are determined
as fol~ows: At least,
three sampled points aredesignated on the photograph; more sampled points
help refine the final result. These points should
be selected on objects which are consideredtoshare a color (trees, for example) but stand atdifferent distances from the viewpoint.
Theilluminanee on these points, which is defined as
the average value in the vicinity of each point,
is detected on the photograph.
The distance toeach of the points should be obtained by using thegeophysical map. Data points are plotted on ailluminance-distanoe plane (see Fig.l).
The least
mean square measure is used to give the mostappropriate exponential function from these data.
A simpler method is to use a pair of objects.
These objects are assumed to possess different
colors and locate at a near position (x=0 in
Eq.(4)) and at the distance x. Let A~ and B~ bethe colors of the pair of objects at the distancex.
Then, these colors are given with their
original colors A e and B e (the color at a nearposition of each object) as:

A'e = Ke(Fe - A e ) + A e 'B'e = Ke(Fe - B e ) + B e 'K e = I - exp(-x/R e ),e= r,g,b(s)

By solving a simple system of equations, thestandard distance R e (which dominates the changeof each color with respect to the distance) andthe standard color F e is determined by:

Fe(AeB ~-

BeA~)/(Ae- A e'-Be* B~) '

(63

R e = -x/ioge[(f e - A~ )/(f e - Ae)]The fogginess of an image heavily depends on
the weather conditions when the photographs aretaken. Therefore, the requirement to add or evenremove the fog in the photographed scenes mightoccur in order to observe the montage image under
the different conditions.
Strictly speaking, a background scene should
be processed in a three-dimenslonal space to know
the exact fog effect.
Since this ks almost
impossible, the next best approximation is useful:
Each object on the background scene is describedwith a combination of planes.
Thedistance
betweentheobjects
andthe viewpoint iscalculated by using this approximated image.
In
the case of Fig.2, mountains (A through D) and agrove of trees (E) are approximated with verticalplanes.
F and G are processed with horizontal
planes. The road (H) and two banks of earth (Iand J) are treated as convex polyhedra.
A cursor or a tablet is used to interactively
divide the background scene into several parts,and each part is checked whether it should beapproximatedwith vertical planes,
horizontal
planes or p olyhedra.
The distance fromtheviewpoint is also specified at this step.
Because each part consists of planes, Eq.(~)
may be applied to calculate the fog effects ateach point.
It should be noted that the longer
the standard distance,
the smaller thefogeffect,and vice versa.
2.5 Anti-Aliasing
It is well known that the diagonal edges ofobjects displayed on a raster-scan type CRT arejagged.
In some cases, moire patterns appear;e.g. the same shaped objects are set periodically.
These undesireable phenomena called'aliasing'make the computer-generated
imagesunnaturalagainst the background scene.
The anti-allaslng for montage methods should
be different from other usual ones because: (1)the background scene includes pixels which ingeneral change pixel by pixel,and (ii) image
overlay operation is performed several
times(overlay process initiates from the further scene,and ends at the closest scene,or foreground).
This
first
characteristic((i)
above)
excludes the use of filters that work based on
more than one pixel.
If these filters were used,
the resulting image would be very vague. Thissecond characteristic((ii)
above)
increases
calculation cost, so a simpler filter is preferredover more complicated filters, such as the conicalfilter or Gausslan filter[2].
The Alpha channel method[3]is useful for
anti-aliasing when
makingextremely
complexmontages requiring many overlay loggers.
However,209

~. S I G G R A P H '86

anti-aliasing is only really required along theedges of each computer-generated
image.
Hence,
for simple montages such as discussed here, thefollowin~ easy anti-aliaslng method is suggested:
The color at each pixel on the CRT screen is
first read out. The color of the background scene
isdetermined pixel by plxel based on thisinformation.
Then,a modified two-dimensional
Fourier window (box window)
is activated.
The
detailed explanation is:
A pixel is approximated with a unit square
shown in Fig. 3- This square is divided
into nparts.
S o is an actual scan line.
Each of otherlines within the plxel is called a virtual scanline (VSL). The color of this pixel is determinedaccording to the product of the length of theimage on each VSL S£ (£=1,2,...,n)and
theilluminanoe of this image.
Each product is addedwith a proper weight to decide the pixel color:
Let m objects exist within the pixel (i,j). The
background scene color at this pixel is supposedtobe~ i'' (this vector consists of threecomponents that correspond to red, green andblue). Furthermore, let Lf, £ and bi,j, Z stand forthe visible interval on the VSL of the object Of
and the background scene, respectively.
The colorCi, j of the pixel (i,j) is calculated according to
the trapezoidal integration rule as:
n

Ci '

J

m

= Z d£(bi,
Z=i

=

d9%

[

i/n;

j ~B i j + f=]
z
'
'
0

< Z

Lf,£

Cf) ,
(7)

< n

[ 1/2n; ~=O,nwhere d£ indicates the weight factor.
color of object Of.

Cf

is

the

2.6 Removal of Computer-Generated Images Hidden byForegrounds
A basic problem of this overlay method isthat the inserted computer-generated images mighthide the scene which exists in front of them. To
avoid this drawback, the scene which is supposed
to exist in front of the computer-generated images
is interactively designated with a cursor or atablet, and the images inside the designated area
are overlaid again.
The boundaries of
eachdesignated area are depicted with straight lines
or splines[6]-- the former isusefulforbuildings, and the latter to mountains or trees.
The aliasing problem mentioned above also
occurs when the foregrounds are superimposed onto
the computer-generated images.
In this case, theboundaries of the foregrounds become jagged.
This
problem is serious especially when the foregrounds
cross the objects,
which consist of straightboundaries (buildings,
for example).
The antialiasing method is also applicable here:
If apixel includes the boundaries of a foreground, itscolor is determined based on (1) the area of theforeground,(ii) the background color and (iii)the color of the computer-generated
image within
this pixel. The color of the pixel is determinedconsidering the intersection of theboundary(straight line or spline) of the image and each
VSL.

210

3. ImplementationFig.
4 indicates R, G and B intensity
variation with respect to the distance.
Trees atvarious distance from the viewpoint are selectedfrom the original background scene.
Actual R, G
and B intensity calculation was carried out by
averaging each small region,
about ten pixels,
within a group of trees.
The exponential function
described in Section 2.4 was determined in thisway. The exponential functions in Fig. 3 is used
for the second example.
Fig. 5 shows the outputs of the proposed
method.
Ca) is the background scene.
(b) depicts
the overlay of the computer-generated images onto
the background scene, and the foreground scene infront of the superimposed computer-generated image
has been re-displayed.
However, anti-allasing andthe distance from the viewpoint are not taken into
account.
Therefore, the windows of the buildings
seem uneven, and the contrast of the building far
from the viewpoint looks too strong because theinserted computer-generated images do not match
their surroundiags.
In (c), anti-aliaslng anddistanceareconsidered,
but there are no shadows from the newbuildings.
As a result,the building near the
viewpoint seems to stand above the earth level.
In (d), the shadows are cast properly on the
ground up to a grove of trees and a house near theright side.
In (el, the fog effect is activated.
Thebuildings seem to be natural due to this effect.
In (f), the fog effect is removed.
The mountains
look
as
if
they
are
too
close.
The
computer-generated images match their surroundings
in cases (d), (e) and (f).
Figures 5(g), (h) and (i) demonstrate anotherset of examples.
(g) correponds to (d) in theprevious example (anti-aliasing, the distance fromthe viewpoint and the shadows are all considered).
(h) and (i) show the cases when the fog effect is
activated and when it is removed, respectively.
Note that the difference between (g) and (i) is
not so clear,
while(f) is very different from(d). This is simply because the weather conditionin (d) was better (fog was thinner) than that in(f).
The fog effect is more apparent in (j).
Upper-left is the original backgroundscene.
Other pictures demonstrate the fog effect. Fourbuildings in upper-right are set on different
distance planes;
the greater the distance of abuilding from the viewpoint, the stronger the fog
effect.
The fog effect is removed in lower
right.
In (k), four different colors are assigned tothe bridge.
This is another application of the
proposed method to assess the visual impact [8] of
new construction on its surroundings.
Calculations were carried out on a TOSBAC
DS600 computer and GRAPHICA M508 color graphic
display (512x512 pixels, 8-bit each for R, G andB).
4. ConclusionThe
because:proposed method produces useful montages

Dallas, August 18-22

Volume 20, Number 4, 1986

(1) The fog effect increases
the sense of
perspectivein the montages,
and it permits
simulation of various weather conditions.
(2) The shading and shadows of the computer
generated
images help a montage match to thebackground scene.
(3)
The
new
anti-aliasing
techniqueenables cost-effective
high quality display ofobjects which might include narrow vertical parts;
e.g. electric light poles or chimneys.
Further improvement should enable:(I) The correction of the distortion whichappears on
thephotograph due to the use of awide-angle lens.
(2) The computer image generation of roadsidetrees and/or flower beds.
(3) Processing for the light that comes frombuildings.
Of
these
improvements,(I)
requirescomplicated geometric transformation. The clue toachieve (2)
is
found
in
Ref.[7].
(3) isimportant when a night view is considered.

Partially differentiating this value with respect
to R, @ and ~, the following system of equations
is given:

ARR

ARO

AR¢

-dR

B1

AOR

A@0

A@¢

-dO

B2

Aq$

-d¢

B3

ASR

A,O

gj

APPENDIX

Location of Viewpoint

The ultimate aim is to estimate the coordinate of the viewpoint.
This point is defined byR(the distance between the viewpoint and the viewreference point), 0 (the azimuth angle) and ¢ (theelevation angle),n sampled points are selectedon the picture for calculating the viewpoint.
3D
coordinateof
thesepoints,(xi,Yi,Zi),
i=1,2,...,n,may be obtained on the geographicalmap.
Thecoordinateof these points onthepicture,(Mxi,Myi),along with the above 3-D
coordinates
are used for calculating the locationof the viewpoint based on the following equation:exi = fi(R,@,~)- Mxi ,

eyl = gi(R,@,¢)- My i ,fi(R,O,@)~ f(R,O,~,xi,Yi,Zi)

gi(R,O,¢)~ g(R,O,¢,xi,Yi,Z

i = 1,2,--,n

,

(A1)

~) ,

(n>3)

In gq.
(AI),
f(g) and exi(eyi) mean the relation
function which maps the points in the 3-D spaceonto
the2-D picture coordinatesandthedifference
between the mapped coordinates and its
measured coordinates on the picture, respectively.
The best viewpoint estimation is described as
the minimization problem of:n
Q = z (e2 + 2
i =I
xl
eyi)

(A2)

,

= iE=l(a}Exi + b)Eyi)

ai =e(afi/3e)R=R0,@=00,~=@0(a;),

b i = (agi/ae)e
R=R0,@=00,~=OO
e = R,O,~ ,5. Acknowledgments
The backgroundscenes were prepared withimage scanners installed
at GRAPHICA Inc. andGovernment Industrial Research Institute, Chugoku.
Thecomputer-generated
imageswerecreatedaccording to data supplied by Kajima Corporation.
We are gratefultoLaurinHerrfor
his
discussions.
Special thanks to Tsuyoshi Tokoro
for his skillful computer operation.

n
, .
i i
g (al.a, 1 + b j b k )i=l j Kn

Ajk =Exi = fi(Ro,O0,$O)- Mxi ,

Ey i = gi(Ro,@o,$o)- My I

Appropriate values for R, 0 and $ are set for
initiating the iteration.
On each iteration, theestimated value of the viewpoint(R0,00,¢0)isreplacedto the value (R,e,¢)obtained by thefollowing equation.
R

=

R 0

-

dR

,

0 = 60 - d0

,

(A4)

qb = qb0 - d 0
The iteration continues until dR, d0 and d0 become
small enough.
REFERENCES

I S.Uno and H.Matsuka,"A General Purpose Graphic
Systemfor Computer Aided Design,"
ComputerGraphics, voi.13, no.2,1979, pp.25-32
2 E.A.Feibush,
M.Leroy and R.L.Cook,"Synthetic
Texturing
Using
DigitalFilters,"
ComputerGraphics, voi.14, no.3, 1980, pp.294-301
3 T.Porter and T.Duff,"Composing Digital Images,"
Computer Graphics, voi. 18, no.3,1984, pp.253-259
4 T.Nishita and E.Nakamae,"A Perspective Depictionof Shaded Time," Proc.
of 4th Int.
Symp. on the
Use of Comp. for Env. Eng. Related to Build.,1983,
pp.565
5 B.J.Schachter, Computer Image Generation,
John
Wiley & Sons.,New York, 1983
6 C.de Boor,Practical Guide to Splines,
SpringerVerlag, New York, 1978
7 R.Marshall,
R.Wilson and W.Carlson,"Procedure
Models for Generating Three-Dimensional
Terrain,"
Computer Graphics, voi.14, no.3,1980, pp.154-162
8 T.W.Maver, C.Purdie and D.Stearn," Visual Impact
Analysis -- Modelling and Viewing the Natural and
Built Environment," Comput.
& Graphics,
vol.9,
no.2, 1985, pp.117-124

211

i

~. S I G G R A P H

250

'86

B

200
L.
0

$i

0

=

150
S~

(D
q-.
0

Sn

i00
C
0
C

50

®

:R

•

; 6

®

; B

Fig. 3Antl-allasing operation in one plxel.

250

0

0

5

i0

15

20

25

Distance (Km)Fig. I
Intensity variation with respect to the
distance.
This Figure is referred along with Fig.4(see Section 4).

2OO
B
e

R

®

150

V °

4.J

®

F

H

4-J

50

.:6

'
)

0

®

:B

i

|

i

!

|

5

I0

15

20

25

Di.stance (Km)Fig. 2
Partition
processing.

212

of

the

image for fog effectFig. 4
Intensity variation with respect to the
distance. This figure was given from the first
background scene.

Dallas, August 18-22

Volume 20, Number 4, 1986

(o)

(b)

(c)

(d)

(e)

(f)
Fig. 5 (continued)213

S I G G R A P H

'86

(h)

(g)

(i)
(j)

Fig. 5
text.

(k)

214

Examples.

Explanations are given

inthe