Computer Graphics

Compositing

DigitalVolume18, Number 3

July 1984

ImagesThomas Porter
Tom Duff 'fComputer Graphics Project
Lucasfilm Ltd.

ABSTRACT

Most computer graphics pictures have been computed all
at once, so that the rendering program takes care of allcomputations relating to the overlap of objects. There areseveral applications, however, where elements must be
rendered separately, relying on eompositing techniques forthe anti-aliased accumulation of the full image. This paper
presents the case for four-channel pictures, demonstratingthat a matte component can be computed similarly to thecolor channels. The paper discusses guidelines for thegeneration of elements and the arithmetic for their arbitrary compositing.
CR Categories and Subject Descriptors: 1.3.3 [Comp u t e r G r a p h i c s ] : Picture/Image Generations - Display algorithms; 1.3.4 [ C o m p u t e rGraphics]:
Graphics Utilities - - Software support; 1.4.1 [ImageP r o c e s s i n g ] : Digitization - - Sampling.
General Terms:

Algorithms

Additional Key Words and Phrases: compositing, mattechannel, matte algebra, visible surface algorithms,
graphics systems

f Author's current ~ddress: AT&T Bell Laboratories, Murray Hill, NJ 07074, Room 2C46S

Permission to copy without fee all or part of this material is granted
provided that the copies are not made or distributed for directcommercial advantage, the ACM copyright notice and the title of the
publication and its date appear, and notice is given that copying is bypermission of the Association for Computing Machinery. To copy
otherwise, or to republish, requires a fee and/or specific permission.

© 1984ACM

0-89791-138-5/84/007/0253$00.75

1. I n t r o d u c t i o n
Increasingly, we find that a complex three dimensional
scene cannot be fully rendered by a single program. The
wealth of literature on rendering polygons and curved
surfaces, handling the special cases of fractals and spheresand quadrics and triangles, implementing refinements for
texture mapping and bump mapping, noting speed-ups on
the basis of coherence or depth complexity in the scene,suggests that multiple programs are necessary.
In fact, reliance on a single program for rendering anentire scene is a poor strategy for minimizing the cost ofsmall modeling errors. Experience has taught us to break
down large bodies of source code into separate modules inorder to save compilation time. An error in one routine
forces only the recompilation of its module and the relatively quick reloading of the entire program. Similarly,
small errors in coloration or design in one object should
not force the "recompilation" of an entire image.
Separating the image into elements which can be
independently rendered saves enormous time. Each element has an associated matte, coverage information
which designates the shape of the element. The eompositing of those elements makes use of the mattes to accumulate the final image.
The compositing methodology must not induce aliasing in
the image; soft edges of the elements must be honored incomputing the final image. Features should be provided
to exploit the full associativity of the compositing process; this affords flexibility, for example, for the accumulation of several foreground elements into an aggregate
foreground which can be examined over different backgrounds. The compositor should provide facilities forarbitrary dissolves and fades of elements during an
animated sequence.
Several highly successful rendering algorithms haveworked by reducing their environments to pieces that canbe combined in a 2 1/2 dimensional manner, and thenoverlaying them either front-to-back or back-to-front [3].
Whitted and Weimar's graphics test-bed [6] and Crow's
image generation environment [2] are both designed todeal with heterogenously rendered elements. Whitted
253

@SIGGRAPH'84

and Weimar's system reduces all objects to horizontal
spans which are eomposited using a Warnock-like algorithm. In Crow's system a supervisory process decides
the order in which to combine images created by independent special-purpose rendering processes. The imaging
system of Warnock and W y a t t [5] incorporates 1-bit
mattes. The Hanna-Barbera cartoon animation system[4] incorporates soft-edge mattes, representing the opacity
information in a less convenient manner than that proposed here. The present paper presents guidelines for
rendering elements and introduces the algebra for compositing.

darkened colors for fractional alphas along edges, andblack where alpha is 0. Silhouette edges of RGBA elements thus exhibit their anti-aliased nature when viewed
on an RGB monitor.

2. The Alpha Channel
A separate component is needed to retain the matte
information, the extent of coverage of an element at a
pixel. In a full color rendering of an element, the RGB
components retain only the color. In order to place theelement over an arbitrary background, a mixing factor isrequired at every pixel to control the linear interpolationof foreground and background colors. In general, there is
no way to encode this component as part of the color
information. For anti-aliasing purposes, this mixing factor needs to be of comparable resolution to the color
channels. Let us call this an alpha channel, and let us
treat an alpha of 0 to indicate no coverage, 1 to mean fullcoverage, with fractions corresponding to partial coverage.

3. R G B A P i c t u r e s
If we survey the variety of elements which contribute to acomplex animation, we find many complete backgroundimages which have an alpha of 1 everywhere. Among
foreground elements, we find that the color components
roll off in step with the alpha channel, leaving large areasof transparency. Mattes, colorless stencils used for controlling the compositing of other elements, have 0 in their
RGB components. Off-line storage of RGBA pictures
should therefore provide the natural data compression forhandling the RGB pixels of backgrounds, RGBA pixels of
foregrounds, and A pixels of mattes.

In an environment where the compositing of elements isrequired, we see the need for an alpha channel as anintegral part of all pictures. Because mattes are naturally
computed along with the picture, a separate alpha component in the frame buffer is appropriate. Off-line
storage of alpha information along with color works conveniently into run-length encoding schemes because thealpha information tends to abide by the same runs.
What is the meaning of the quadruple (r,g,b,a) at a pixel?
How do we express that a pixel is half covered by a full
red object? One obvious suggestion is to assign (1,0,0,.5}
to that pixel: the .5 indicates the coverage and the (1,0,0)
is the color. There are a few reasons to dismiss this proposal, the most severe being that all compositing operations will involve multiplying the 1 in the red channel by
the .5 in the alpha channel to compute the red contribution of this object at this pixel. The desire to avoid thismultiplication points up a better solution, storing the
pre-multiplied value in the color component, so that(.5,0,0,.5) will indicate a full red object half covering apixel.
The quadruple (r,g,b,a) indicates that the pixel is acovered by the color ( r / a , g / a , b/a). A quadruple where
the alpha component is less than a color component indicates a color outside the [0,1] interval, which is somewhat
unusual. We will see later that luminescent objects can be
usefully represented in this way. For the representationof normal objects, an alpha of 0 at a pixel generally
forces the color components to be 0. Thus the RGB
channels record the true colors where alpha is 1, linearly254

It is important to distinguish between two key pixel
representations:
black ~- (0,0,0,1);
clear-~ (0,0,0,0).
The former pixel is an opaque black; the latter pixel istransparent.

There are some objections to computing with these
RGBA pictures. Storage of the color components premultiplied by the alpha would seem to unduly quantize
the color resolution, especially as alpha approaches 0.
However, because any compositing of the picture will
require that multiplication anyway, storage of the product forces only a very minor loss of precision in thisregard. Color extraction, to compute in a different color
space for example, becomes more difficult. We must
recover (r/a, g / a , b / a ) , and once again, as alphaapproaches 0, the precision falls off sharply. For ourapplications, this has yet to affect us.

4. The Algebra of Compositlng
Given this standard of RGBA pictures, let us examine
how compositing works. We shall do this by enumerating
the complete set of binary compositing operations. For
each of these, we shall present a formula for computing
the contribution of each of two input pictures to the output composite at each pixel. We shall pay particularattention to the output pixels, to see t h a t they remain
pre-multiplied by their alpha.

4.1. Assumptions
When blending pictures together, we do not have information about overlap of coverage information within apixel; all we have is an alpha value. When we consider
the mixing of two pictures at a pixel, we must make someassumption about the interplay of the two alpha values.
In order to examine that interplay, let us first consider
the overlap of two semi-transparent elements like haze,then consider the overlap of two opaque, hard-edged elements.

Computer Graphics

If a A and aB represent the opaqueness of semitransparent objects which fully cover the pixel, the computation is well known. Each object lets ( l - a ) of the
background through, so that the background shows
through only (1-aA)(1-aB) of the pixel, a A ( l - a ~ ) of the
background is blocked by object A and passed by object
B; (1-~A)a B of the background is passed by A andblocked by B. This leaves OlAOlB of the pixel which we
can consider to be blocked by both.
If olA and a B represent subpixel areas covered by opaquegeometric objects, the overlap of objects within the pixel
is quite arbitrary. We know that object A divides the
pixel into two subpixel areas of ratio C~A:l-a A. We knowthat object B divides the pixel into two subpixel areas ofratio crB:l-cr13. Lacking further information, we make thefollowing assumption: there is nothing special about theshape of the pixel; we expect that object B will divide eachof the subpixel areas inside and outside of object A into
the same ratio a/3:l-a B. The result of the assumption is
the same arithmetic as with semi-transparent objects andis summarized in the following table:
description
A(']B
A~B
AOB
Af']B

area
(l-at)(1-aB)
OtA(1-otB)(1-aA)a B
C~Aa B

The assumption is quite good for most mattes, though it
can be improved if we know that the coverage seldom
overlaps (adjacent segments of a continuous line) oralways overlaps (repeated application of a picture). Forease in presentation throughout this paper, let us makethis assumption and consider the alpha values asrepresenting subpixel coverage of opaque objects.

4.2. Compositing Operators
Consider two pictures A and B. They divide each pixel
into the 4 subpixel areas
B
0
0
1
1

A
0
1
0
1

name
0
A
B
AB

descriptionmf"]B_
A_~B
AAB
ANB

choices0
O, A
O, B
O, A, Blisted in this table along with the choices in each area forcontributing to the composite. In the last area, for example, because both input pictures exist there, either could
survive to the composite. Alternatively, the composite
could be clear in that area.
A particular binary compositing operation can be
identified as a quadruple indicating the input picture
which contributes to the composite in each of the four
subpixel areas 0, A, B, AB of the table above. With
three choices where the pictures intersect, two where only
one picture exists and one outside the two pictures, there
are 3 X 2 X 2 X 1~---12 distinct compositing operations listed

Volume18, Number 3

July 1984

in the table below. Note that pictures A and B are
diagrammed as covering the pixel with triangular wedges
whose overlap conforms to the assumption above.

operation

quadruple

clear

diagramFAFI?

(0,0,0,0)0

0A(0,A,0,A)

1

0

B(O,O,B,B)

A over B(0,A,B,A)

1

1-a A

B over A(0,A,B,B)

1-a B

1

A in B(0,0,0,A)aB

0

B in A(0,0,0,B)

0aA

A out B

(0,A,0,0)

1-a B

0

B out A

(0,0,B,0)

0

l-a A

A atop B(0,0,B,A)

~B

1-aA

B atop A(0,A,0,B)

~

1-a B

o~A

A xor B

(0,A,B,0)

~

l-a B

1-a A

V

~

Useful operators include A o v e r / 5 , A in B, and A heldout b y B. A over B is the placement of foreground A in
front of background B. A in B refers only to that part of
A inside picture B. A h e l d o u t b y B, normally shortened to A o u t B, refers only to that part of A outside
picture B. For completeness, we include the less usefuloperators A atop B and A x o r B. A a t o p B is the unionof A in B and B o u t A. Thus, paper a t o p table includes
paper where it is on top of table, and table otherwise; area
beyond the edge of the table is out of the picture.
A xor B i s the union of A out B and B o u t A.

255

@SI6GRAPH'84i4.3. C o m p o s i t i n g Arithmetic

4.4. U n a r y operators

For each of the compositing operations, we would like tocompute the contribution of each input picture at eachpixel. This is quite easily solved by recognizing that each
input picture survives in the composite pixel only within
its own matte. For each input picture, we are looking forthat fraction of its own matte which prevails in the output. By definition then, the alpha value of the composite,
the total area of the pixel covered, can be computed byadding aA times its fraction F A to aB times its fraction

To assist us in dissolving and in balancing color brightness of elements contributing to a composite, it is useful
to introduce a darken factor ¢ and a dissolve factor 6:

The color of the composite can be computed on a component basis by adding the color of the picture A times
its fraction to the color of picture B times its fraction.
To see this, let CA, cB, and c o be some color componentof pictures A, B and the composite, and let CA, CB, andC o be the true color component before pre-multiplicationby alpha. Then we have
C0 ~

OLoC0

Now C o can be computed by averaging contributions
made by C A and CB, so
Co = o~o.

OIAFACA-I-OeBFBCB
aAFA+O~BFB

but the denominator is just a o , so~o = " A F A C a + . B F , %
CA

:

CB

CAFA-FCBFB(1)

Because each of the input colors is pre-multiplied by its
alpha, and we are adding contributions from nonoverlapping areas, the sum will be effectively premultiplied by the alpha value of the composite just computed. The pleasant result that the color channels arehandled with the same computation as alpha can betraced back to our decision to store pre-multiplied RGBA
quadruples. Thus the problem is reduced to finding atable of fractions F A and F B which indicate the extent ofcontribution of A and B, plugging these values into equation 1 for both the color and the alpha components.
By our assumptions above, the fractions are quickly
determined by examining the pixel diagram included in
the table of operations. Those fractions are listed in the
F A and F B columns of the table. For example, in the
A over B case, picture A survives everywhere while picture B survives only outside picture A, so the corresponding fractions are 1 and (1-~A). Substituting into equation 1, we findco :CAX l + c B X (1-aA).

This is almost the well used linear interpolation of foreground F with background B/~ = F X c ~ + B x ( 1 - a ) ,except that our foreground is pre-multiplied by alpha.

256

d a r k e n ( A , ¢ ) = ( ¢ r A,~gA,dPbA,Ol A). dlssolve(A,6)----(6rA,6gA,6bA,6OtA) .

Normally, 0<~,6_~1 although none of the theory requiresit.
As ~ varies from 1 to 0, the element will change fromnormal to complete blackness. If 4 > 1 the element will
be brightened. As 6 goes from 1 to 0 the element will
gradually fade from view.
Luminescent objects, which add color information
without obscuring the background, can be handled with
the introduction of a opaqueness factor w, 0 < w < 1:opaque(A,w)----(rA,gA, bA,W°tA) •As w varies from 1 to 0, the element will change fromnormal coverage over the background to no obscuration.
This scaling of the alpha channel alone will cause pixel
quadruples where (~ is less than a color component, indicating a representation of a color outside of the normal
range. This possibility forces us to clip the output composite to the [0,1] range.
An w of 0 will produce quadruples (r,g,b,O) which do havemeaning. The color channels, pre-multiplied by the original alpha, can be plugged into equation 1 as always. The
alpha channel of 0 indicates that this pixel will obscurenothing. In terms of our methodology for examining subpixel areas, we should understand that using the opaque
operator corresponds to shrinking the matte coveragewith regard to the color coverage.

4.5. T h e PLUS o p e r a t o r
We find it useful to include one further binary compositing operator p l u s . The expression A plus B holds nonotion of precedence in any area covered by both pictures; the components are simply added. This allows us
to dissolve from one picture to another by specifying

dlssolve(A,~) plus dissolve(B,l-(~).
In terms of the binary operators above, p l u s allows
both pictures to survive in the subpixel area AB. The
operator table above should be appended:
operation(0,A,B,AB) ~A plus B

diagram

FA

FB

~

1

I

Computer Graphics

5. E x a m p l e s
The operations on one and two pictures are presented asa basis for handling compositing expressions involving
several pictures. A normal case involving three pictures
is the compositing of a foreground picture A over a background picture B, with regard to an independent matte
C. The expression for this compositing operation is(A in C) over B.
Using equation 1 twice, we find that the composite in thiscase is computed at each pixel by
co = e a ~ e + e d l - ~ A ~ C ) .

As an example of a complex compositing expression, let
us consider a subwindow of Rob Cook's picture Road toVolume18, Number 3

July 1984

Point Reyes [1]. This still frame was assembled frommany elements according to the following rules:
Foreground ---- FrgdGrass o v e r Rock o v e r Fenceo v e r Shadow o v e r BkgdGrass;
GIossyRoad ~ Puddle o v e r (PostRefleetion a t o p( PlantRefleetion a t o p Road));Hillside ~ Plant o v e r GlossyRoad o v e r Hill;
Background -~- Rainbow plus Darkbow o v e r
Mountains o v e r Sky;,
Pt.Reyes = Foreground o v e r Hillside o v e r Background.
Figure 1 shows three intermediate composites and thefinal picture.

Foreground ~ FrgdGrass o v e r Rock o v e r Fenceo v e r Shadow o v e r BkgdGrass;

Hillside ~ Plant o v e r GlossyRoad o v e r Hill;

Background = Rainbow plus Darkbow o v e r
Mountains o v e r Sky;,

Pt.Reyes ~-- Foreground o v e r Hillside o v e r Background.
Figure 1257

@SIGGRAPH'84

Planet

Stars

!
FFire

BFire

"(
BFire o u t Planet

Composite

Figure 2258Computer Graphics

A further example demonstrates the problem of correlated mattes. In Figure 2, we have a star field background, a planet element, fiery particles behind theplanet, and fiery particles in front of the planet. We wish
to add the luminous fires, obscure the planet, darkened
for proper balance, with the aggregate fire matte, andplace that over the star field. An expression for this eompositing is

Volume18, Number 3

July 1984

There are several problems to be resolved in relatedareas, which are open for future research. We areinterested in methods for breaking arbitrary three dimensional scenes into elements separated in depth. Such elements are equivalent to clusters, which have been a subject of discussion since the earliest attempts at hidden
surface elimination. We are interested in applying thecompositing notions to Z-buffer algorithms, where depth
information is retained at each pixel.

(FFire p l u s (BFire o u t Planet))o v e r darken(Planet,.8) o v e r Stars.
7. R e f e r e n c e s

We must remember that our basic assumption about thedivision of subpixel areas by geometric objects breaks
down in the face of input pictures with correlated mattes.
When one picture appears twice in a compositing expression, we must take care with our computations of FA andF B. Those listed in the table are correct only for uneorrelated pictures.
To solve the problem of correlated mattes, we must
extend our methodology to handle n pictures: we must
examine all 2" subareas of the pixel, deciding which of
the pictures survives in each area, and adding up all contributions. Multiple instances of a single picture or pictures with correlated mattes are resolved by aligning their
pixel coverage. Example 2 can be computed by building
a table of survivors (shown below) to accumulate theextent to which each input picture survives in the composite.
FFire

BFire

PlanetStars

Survivor
Stars
PlanetPlanet
BFire
BFire
PlanetPlanet
FFire
FFire
FFire
FFire
FFire,BFire
FFire,BFire
FFire
FFire

6. C o n c l u s i o n
We have pointed out the need for matte channels in synthetic pictures, suggesting that frame buffer hardware
should offer this facility. We have seen the convenienceof the RGBA scheme for integrating the matte channel. A
language of operators has been presented for conveying afull range of compositing expressions. We have discussed
a methodology for deciding compositing questions at the
subpixel level, deriving a simple equation for handling allcomposites of two pictures. The methodology is extended
to multiple pictures, and the language is embellished tohandle darkening, attenuation, and opaqueness.

1.

Cook, R. Road to Point Reyes. Computer Graphics
Vol 17, No. 3 (1983), Title Page Picture.

2.

Crow, F. C. A More Flexible Image Generation
Environment. Computer Graphics Vol. 16, No. 3(1982), pp. 9-18.
Newell, M. G., Newell, R. G., and Saneha, T. L.. A
Solution to the Hidden Surface Problem, pp. 443448. Proceedings of the 1972 ACM National Confer-

3.

ence.
4.

5.

6.

Wallace, Bruce. Merging and Transformation ofRaster Images for Cartoon Animation. Computer
Graphics Vol. 15, No. 3 (1981), pp. 253.262.
Warnoek, John, and Wyatt, Douglas. A Device
Independent Graphics Imaging Model for Use with
Raster Devices. Computer Graphics Vol. 16, No. 3(1982), pp. 313-319.
Whitted, Turner, and Weimer, David. A Software
Test-Bed for the Development of 3-D Raster Graphies Systems. Computer Graphics Vol. 15, No. 3(1981), pp. 271-277.

8. A c k n o w l e d g m e n t •The use of mattes to control the compositing of pictures
is not new. The graphics group at the New York Institute of Technology has been using this for years. NYIT
color maps were designed to encode both color and matte
information; that idea was extended in the Ampex AVA
system for storing mattes with pictures. Credit should begiven to Ed Catmull, Alvy Ray Smith, and Ikonas Graphics Systems for the existence of an alpha channel as anintegral part of a frame buffer, which has paved the way
for the developments presented in this paper.
The graphics group at Lucasfilm should be credited with
providing a fine test bed for working out these ideas.
Furthermore, certain ideas incorporated as part of thiswork have their origins as idle comments within thisgroup. Thanks are also given to Rodney Stock for comments on an early draft which forced the authors to clarify the major assumptions.

259