The Great Buddha Project:
Modelling Cultural Heritage throughObservation
Daisuke Miyazaki*, Takeshi Ooishi*, Taku Nishikawa*, Ryusuke Sagawa*, Ko Nishino*
Takashi Tomomatsu**, Yutaka Takase**, Katsushi Ikeuchi** Institute of Industrial Science, University of Tokyo
7-22-1, Roppongi, Minato-ku, Tokyo 106-8558 Japan** CAD CENTER Corporation
1-7-16, Sendagaya, Shibuya-ku, Tokyo 151-0051 Japan
Abstract. This paper presents an overview of our efforts in modeling cultural heritage through observation.
These efforts span three aspects: how to create geometric models of cultural heritage; how to create
photometric models of cultural heritage; and how to integrate such virtual heritages with real scenes. For
geometric model creation, we have developed a two-step method: simultaneous alignment and volumetricview merging. For photometric model creation, we have developed the eigen-texture rendering methods,
which automatically create photorealistic models by observing the real objects. For the integration of virtualobjects with real scenes, we have developed a method that renders virtual objects based on real illumination
distribution. We have applied these component techniques to constructing a multimedia model of the Great
Buddha of Kamakura, and demonstrated their effectiveness.

1. Introduction
One of the most important research issues in virtual reality is how to obtain models forvirtual reality. Currently such models are manually created by a human programmer. Thismanual method requires long developing time and resulting virtual reality systems areexpensive. In order to overcome this problem, we have been developing techniques tocreate virtual reality models through observation of real objects; we refer to thesetechniques as modeling-from-reality. As shown in Figure 1, the modeling-from-reality
spans three aspects: how to create geometric models of virtual objects; how to create
photometric models of virtual objects; and how to integrate such virtual objects with realscenes.
Geometric Modelingpartial views

Photometric Modelingcolor images
Environmental Modelingenvironmental
map

Figure 1 Three Components in modeling-from-reality

In this paper, we first overview the modeling-from-reality techniques in Chapter 2. Then,in Chapter 3, we explain how these component techniques were applied to model the Great
Buddha of Kamakura, what kind of unforeseen difficulties are encountered, and how wesolve them. Chapter 4 contains a summary of the paper.
2. Modeling from Reality
2.1 Geometric modeling

Several computer vision techniques, such as traditional shape-from-X and binocularstereo, or modern range sensors, provide cloud of point information. The cloud of point
information certainly carries three-dimensional information pertaining to observed objects.
However, there is no structural information among those points. Namely, there is noinformation to represent adjacency among the points. The first step of geometric modelingis to convert this cloud of points with a surface representation such as a mesh model asshown in Figure 2.
Since each observation provides only partial information, we have to connect these partial
mesh representations into a whole geometric mesh representation. Thus, the second step ingeometric modeling is to align these meshes so that the corresponding parts overlap one
another. The third geometric modeling step is to merge these aligned data by weighing thereliability of each data point. For this view merging, we have developed a stochastic
volumetric merging method.
mesh
generation

simultaneousalignmentalignment
merging

level set

Figure 2 A three step method

2.2 Photometric Modeling

Currently, most VR (virtual reality) systems utilize image-based rendering [2,5,10]. The
image-based rendering samples a set of color images of a real object and stores them on
the disk of a computer. A new image is then synthesized either by selecting an appropriate
image from the stored set or by interpolating multiple images. Apple's QuickTime VR is
one of the earlier successful image-based rendering methods. Image-based rendering does
not assume any reflectance characteristics of objects nor does it require any detailed
analysis of the reflectance characteristics of the objects; rather, the method need only totake images of an object. On the other hand, image-based methods have critical
disadvantages on application to mixed reality. Few image-based rendering methods
employ accurate 3D models of real objects. Thus, it is difficult to make cast shadows underreal illuminations corresponding to the real background-image.
We propose a new rendering method, which we refer to as the eigen-texture rendering
method [12,13]. First, the eigen-texture rendering method creates a 3D model of an object

from a sequence of range images. Second, the method aligns and pastes color images of theobject onto the 3D surface of the object model. Third, the method compresses those images
in the coordinate system defined on the 3D-model surface. This compression isaccomplished using the eigenspace method. The synthesis process is achieved using theinverse transformation of the eigenspace method. Cast shadows are generated using the 3Dmodel. A virtual image under a complicated illumination condition is generated bysummation of component virtual images sampled under single illuminations thanks to thelinearity of image brightness.
2.3 Environmental Modeling

Virtual objects are usually displayed with background by superimposing them onto areal/virtual background. For superimposing virtual objects onto an appropriate background[1,20], geometric and photometric aspects have to be taken into account; for environmental
modeling, the virtual object has to be located at a desired location in the real scene, and theobject must appear at the correct location in the image. At the same time, shading of thevirtual object has to match that of other objects in the scene, and the virtual object must
cast a correct shadow, i.e., a shadow whose characteristics are consistent with those of theshadows in the real scene.
3. Modeling the Great Buddha of Kamakura

By using the component techniques developed in the modeling-from-reality project, wehave begun to obtain virtual models of historic heritage in Japan. We Japanese occupy aunique position in this respect. Most Japanese cultural heritages are made of wood andpaper; thus, at any moment, they could be lost due to fire or water damage. It is importantto create digitized representations of these cultural assets in order to preserve them forposterity.
As the starting point of the project, we obtain the geometric information of the Great
Buddha of Kamakura. This Buddha was built in the thirteen century originally of wood,then of bronze. As a matter of fact, Japanese built a hall of the Great Buddha. It wasdestroyed three times by tidal waves. Figure 3 shows the current Great Buddha ofKamakura without such a hall.

Figure 3 The Great Buddha of Kamakura3.1 Geometric modeling
Figure 4 shows the flow of geometric modeling of the Great Buddha. The flow consists ofscanning, alignment, and merging.

Fig.4-1 Measuring with range sensor
Fig.4-2 Range data from several directions

Fig.4-3 Alignment

Fig.4-4 Merging

Figure 4 Modeling flow

Scanning
To obtain a virtual model of the Great Buddha, we first obtained twenty-four range
images of the Buddha by using a Cyrax range scanner. In order to scan the upper part ofthe Buddha, we built a scaffold and mounted the sensor on it as shown in Figure 4-1. This
work was done during the night to avoid sightseers.
Alignment
A simultaneous alignment method has been developed to avoid accumulation of errors.
Traditional sequential methods such as ICP, align these meshes one by one, andprogressively align a new partial mesh with previously aligned meshes. If a few partialmeshes can cover an object, accumulation of alignment errors is relatively small and canbe ignored. A sequential alignment works well. However, some cultural heritages are verylarge; we need more than twenty views. In this case, the error accumulation caused bysequential alignment methods is very large. Thus, by considering alignment of all the pairs
simultaneously, we align all partial meshes so as to reduce the errors among all the pairs
simultaneously.
Merging
After aligning all range images, a volumetric view-merging algorithm generates aconsensus surface of the Buddha from them. Our method merges a set of range images intoa volumetric implicit-surface representation which is converted to a surface mesh using avariant of the marching-cubes algorithm. Unlike previous techniques based on implicitsurface representations, our method estimates the signed distance to the object surface byfinding a consensus of locally coherent observations of the surface. We utilize octrees torepresent volumetric implicit surfaces -- effectively reducing the computation and memory
requirements of the volumetric representation without sacrificing accuracy of the resulting
surface.
We made up software in order to merge the aligned 20 range data. Since the input data isunpredictably huge, we built up a PC cluster to run this merging software, which parallelprocesses the merging algorithm for saving the computation time and utilizing a large
memory space of many PCs. We made one integrated digital Great Buddha with thissoftware. Figure 4-4 shows the obtained virtual Buddha.
Once we obtain the geometric information of the Great Buddha, we can easily determine
its cross-sectional shapes as shown in Figure 5. This is important information to comparewith several shapes of the Great Buddha including Kamakura Buddha and Nara Buddha.
We plan to scan the Great Buddha of Nara this fall.

Figure 5 Cross-Sectional Shape of the Great Buddha

3.2 Photometric Modeling and Environmental Modeling
Using merged data of the Great Buddha, we made CG animation and finally produced aDVD work on the Great Buddha of Kamakura. The DVD work includes restored CG imagesof original wooden Buddha which was completed in 1243 and disappeared soon, as well asgold-leaf covered bronze Buddha which was made in 1260s. We also attempted to restore theMain Hall of the Great Buddha which was also completed in 1243.
Compared to the fact that the Great Buddha of Todai-ji temple in Nara was destroyed
through earthquakes and wars, and only little part of the Great Buddha remains presently, theGreat Buddha of Kamakura mostly remains since it was originally made, keeping enormous
value as a national heritage. When we made CG animation of the Great Buddha of Kamakura,
we reduced the merged data of the Great Buddha from around a few million polygons to 100
thousand polygons, and mapped it with textures of wood and gold-leaf.
Though the Great Buddha of Kamakura is now placed open-air, historical documentsshows that the Main Hall of the Great Buddha once existed. According to available data, atleast four halls were constructed for the Great Buddha. And after the hall was destroyed by
the tidal wave in 1498, it was never rebuilt.
In restoring the Main Hall of the Great Buddha, we received advice and assistance from
Professor Emeritus Kiyoshi Hirai of the Tokyo Institute of Technology, an expert on
historical architecture. The Main Hall of the Great Buddha was designed in “Daibutsu-yo,”
the same style as other great buildings of that time such as the Main Hall of Todai-ji temple
in Nara which was reconstruced in Kamakura era. The design of the hall was based on
drawings of the Main Hall at Todai-ji temple, that was renovated during the Kamakura
period. Other models for the design included the Jodo-do hall of Jodo-ji temple in Hyogo.
We modelled the data of the Main Hall using 3D CAD software ended up with 300 thousand
poligons.

Figure 6 Drawings of Main Hall, Todai-ji, reconstructed in Kamakura era(by Minoru Ooka)

Figure 7 Drawings of Jodo-do, Jodo-jiFigure 8 The Great Buddha of Kamakura in the Main HallIn addition, in measurements taken in the autumn of 1999, data on the inside of the Great
Buddha was also obtained and added to the outside data. At the same time, its thickness wasmeasured to within an accuracy of less than one mm by using supersonic waves. In the big
renovation project on the Great Buddha during the Showa era, the average thickness of thestatue was estimated to be around 50mm based on its weight and surface area. This latest
measurement showed that the most common thickness was 20-30 mm so it can be safely
assumed that the overall thickness of the statue is quite uneven. By using the available data,
more accurate three-dimensional data of the thickness of the statue was complied for taking
countermeasures against corrosion and structural weakness in the Great Buddha.
4. Summary
This paper presents an overview of our efforts in modeling-from-realty to create virtual
reality models of cultural heritage items through observation of real heritage items. Theseefforts span three aspects: how to create geometric models of virtual heritages; how tocreate photometric models of virtual heritages; and how to integrate such virtual heritages
with real scenes.
We scanned the Great Buddha of Kamakura with Cyrax laser range sensor to obtain theexact geometrical information of the Buddha for the purpose of preserving its culturalheritage. We developed a system for obtaining the 3D geometric data automatically. The
alignment software we developed sticks multiple range images taken by laser range sensor
from many directions to the ideal position where those images represent the actual object.
The program minimizes the geometrical difference of the covered part of each two images;
this works simultaneously with all images until the whole differences minimize or until wethink the differences are sufficiently minimized. We designed a program for merging thealigned range images into a mesh model. Since the aligned range images themselves have
non-integrated geometric information, we cannot analyze the geometric features of the object.
The merging software did a great job of obtaining the exact model of the scanned object. Theold literature says that there once existed a building like the Great Buddha of Nara. Werestored the Main Hall of the Great Buddha of Kamakura from many pieces of literature andmany experts’ technical knowledge, and then made up a computer graphic of how the Great
Buddha looked like in 1267.
M. Levoy and his team members of Stanford University are also carrying out a project
similar to ours [21]. They digitized some statues created by Michelangelo. However, thosestatues are relatively small (around 5m) when compared with the Great Buddha (around 15m).
Due to that size difference, we encounter a large number of data points. We overcome thisdifficulty by using parallel merging algorithm. Another difference is that the Michelangelo
statues are indoors and therefore less affected by sunlight. We have to employ a more
powerful range sensor to measure the Great Buddha.
We plan to further extend this project through developing new sensors as well as refining
algorithms in order to overcome problems encountered in completing the Great Buddha
project.
Acknowledgements
This work is supported, in part, by Ministry of Education under Shin-Pro 09NP1401, in part, by JST underIkeuchi CREST project, and, in part, by IPA. Thanks go to to Kotokuzi-temple for allowing us to digitize theGreat Buddha.

References[1] R. Azuma, "A survey of augmented reality, " Presence, vol. 6, no. 4, pp. 355-385, August 1997.
[2] M. Bajura, H. Fuchs, and R. Ohbuchi, "Merging virtual objects with the real world," Proc. SIGGRAPH
92, pp. 203-210.
[2] S. Chen, "Quicktime VR," Proc. SIGGRAPH 95, pp.29-38.
[3] B. Curless and M. Levoy, "A volumetric method for building complex models from range images," Proc.
SIGGRAPH 96, pp. 303-312.
[4] A. Fournier, A. Gunawan and C. Romanzin, "Common Illumination between Real and Computer
Generated Scenes," Proc. Graphics Interface '93, pp.254-262.
[5] S. Gortler, R. Grzeszczuk, R. Szeliski, and M. F. Cohen, "The lumigraph," Proc. SIGGRAPH 96, pp. 4354.
[6] H. Hoppe, T. DeRose, T. Duchamp, J. McDonald, and W. Stuetzle, "Surface reconstruction fromunorganized points," Proc. SIGGRAPH 92, pp. 71-78.
[7] K. Ikeuchi and K. Sato, "Determining reflectance properties of an object using range and brightness
images," IEEE Trans. PAMI, 13(11): 1139-1153, 1991.
[8] G. Kay and T. Caelli, Inverting an illumination model from range and intensity maps, CVGIP: IU, vol. 59,pp. 183-201, 1994.
[9] G. K. Klinker, S. Shafer, and T. Kanade, The measurement of highlights in color images, Int. J. Comp.
Vision, 2: 7-32, 1988.
[10] M. Levoy and P. Hanrahan, "Light field rendering," Proc. SIGGRAPH 96, pp. 31-42.
[11] S. Nayar, K. Ikeuchi, and T. Kanade, "Surface reflection," IEEE Trans. PAMI, 13(7): 611-634, 1991.
[12] K. Nishino, Y. Sato, and K. Ikeuchi, "Eigen-texture method," IEEE Conf CVPR, pp.618-622, June 1999.
[13] K. Pulli, M. Cohen, T. Duchamp, H. Hoppe, L. Shapiro, and W. Stuetzle, "View-based rendering," Proc.
8th Eurographics Workshop on Rendering, June 1997.
[14] K. Torrance and E. Sparrow, "Theory for off-specular reflection from roughened surface," J. Opt.
Society of America, 57: 1105-1114, 1967.
[15] I. Sato, Y. Sato, and K. Ikeuchi, "Acquiring a radiance distribution to superimpose virtual objects onto areal scene," IEEE Trans Visualization and Computer Graphics, Vol. 5, No. 1, pp.1-12, January 1999.
[16] I. Sato, Y. Sato, and K. Ikeuchi, "Illumination distribution from shadows," IEEE Conf. CVPR, pp.306312, June 1999.
[17] Y. Sato and K. Ikeuchi, "Temporal-color space analysis of reflection," J. Opt. Society of America A,
11(11): 2990-3002, November 1994.
[18] Y. Sato, M. Wheeler, and K. Ikeuchi, "Object shape and reflectance modeling from observation," Proc.
SIGGRAPH 97, pp.379-387.
[19] M. Wheeler, Y. Sato, and K. Ikeuchi, "Consensus surfaces for modeling 3D objects from multiple rangeimages," Proc. 6th Int. Conf. Comp. Vision, pp. 917-924, 1998.
[20] Z. Zhang, "Modeling geometric structure and illumination variation of a scene from real images," Proc.
6th Int. Conf. Comp. Vision, pp.1041-1046, 1998.
[21] M. Levoy et. al. “The Digital Michelangelo Project” SIGGRAPH 00, pp.131-144.