December 12, 2007 20:55 WSPC/115-IJPRAISPI-J068 00599

International Journal of Pattern Recognition
and Artiﬁcial IntelligenceVol. 21, No. 8 (2007) 1307–1322
c World Scientiﬁc Publishing Company


REPRESENTATION OF 3-D VOLUMETRIC OBJECT
FROM THE PANTOMIME EFFECT AND
SHADING CUES IN HUMAN BRAIN

QI ZHANG∗ and KEN MOGI†Sony Computer Science Laboratories, Inc.
Takanawa Muse Bldg., 3-14-13, Higashigotanda
Shinagawa-ku, Tokyo 141-0022, Japan∗zhangq@csl.sony.co.jp†kenmogi@csl.sony.co.jp

Human ability to process visual information of outside world is yet far ahead of manmade systems in accuracy and speed. In particular, human beings can perceive 3-D object
from various cues, such as binocular disparity and monocular shading cues. Understanding of the mechanism of human visual processing will lead to a breakthrough in creating
artiﬁcial visual systems. Here, we study the human 3-D volumetric object perceptionthat is induced by a visual phenomenon named as the pantomime eﬀect and by the
monocular shading cues. We measured human brain activities using fMRI when thesubjects were observing the visual stimuli. A coordinated system of brain areas, including those in the prefrontal and parietal cortex, in addition to the occipital visual areas
was found to be involved in the volumetric object perception.
Keywords: 3-D volumetric object; perception; pantomime eﬀect; shading cues; fMRI.

1. Introduction
Human ability to process visual information of outside world is yet far ahead ofman-made systems in accuracy and speed. Understanding of the mechanism of
human visual processing will lead to a breakthrough in creating smarter artiﬁcial
visual systems and accomplishing more eﬃcient pattern recognition in the same
way as human beings. Object perception is an important issue with long history
in vision research. In particular, volumetric object perception that brings about
the perception of thickness is one of the interesting topics in object perception. 3-Dvolumetric objects can be perceived via many types of inducing cues. The monocular
cues such as shading, texture, are the natural ones to depict 3-D volumetric objects
in paint works and photos, and are widely used in the computer graphics. On
the other hand, binocular disparity is one of the important cues to perceive 3-Dvolumetric objects in the real world.
Several previous researches6,10,12,13 reported that the lateral occipital complex(LOC) area is involved in the object perception. In addition, researches on binocular
1307

December 12, 2007 20:55 WSPC/115-IJPRAI1308

SPI-J068 00599

Q. Zhang & K. Mogi

stereopsis have reported that sensitivity to binocular disparity were found in many
extrastriate areas, including V2, V3, V3A, VP, MT, MST and IT.1,19 Brouwer
et al. reported that all visual areas responsive to disparity-deﬁned depth.2 Neurons
in V4,7 in middle temporal cortex14 and the caudal intraparietal sulcus18 are sensitive to the orientation of disparity-deﬁned planes, and neurons in inferior temporal
cortex (IT) are sensitive to the shape and curvature of disparity-deﬁned surfaces.８17
These ﬁt with the idea that higher brain areas extract more complicated featuresfrom the patterns of retinal disparities, and integrate stereo information into different perceptual systems.
On the other hand, a phenomenon named as “shape from shading” is well known,where the monocular shading cues can induce 3-D shape perception, and this perception is reported as an “early” visual process computed in the occipital areas andis mostly a bottom-up mechanism.11,17
Furthermore, objects are usually perceived based on the direct physical cues.
However, sometimes they are perceived in an alternative and indirect fashion, e.g.
through the hands of pantomimists in pantomimes. Previously, one of the authors
reported the pantomime eﬀect in visual perception,21 where a 3-D volumetricobject was induced by suitably related and stereoscopically displayed objects. These
objects were displayed with binocular disparities and were observed with binocular
viewing. The most signiﬁcant diﬀerence from the previous researches is that theperceived object is not only induced by the binocular disparities, but also inducedby the interrelationships among the inducing objects. What is the brain mechanism
for the 3-D object perception induced by the pantomime eﬀect and the monocularshading cues? To pursue the answer, we conducted the fMRI experiments to measure the brain activities when the subjects were observing these two types of visual
stimuli.
2. Experimental Methods
2.1. Subjects and experimental protocols
Five healthy right-handed volunteers (aged 23–30 years; male) with normal stereoscopic vision ability took part in the fMRI experiments. All subjects gave informed
written consent for participation. The subjects were tested in advance to ensurethat they could attain the 3-D stereoscopic perception under the experimental
conditions.
2.2. Stimuli
Stimuli from the pantomime eﬀect are used in our experiments. In the pantomime
eﬀect, a 3-D volumetric object is perceived from some stereoscopically displayed
inducing objects based on both binocular disparity and spatially distributed relations. The typical example of the stimuli in stereogram is shown in Fig. 1(a). The
inducing objects are suitably related and arranged in 3-D space without occlusion

December 12, 2007 20:55 WSPC/115-IJPRAISPI-J068 00599

Representation of 3-D Volumetric Object from the Pantomime Eﬀect

L

R

1309

L

(a)

With binocular disparities

Without binocular disparities
Non 3-D Object perception

3-D Object perception

Block

o

+

(b)
Fig. 1. (a) An example stereogram of the typical perception in the pantomime eﬀect for an intact
transparent sphere. (L: left eye image, R: right eye image; left and centre are for uncrossed fusion,
while centre and right are for crossed fusion.) (b) The conceptual ﬁgure for the six types of stimuli
used in the fMRI experiments. The ﬁrst two columns of images were displayed with disparities forthe two eyes, and subjects attained 3-D perception by observing through binocular glasses. The
other columns were displayed without disparities, namely the image for the left eye was the sameas that for the right eye, and the subjects obtain 2-D perception even when they observed throughbinocular glasses. In addition, for the left 3 columns of images, a 3-D object can be perceived,
while no 3-D object is perceived for the right 3 columns.

relationship, and they are perceived to sustain the 3-D illusory volumetric object
in the same way as the hands of pantomimists. In this sense, this phenomenon iscalled the pantomime eﬀect. The 3-D illusory object is perceived as a sphere with
its inside brightness diﬀerent from that of the surrounding space, as if it is “ﬁlled”
with transparent medium and to occupy volumetric space, not simply a spherical surface. There is no direct physical stimulus on the perceived sphere, not likea shaded sphere with direct brightness stimulus on it. As the comparison, a 3-Dvolumetric sphere induced by monocular shading cues was adopted.
Figure 1 shows the six types of stimuli which were applied in our fMRI
experiments. These stimuli include that for volume perception induced by bothpantomime eﬀect and monocular shading cues. The two left columns of images weredisplayed with disparities for the two eyes, and subjects stereoscopically attained
3-D perception by observing through binocular glasses. The other columns wereDecember 12, 2007 20:55 WSPC/115-IJPRAI1310

SPI-J068 00599

Q. Zhang & K. Mogi

displayed without disparities, namely the image for the left eye was the same asthat for the right eye. Subjects obtain 2-D perception even when they observed
through binocular glasses. For detailed description, the ﬁrst column represents thestimulus for the 3-D volumetric object. The upper one is for an intact transparent
sphere, and the lower is a hemisphere. The second column is an opaque volumetricobject and acted as a null stimulus in the experiments. The third column is the
3-D object induced by monocular shading cues. For the upper image, more cones
stand on the sphere, and the lower one with fewer cones. The fourth column acts as
a control stimulus of 2-D perception with the similar retina projection. The images
to both eyes are identical and the same as that for the left eye in the ﬁrst column,
thus only some triangles on a plane are perceived due to the absence of binoculardisparity. The ﬁfth column is the control stimulus where no 3-D sphere is perceived
except for the same cones as in the third column, and it is also a null stimulus in
the experiments. The last column is the ﬁxation point stimulus. The upper is a
circle, and the lower is a crosshair.
These stimuli were presented to the subjects using the Avotect Silent VisionSV-4021 ﬁber optic visual system, so that the subjects could easily attain thestereoscopic perception of the 3-D object induced by the binocular disparity. Subjects observed the visual stimuli through binocular glasses which were attached to
the Avotec SV-4021 2-channel separated ﬁber optic binocular display system. The
glasses were in SVGA resolution and ﬁxed on a standard head coil of the fMRI system, above the subject’s head when he/she was lying on his/her back. The stereo
stimuli with binocular disparity were created using OpenGL in the situation where asphere with the diameter of 110 mm was observed from 500 mm in distance, and theinterpupillary distance was set as 60 mm. The height of the cones was set as 21 mm,
and the diameter of the bases for cones was 12 mm. In the experiments, the stimuli
for the left eye and the right eye were displayed through independent 2-channel
glasses. The interpupillary distance and the refractive correction (focus) for eachpiece of the glasses were adjusted for each subject to ensure the correct fusion forthe stereoscopic perception and a clear observation. There was an eye relief (about
20 mm) from the eye to the eyepiece, and it was a little diﬀerent for each subject.
In addition, for the stereo stimuli, the cones were unsymmetically standing on thesurface of the sphere, pointing to a certain direction in 3-D space. Hence, it wasdiﬃcult to know the precise visual angle of the stimuli, but an approximate value
was 16◦ . The ﬁeld of view of the binocular glasses was 30◦ horizontal × 23◦ vertical(nominal). The stimuli were displayed in red color of [255, 0, 0] in RGB mode. For
the shading stimuli, two white lighting sources were introduced for lighting a sphereof diﬀused materials.

2.3. Experimental protocols
The six types of stimuli shown in Fig. 1(b) were divided in six blocks and eachblock contained two types of stimuli for the same kind of perception. Each stimulus

December 12, 2007 20:55 WSPC/115-IJPRAISPI-J068 00599

Representation of 3-D Volumetric Object from the Pantomime Eﬀect

1311

image was displayed in red color on a black background for two seconds in random
order, followed by a 0.5 s interval, and repeated six times in one block. One block
lasted 30 s with twelve images. One session consisted of all the six blocks repeating
three times, and lasted nine minutes. The blocks also appeared in random order in
one session.
The brain activities in which we are interested in this paper are those activatedby the visual perceptions. Because the subjects were lying on their backs in adark room, the pure observation task would make them feel sleepy. In order toprevent them from falling asleep, they were asked to do a simple task during theirobservation of the visual stimuli. Subjects clicked a right or left button when they
were presented the stimulus image each time. They clicked the left button when theupper image was displayed, and clicked the right button when the lower one wasdisplayed. The clicking ﬁngers were the right index and middle ﬁnger. The button
assignment was reminded again before each session began. Since the two images arequite diﬀerent, the subjects did not need any eﬀect to discriminate them. The motor
activity by the button press task should disappear by the subtraction between two
conditions during the fMRI data analyses.
2.4. fMRI data acquisition and analysis
EPI images were acquired with a Shimadzu-Marconi ECLIPSE 1.5T Power Drive
250 fMRI System, and the TR (repetition time) = 2.5 s, TE (echo time) = 55 ms.
T2*-weighted functional images with 5 mm thick slices were obtained in the resolution of 3 mm × 3 mm, and 23 slices were acquired in one scan. The EPI data wereanalyzed using the SPM5 software (Wellcome Department of Cognitive Neurology, University College London).3 Functional data were preprocessed by slice time
adjustment, realignment, coregistration, segmentation, normalization and smoothing. For the realignment, the estimation quality is 0.9; the separation and smoothing
parameters were 4 and 5; and 2nd Degree B-Spline was used for interpolation. In
the coregistration step where the structural and functional data were coregistered,normalized mutual information was used for the objective function of estimation;
the separation parameters were 4 and 2, and the tolerances and histogram were0.02 and 7, respectively. The coregistered T2 anatomical structural images weresegmented into gray and white matter images and bias-ﬁeld corrected structural
images were obtained. The spatial normalization parameters obtained in the segmentation were used to normalize both the functional and anatomical structural
images. Finally, a 6 mm full-width at half maximum (FWHM) of the isotropic Gaussian smoothing kernel was applied in smoothing. Fixed-eﬀects models were used at
the individual subject level of analysis, and random-eﬀects models were used forgroup-level analyses. A canonical HRF with time and dispersion derivatives was
adopted as the basis function in the individual level of analysis, and an autoregressive AR(1) model was used during classical parameter estimation to account for theserial correlations in fMRI time series due to aliased biorhythms and unmodelled
neuronal activities. In addition, the estimated time series data of translations andDecember 12, 2007 20:55 WSPC/115-IJPRAI1312

SPI-J068 00599

Q. Zhang & K. Mogi

rotations which were obtained in the preprocessing of realignment step were used
as regressors when ﬁtting GLMs, so that the movement eﬀects were discounted
when looking for brain activations. In the group level of analysis, an informed basis
set, consisting of the canonical HRF and its two partial derivatives with respect
to time (onset latency) and dispersion, was used. This used three observations per
subject and data were analyzed using a “One-way ANOVA” with three levels. The
t-contrast of the positive loadings on the canonical HRF were considered signiﬁcant
at p <0.003.
3. Experimental Results
3.1. 3-D volumetric object from the pantomime eﬀect vs 2-D
perception
In this paper, our interests are in the brain mechanism for 3-D perception. We ﬁrst
examine the cortical areas for the 3-D volume perception induced by the pantomime
eﬀect, and compared the activities when the subjects observed the stimuli for thevolume perception induced by the pantomime eﬀect (Fig. 1(b), 1st column) to thatfor 2-D perception (Fig. 1(b), 4th column). Figure 2 shows the results of the grouplevel analysis using the random-eﬀects models. We can see that besides the activities
in the occipital visual areas, the left parietal and prefrontal cortex were activated.
The signiﬁcant activated cerebral areas are listed in Table 1. The main activatedareas were located signiﬁcantly in bilateral occipital visual areas (Brodmann Areas(BA) 17 and 18). The anatomical locations were lingual gyrus, middle occipitalgyrus, and cuneus. The areas in the prefrontal cortex were located in the left BA 9
and 10 with the anatomical locations as the left medial and middle frontal gyrus,
and the left putamen. The right anterior cingulate (BA 24, 32 and 42) also showed
activations. In addition, the activated areas included the left angular gyrus and theleft inferior parietal lobule (BA 39).
3.2. Volumetric object from shading cues vs 2-D perception
In contrast to the 3-D perception induced by the pantomime eﬀect, where thebinocular disparity and the relationship among the inducing objects play important roles to the 3-D perception, the 3-D sphere perception induced by monocularshading cues results from the direct physical stimulus on the perceived sphere, suchas the variation in brightness. Though either type of stimuli brings about the 3-D
perception, we wonder if their perceptual mechanisms in the brain are diﬀerent.
Therefore, we examine a similar stimulus [3rd column in Fig. 1(b)] where a sphereis perceived with several 3-D objects (cones in Fig. 1) standing on it. Both thesphere and cones are perceived as 3-D objects induced by the monocular shadingcues without any binocular disparity. Because our interests are the brain activity
diﬀerences between the 3-D perception and 2-D perception, a 2-D control stimulus
is necessary in the fMRI experiments. We had considered several choices, such as

Hemisphere

Brodmann areas

3-D from shading cues > 2-D
LR

Middle Frontal Gyrus (L)Superior Frontal Gyrus (L)Precuneus (L)Superior Parietal Lobule (L)Fusiform Gyrus (L)Culmen (L)Middle Occipital Gyrus (L)Cuneus (L)Sub-Gyral (L)Fusiform Gyrus (L)Inferior Occipital Gyrus (L)Inferior Temporal Gyrus (L)
Lingual Gyrus (L)Middle Occipital Gyrus (L)Middle Temporal Gyrus (L)Middle Frontal Gyrus (R)Medial Frontal Gyrus (L)Middle Frontal Gyrus (L)Putamen (L)Angular Gyrus (L)Inferior Parietal Lobule (L)
Lingual Gyrus (L)Cuneus (L)Anterior Cingulate (R)Lingual Gyrus (R)Middle Occipital Gyrus (R)Cuneus (R) (C)

ROI

−36
−84
−66

−33
−21
−51

21

−63

−18

54

54

−33

36
−90

−93

−6
3
9

57
30
6
−66

y

−9
−33
−12
−51

x

36

−12

12

−27

60

0

12
0

−9

6
36
−9
39

z

6.37

6.59

4.92

4.96

5.20

3.98

4.03
6.07

4.05

3.79
3.37
3.27
3.69

T statistic

0.000

0.000

0.000

0.000

0.000

0.001

0.001
0.000

0.001

0.001
0.003
0.003
0.002

P value(uncorrected)

Representation of 3-D Volumetric Object from the Pantomime Eﬀect

BA 8/9 (R)BA 18/19/37 (L)BA 18 (L)BA 20/37 (L)BA 7 (L)BA 10 (L)BA 24/32/42 (R)BA 17/18 (R)BA 17/18 (L)BA 39 (L)

3-D from pantomime eﬀect > 2-D
L
BA 10 (L)BA 9 (L)Contrasts

MNI coordinates

Table 1. Statistical signiﬁcance of found clusters of activation for the 3-D perception induced by the pantomime eﬀect and the monocular shadingcues versus the 2-D perception.

December 12, 2007 20:55 WSPC/115-IJPRAISPI-J068 00599

1313

Sub-Gyral (R)Fusiform Gyrus (R)Inferior Occipital Gyrus (R)Inferior Temporal Gyrus (R)Parahippocampal Gyrus (R)Lingual Gyrus (R)Middle Occipital Gyrus (R)Middle Temporal Gyrus (R)Lingual Gyrus (R)Cuneus (R)BA 18/19/
20/37 (R)BA 17 (R)

Precuneus (R)Middle Temporal Gyrus (R)BA 7/19 (R)BA 21 (R)BA 40 (R)BA 7 (R)Supramarginal Gyrus (R)Precentral Gyrus (R)Middle Frontal Gyrus (R)Sub-Gyral (R)Superior Parietal Lobule (R)BA 9 (R)BA 6 (R)

Inferior Frontal Gyrus (R)Inferior Frontal Gyrus (R)Middle Frontal Gyrus (R)Superior Frontal Gyrus (R)Medial Frontal Gyrus (R)BA 47 (R)BA 10 (R)R

ROI

Brodmann areas

Hemisphere

6

27

9
66

60

30

42

3

42
33

x

33
39
−6
−9

−51
−78
−45
−69

0

60

−60

−87

42

39

−12
3

z

24

42

24
54

y

MNI coordinates

4.92

6.64

3.76
4.31

3.46

4.49

3.40

4.08

4.35
4.23

T statistic

0.000

0.000

0.001
0.001

0.002

0.000

0.003

0.001

0.001
0.001

P value(uncorrected)

MNI, Montreal Neurological Institute. x-, y-, and z-coordinates, T statistic, and P value apply to the most signiﬁcant voxel in each cluster.

Contrasts(Continued )1314

Table 1.

December 12, 2007 20:55 WSPC/115-IJPRAISPI-J068 00599

Q. Zhang & K. Mogi

December 12, 2007 20:55 WSPC/115-IJPRAISPI-J068 00599

Representation of 3-D Volumetric Object from the Pantomime Eﬀect

1315

(a)

A L

R

P
(b)
Fig. 2. fMRI experimental results of the group-level analysis using random-eﬀects models. The
color blobs represent the areas activated by the 3-D perception from the pantomime eﬀect versus the 2-D perception. Besides the activities in the occipital visual areas, the left parietal andprefrontal cortex was activated. (a) Activities rendered on a 3-D brain (t > 3.33, p < 0.003).
(b) Activities plotted in four slices with statistic data (t > 2.0). The number in the ﬁgure shows
the z value of MNI coordinates. A: Anterior, P: Posterior, L: Left, R: Right.

a ﬁxation point or some shaded cones but without a sphere. Since the physical differences in the stimuli between the 3-D perception and ﬁxation point are too large,
the diﬀerences in the brain activities could be considered due to the diﬀerence ofthe retina projection instead of the diﬀerence between the 3-D and 2-D perception.
As for the stimulus of several shaded cones, because the shading cues on the cones
even bring about 3-D perception for 3-D cones, it could not be used as the control of
2-D perception. Finally, we used the same 2-D control as the previous section where
neither binocular disparities nor shading cues are included [4th column in Fig. 1(b)].
We think that the diﬀerence in brightness between these two kinds of stimuli
exactly express the diﬀerent inducing ways for the 3-D perception and 2-D perception. The 3-D perception is constructed through the processing of the brightness
information.
The diﬀerences in the brain activities between these two kinds of stimuli areshown in Fig. 3. The activities were located in the bilateral occipital cortex including the lingual gyrus, inferior and middle occipital gyrus, cuneus, and sub-gyral(BA 17, 18, 19). Besides the occipital activation which were also observed in the
pantomime eﬀect perception, there were many activities in the occipito-temporal
cortex, including the bilateral fusiform gyrus, inferior and middle temporal gyrus,
and the right parahippocampal gyrus (BA 19, 20, 21, and 37). In addition, theDecember 12, 2007 20:55 WSPC/115-IJPRAI1316

SPI-J068 00599

Q. Zhang & K. Mogi

(a)

A

L

R

P
(b)
Fig. 3. fMRI experimental results of the group-level analysis using random-eﬀects models. The
color blobs represent the areas activated by the 3-D perception from the monocular shading cues
versus the 2-D perception. Similar to the activation induced by the pantomime eﬀect, there are
activities in the prefrontal cortex. Particularly, the right prefrontal cortex is more activated. In
addition, there are activations in the right parietal cortex. (a) Activities rendered on a 3-D brain(t > 3.33, p < 0.003). (b) Activities plotted in four slices with statistic data (t > 2.0). The numberin the ﬁgure shows the z value of MNI coordinates. A: Anterior, P: Posterior, L: Left, R: Right.

parietal cortex was activated in the areas of bilateral precuneus and superior parietal lobule (BA 7), and the right supramarginal gyrus (BA 40). In the frontal cortex,
there were also activities in the left middle and superior frontal gyrus (BA 10). In
addition, there were more activities in the right frontal/prefrontal cortex, whichwere not observed in the condition of pantomime eﬀect, including the inferior, middle, superior and medial frontal gyrus, and precentral gyrus (BA 6, 8, 9, 10 and47). In summary, there are more right frontal and parietal activations, comparedwith the activities induced by the pantomime eﬀect, where more left hemisphereactivities were observed.
3.3. Common areas and diﬀerences in the two types of volumetricobject perception
The above fMRI experimental results show that the frontal, parietal, as well as theoccipital cortex were involved in both types of 3-D volumetric object perception
induced by the pantomime eﬀect and the monocular shading cues. Both commonand diﬀerent cortical areas exist in the two kinds of 3-D perception. The common
areas were located in the occipital visual cortex and the left prefrontal cortex. We
propose that activities in the occipital cortex were in charge of the general visualDecember 12, 2007 20:55 WSPC/115-IJPRAISPI-J068 00599

Representation of 3-D Volumetric Object from the Pantomime Eﬀect

1317

processing to recognize the inducing objects, and the left prefrontal cortex in BA
9/10 was involved in the consciousness awareness of the perception.
On the other hand, there are diﬀerent brain areas for these two kinds of 3-D
perception. Examples of group-level BOLD signal changes in these areas are shownin Fig. 4. For the perception induced by the pantomime eﬀect, there were activities in the left parietal cortex [Fig. 4(a)] and the right anterior cingulate cortex(ACC) [Fig. 4(b)]. In contrast, the perception induced by the monocular shadingcues activated more the right frontal/prefrontal and parietal cortex, as well bilateral
occipito-temporal cortex. For the occipito-temporal cortex [Fig. 4(c)], the diﬀerent
areas were mainly located in the bilateral BA 19, 20, 37, which included bilateralfusiform gyrus, inferior and middle temporal gyrus, and the right parahippocampal gyrus. These areas were mainly in the ventral pathway of the visual processing
system, and we propose that they were in charge of the processing for the brightness of the shading cues. As for the right parietal cortex [Fig. 4(d)], including the
sub-gyral, right parietal lobule, precuneus, superior parietal lobule, and postcentral
gyrus (BA 7, 40), we propose that these areas are in charge of the spatial processing for the 3-D perception. For the prefrontal cortex [Fig. 4(e)], the activities
were located in the right prefrontal cortex of BA 6, 8, 9, 10, 47. They were theright inferior, middle, superior and medial frontal gyrus, as well as the precentral
gyrus. We suppose that these areas processed the information from the occipital and temporal cortex and constructed the 3-D perception from the brightness
information.

3.4. Temporal features of volumetric object perception
So far we have shown that diﬀerent cortical areas are involved in these two typesof 3-D perception, and supposed that their processing mechanisms were diﬀerent.
Since the diﬀerent brain mechanisms may result in diﬀerent processing time forattaining the perception, we studied the response time of the subjects when theyattained the perception. The results are shown in Fig. 5, where the individual
response time for the 3-D perception induced by the pantomime eﬀect and monocular shading cues, as well as the 2-D perception were plotted in diﬀerent colors.
The data for all ﬁve subjects were shown. For four of ﬁve subjects, the monocularshading cues took the longest time for the 3-D object perception, except for one
subject (s2). The t-test results show that the response time did not have a significant diﬀerence for subject 2, whose response time for shading cues was shorter,
while the diﬀerences for the other four subjects were signiﬁcant. In addition, theresponse time for the 3-D perception induced by the pantomime eﬀect was not signiﬁcantly diﬀerent from that for the 2-D perception, while the time induced by the
monocular shading cues was signiﬁcantly diﬀerent from that for the 2-D perception.
Considering that the perception from shading cues involves the higher-level brain
cortical areas, we therefore propose that the perception of volumetric object fromshading is not a simple early process,11,17 and the right prefrontal cortex processes

December 12, 2007 20:55 WSPC/115-IJPRAI1318

SPI-J068 00599

Q. Zhang & K. Mogi

(a)

(b)

(c)

(d)

(e)
Fig. 4. The group-level BOLD signal changes corresponding to the 3-D perception induced by
the pantomime eﬀect and the monocular shading cues.

the information from the lower-level visual cortex, and projects a top-down signal
back to construct a volumetric object perception. On the other hand, the perception from the pantomime eﬀect takes almost the same time as the 2-D perception.
It is accelerated by binocular information though it also recruits some higher-level
cortex.

December 12, 2007 20:55 WSPC/115-IJPRAISPI-J068 00599

Representation of 3-D Volumetric Object from the Pantomime Eﬀect1319

Response time for 3D, shading, and 2D
1600

Response time (ms)

1200

3D_P
3D_shading
2D

*

1400

*

**
**

* *

s3
subjects

s4

1000

*

*

* p<0.05** p<0.001

*

800
600
400
200
0
s1

s2

s5

Fig. 5. Response times for diﬀerent types of perception. The 3-D perception induced by the
monocular shading cues takes the longest time.

4. Discussions
It is well known that visual system consists of two pathways: the ventral pathway
is in charge of “what” information such as form, colour, texture, and so on; andthe dorsal pathway deals with “where” or “how” issues, including the spatial andmotion information.5,20 Several years ago, Goodale et al. studied an individual
patient (DF) with visual form agnosia4, who was able to use vision to adjust theposture of her hand to the size, shape and orientation of a goal object as she reached
out to pick it up — even though she showed no perceptual awareness of these same
object attributes. Their fMRI experiments revealed no diﬀerential activation inventral-stream areas which were severely damaged in DF when she was presented
with line drawings of common or novel objects; but when DF reached out to grasp
objects of diﬀerent sizes and orientations, she showed relatively normal activationin dorsal-stream areas implicated in the visual control of grasping. They concluded
that vision-for-perception and vision-for-action depended on quite separate visual
pathways in the cerebral cortex. Here our fMRI results show that both the ventral
and dorsal pathways were involved in 3-D volumetric object perception. We did not
ask the subjects act on the 3-D perception. The activity for the button click wasquite similar for every stimulus, and was thought to disappear by subtraction of
two conditions. So the diﬀerence in the brain activities should be due to the visualobservation. We provide the suggestion that maybe action probably does not needthe ventral stream areas, but the visual perception indeed needs the dorsal stream
areas, in particular, the parietal cortex for the spatial processing. The vision-forperception pathway should include both the ventral and partial dorsal pathways.
In addition, we found that there is laterality in the prefrontal cortex for the twoperceptions. The perception induced by the pantomime eﬀect speciﬁcally activatedDecember 12, 2007 20:55 WSPC/115-IJPRAI

1320

SPI-J068 00599

Q. Zhang & K. Mogi

more the left hemisphere. In contrast, the monocular shading cues activated more
the right prefrontal cortex. The right prefrontal cortex was reported in charge of
the working memory processing during encoding process and retrieval process fornonverbally coded information16 , and associated with recognition.15 Furthermore,
Ramachandran pointed out that the processing of shape from shading depended
on the previous experience of the light source illuminating.17 Hence the role of
the right prefrontal cortex in the 3-D perception by shading cues is related withthe working memory function. The information from lower-level cortex was encodedagain here, together with the retrieval processing of previous experiences on the lightillumination, to construct the 3-D object perception. Then top-down instructions
were sent to the inferior temporal cortex. This is the reason why the perception
induced by the shading cues takes longer time. Our ﬁndings extend the knowledge
about the well-known “Shape from shading” perception, and show that object fromshading also recruited higher-level processing in the parietal and prefrontal cortex,
and include top-down processing.
On the other hand, Kourtzi et al.9 found that no adaption was observed between
the observations of stereoscopically deﬁned silhouettes and shaded 3-D images ofthe same objects, and suggested that the stereo and shaded object conditions may
involve diﬀerent neural populations in the LOC. Though our research did not study
the details in LOC, we found that the processing for the stereo and shading cues
activated quite diﬀerent cortical areas, besides the occipital areas. So we think thatthe processing of the 3-D perception from diﬀerent types of inducing cues might
employ quite diﬀerent mechanisms from the occipital LOC to the parietal andprefrontal cortex.

5. Conclusions
Our study revealed that the 3-D perception recruited a wide range of cortical areas,from the elementary processing in the occipital visual areas to parietal and prefrontal cortex, in particular for the 3-D perception induced by the monocular shading cues. Our fMRI experiments not only conﬁrmed previous studies on the object
perception in the occipital cortex, but also extended the knowledge of the 3-D processing in the prefrontal cortex, and clariﬁed the mechanism for two diﬀerent kindsof 3-D perception induced by the pantomime eﬀect and monocular shading cues.
For the perception induced by the pantomime eﬀect, the left frontal/prefrontal cortex were more activated; in contrast, the right prefrontal/frontal cortex were more
involved in the perception from shading. Furthermore, we compared their response
times, and the perception induced by the shading cues took longer time than thatby the pantomime eﬀect. Hence we suppose that the shading cues need to be processed at the ventral pathway ﬁrst and then forwarded to the right frontal cortex
for extracting the spatial information to construct the 3-D perception. In contrast,
the pantomime eﬀect is related with the processing of stereopsis, and the perception is accelerated by binocular information. These ﬁndings suggest that we mayDecember 12, 2007 20:55 WSPC/115-IJPRAISPI-J068 00599

Representation of 3-D Volumetric Object from the Pantomime Eﬀect

1321

need to handle the 3-D object pattern recognition in diﬀerent algorithms for the
stimuli induced by binocular and monocular cues when we create artiﬁcial visualsystems.

Acknowledgments
The fMRI experiments were conducted at ATR, Kyoto, Japan. We thank Akiko
Callan, Yasuhiro Shimada, Ichiro Fujimoto, Yuko Shakudo, Shinobu Masaki, andKayoko Nakagawa at Brain Activity Imaging Center, for their support in acquiring
the fMRI data, and helpful advice in experimental design and data analysis. We
also thank the two reviewers for their insightful comments to improve this paper.

References
1. B. T. Backus, D. J. Fleet, A. J. Parker and D. J. Heeger, Human cortical activity
correlates with stereoscopic depth perception, J Neurophysiol. 86 (2001) 2054–2068.
2. G. J. Brouwer, R. van Ee and J. Schwarzbach, Activation in visual cortex correlateswith the awareness of stereoscopic depth, J Neurosci. 25 (2005) 10403–10420.
3. K. Friston, A. Holmes, K. Worsley, J. Poline, C. Frith and R. Frackowiak, Statistical parametric maps in functional imaging: a general linear approach, Human Brain
Mapping 2 (1995) 189–210.
4. M. A. Goodale, T. W. James, J. C. Culham, G. K. Humphrey and A. D. Milner, fMRI
studies of action and perception pathways in an individual with visual form agnosia,
Perception 32 (Supplement) (2003) 15.
5. M. A. Goodale and A. D. Milner, Separate visual pathways for perception and action,
Trends Neurosci. 15 (1992) 20–25.
6. K. Grill-Spector, T. Kushnir, S. Edelman, G. Avidan, Y. Itzchak and R. Malach, Differential processing of objects under various viewing conditions in the human lateral
occipital complex, Neuron 24 (1999) 187–203.
7. D. A. Hinkle and C. E. Connor, Three-dimensional orientation tuning in macaque
area V4, Nat. Neurosci. 5 (2002) 665–670.
8. P. Janssen, R. Vogels, Y. Liu and G. A. Orban, Macaque inferior temporal neurons
are selective for three-dimensional boundaries and surfaces, J Neurosci. 21 (2001)
9419–9429.
9. Z. Kourtzi, M. Erb, W. Grodd and H. H. Bülthoﬀ, Representation of the perceived
3-D object shape in the human lateral occipital complex, Cereb. Cortex 13 (2003)
911–920.
10. R. Malach, J. Reppas, R. Benson, K. Kwong, H. Jiang, W. Kennedy, P. Ledden,
T. Brady, B. Rosen and R. Tootell, Object-related activity revealed by functional
magnetic resonance imaging in human occipital cortex, PNAS 92 (1995) 8135–8139.
11. P. Mamassian, I. Jentzsch, B. A. Bacon and S. R. Schweinberger, Neural correlatesof shape from shading, Neruoreport 14 (2003) 971–975.
12. J. D. Mendola, A. M. Dale, B. Fischl, A. K. Liu and R. B. H. Tootell, The representation of illusory and real contours in human cortical visual areas revealed by functional
magnetic resonance imaging, J. Neurosci. 19 (1999) 8560–8572.
13. C. Moore and S. A. Engel, Neural response to perception of volume in the lateral
occipital complex, Neuron 29 (2001) 277–286.

December 12, 2007 20:55 WSPC/115-IJPRAI

1322

SPI-J068 00599

Q. Zhang & K. Mogi

14. J. D. Nguyenkim and G. C. DeAngelis, Disparity-based coding of three-dimensionalsurface orientation by macaque middle temporal neurons. J Neurosci. 23 (2003) 7117–
7128.
15. S. F. Nolde, M. K. Johnson and C. L. Raye, The role of prefrontal cortex during testsof episodic memory, Trends Cogn. Sci. 2 (1998) 399–406.
16. B. Optiz, A. Mecklinger and A. D. Friederici, Functional asymmetry of human prefrontal cortex: encoding and retrieval of verbally and nonverbally coded information,
Learning & Memory 7 (2000) 85–96.
17. V. S. Ramachandran, Perception of shape from shading, Nature 331 (1988) 163–166.
18. M. Taira, K. I. Tsutsui, M. Jiang, K. Yara and H. Sakata, Parietal neurons represent
surface orientation from the gradient of binocular disparity, J Neurophysiol. 83 (2000)
3140–3146.
19. D. Y. Tsao, W. Vanduﬀel, Y. Sasaki, D. Fize, T. A. Knutsen, J. B. Mandeville, L. L.
Wald, A. M. Dale, B. R. Rosen, D. C. Van Essen, M. S. Livingstone, G. A. Orban and
R. B. Tootell, Stereopsis activates V3A and caudal intraparietal areas in macaquesand humans, Neuron 39 (2003) 555–568.
20. L. G. Ungerleider and M. Mishkin, Two cortical visual systems, in Analysis of Visual
Behavior, eds. D. J. Ingle, M. A. Goodale and R. J. W. Mansﬁeld (The MIT Press,
Cambridge, MA, 1982), pp. 549–586.
21. Q. Zhang, M. Idesawa and Y. Sakaguchi, Pantomime eﬀect in the perception of volumetric transparent illusory objects with binocular viewing, Jpn. J. Appl. Phys. 37(1998) L329–L332.

Qi Zhang received the
B.E. and M.E. degrees
in information scienceand electronic engineering from Zhejiang University, China, in 1990
and 1993 respectively,
and the Ph.D. in information systems from
The
UniversityofElectro-Communications, Japan, in 1998.
Currently, she is working in the ﬁeld of systems brain science with Sony Computer Science Laboratories, Japan.

Ken Mogi received
his Ph.D from Department of Physics, University of Tokyo. He
conducted research inbrain science in Riken(Japan) and the University of Cambridge(U.K.). Currently, he isa senior researcher at
Sony Computer Science Laboratories, Tokyo,
and is a visiting Professor at Tokyo Instituteof Technology.