R&D Spirits
Search for Ideal Image Presentation Technologyfor Human Visual System
Shin’ya Nishida
Senior Research Scientist, Supervisor
Sensory and Motor Research GroupNTT Communication Science Laboratories

What is the ideal image for human viewing? Senior Research Scientist Shin’ya Nishida of theSensory and Motor Research Group of the NTT Communication Science Laboratories aims toprovide the answer to this question through his research into the mechanism via which the brain“sees” things. The editors spoke with Dr. Nishida about his theory on the asynchronous perception of color and motion, a theory that may lead to the development of new technology andthe changing of paradigms in brain studies.

Psychophysics approach to the human“viewing” mechanism

E: Could you outline your group’s current researchfor our readers?

Editor: What is the main theme of your researchgroup?

N: One of our research themes relates to the humanperception of the pattern of surface light reflectance.
In a visual illusion created by Professor Adelson ofMIT, with whom our group is conducting joint
research, the human eye perceives two images of thesame gray color to be completely different colors
because of the context they are embedded in. This isbecause what the human visual system sees is the pattern of light reflection of the surface, rather than theabsolute magnitude of reflected light. By taking into
account changes in illumination, one sees surface
color the same no matter what the circumstances. At
the same time, we also know that our ability of perceiving constant surface reflectance is not perfect due
to limitations of the brain processing. We would liketo know what actually happens in our brain, and touse the outcome of the research for developing an easy
way to synthesize an image that looks similar to theoriginal real image without matching image details.
Another theme is visual presentation through slits.
If a moving pattern is viewed through a stationary slit
the eyes store the visual information that passes
before it and the brain then pulls the separate pieces
of information together into one complete image (seeSelected Papers in this publication, “Human Visual

Nishida: Our group is conducting basic research into
the relationship between the human senses and communication. My personal field of research deals withhuman vision. It has been proven recently in brain
studies that when we see an apple falling from a tree,
for example, our brain first processes separately thedata relating to color, shape, and motion, and afterward integrates those elements to recognize thefalling apple. But that leaves us with the new questionof how the brain integrates the data it processes. In
search for an answer to that question, I am researching the brain’s functions by examining the humanperception of various visual illusions. The field I
work in is called psychophysics. Although some people may think it unusual for NTT to be conducting
psychophysical research, visual presentation play avery important role in communication today. In asense, our group’s research is to provide basis ofhuman communications technology.

4NTT Technical Review

R&D Spirits

Mechanism,” p. 13). Although this presentation technique was developed as early as 30 years ago, I
acquired a renewed interest in it after I saw an animated advertising display that applied the old technique. The effect of the display cannot be sufficiently
explained by currently prevailing notion that the brain
processes shape and movement separately. The spatial resolution of the display was extremely low, andwhen the character patterns in the display were notmoving they were almost unreadable. When moving,
however, the characters were completely readable. In
short, information regarding movement was used toincrease the amount of information regarding shape.
It seemed to me that by studying this phenomenon
further our group might be able to understand how thebrain integrates the shape and movement of images.

Adaptation

Test (stimulus)

Unique “Time Marker” theory derived bystudying contradictions in accepted theories
E: What is the main point in your research?
N: The main point in our research is the unique theory we have proposed concerning the visual system inthe brain. As mentioned, it has been long believedthat the brain processes separately the various typesof visual information it receives. As suggested in theslit discussion above, however, I do not believe thatthe brain completely separates the visual information
it receives. One result of our research is seen in Fig. 1,
which shows the interaction between movement andshape. While looking at the center of the figure, rotate
the windmill on both sides for a while. After you stop
rotating the images, you will see them appear torotate in the reversed directions; at the same time, theorientations appear to be tilted although the windmills are physically oriented vertically. In this experiment, the sense of shape has been warped by movement information produced by a visual illusion. If thebrain were processing shape and movement information separately, such an interaction would not occur.
The fact that the interaction does occur suggests thatin order to smoothly process the changes in shape, thebrain used movement information to predict the succeeding sequence of events.
Another example that argues against the notion ofseparate processing is the visual illusion called “perceptual asynchrony of color and motion.” This visualillusion shows that color and motion can actually besynchronous while they do not appear to be so, andwhen they actually are not synchronous they canappear to be so (see Selected Papers in this magazine,
Vol. 1 No. 2 May 2003

Test (perception)

Fig. 1. Interaction between movement and shape.

“Human Visual Mechanism,” p. 15). This time lag
has been explained by saying that the brain processes
color faster than it processes motion. In the asynchrony of color and motion, however, as the exchange
cycle becomes longer the asynchrony dissipates.
Also, in experiments where we measured the lengthof time it took to respond separately to changes incolor and motion, absolutely no time disparity wasobserved between the two. The hypothesis that thebrain processes color faster than it processes motion
thus cannot completely explain the asynchrony ofcolor and motion. To replace that time processing
hypothesis, we developed a “time marker” hypothesis
in which we based time-related judgments on a comparison of salient features in the stimulus sequence(time markers). Because it is difficult for humans todiscern the instant that motion changes direction—at5

R&D Spirits

high speeds it is next to impossible—we surmise thatasynchrony arises. The time marker hypothesis is a
fresh approach in that it says the brain processes separately both the contents of an events and the time theevents occurs. Clear evidence that our hypothesis iscorrect would lead to the changing of paradigms inbrain studies.
E: What is the significance to society-at-large ofresearch that clarifies the brain’s visual processing
mechanisms?
N: Because next-generation communications technology will rely heavily on animation, we believe thatknowledge about the brain’s visual processing mechanisms will play a major role in developing that technology. An example is an unexpected noise problem
for DLP (digital light processing) projectors and plasma displays. These problems probably emerged
because the products were developed without a complete appreciation of human sensory characteristics.
In other words, the products were developed based on
the belief that human visual system cannot detect
rapid stimulus changes regardless of the pattern ofeye movements. In contrast, although the NTSC
(National Television Systems Committee) color television format projects a low-resolution image on television screens, the resultant image is of acceptable
quality when viewed by the human eye. This result is
probably due to unintentional tuning by the productdevelopment team. It can also be said that the thinking mentioned earlier—in talking about research
related to motion and shape, where information about
motion is used for predicting the succeeding
sequence of shape events—closely resembles moving
picture compression technology, such as that used inthe MPEG (Moving Picture Experts Group) format.
In these ways, the mechanism via which humans see
things is at the root of all motion picture technology.
E: How might the results of your research be utilized?
N: The results of our research will directly affect thedevelopment of products such as those just mentioned, including new displays. In addition, features
like those expressed in words like “easy to see” and“flickering feeling” will eventually be quantified. By
proposing a scientific basis for them our group will beable to provide guidelines about moving images to awide audience. From that viewpoint, the senior
6

research professor in our group is studying the effects
that moving images exert on the health and safety on
human beings. At any rate, this is research conductedby NTT and the results are certain to be put to gooduse in developing communications technology thatutilizes multi-modal information, including information from the auditory sense, as well as time-spaceinformation.

Research into visual system allows self to beresearch subject
E: What was your principal motivation for entering
the field of research into the visual system?
N: The original motivation for me to begin theresearch I am doing today was that I wanted to know
more about the functions of my own brain. When I
was young I was a bit philosophical and I remember
wondering what it actually meant when a person said
“there’s something over there.” But rather than taking
a humanities approach to answering that question, I
searched for a scientific answer. Although I originally also had an interest in human cognition as well aslanguage, I gradually zoned in naturally toward
human vision, a field I could approach scientificallythrough experimentation.
E: What were the reasons for choosing NTT as thevenue for your research?
N: Research into the visual system and visual information is a relatively new field for NTT. In fact, manymembers of my research group were hired midway
through their professional careers. I came to NTT, forexample, from the ATR Auditory and Visual Perception Laboratories. Dr. Takao Sato, currently a professor at the University of Tokyo, was seconded fromNTT to ATR at the time. Several months after hereturned to NTT, I moved here to work with him. I
think the basic choice for researchers is either to conduct a scientific research in university together with
graduate students or to plug away in a research institute. Different people think differently about which isbest, of course, but conducting research in a research
institute in the private sector is much more attractive
to me. There is also more freedom for individual initiative in a research institute. Based on that thinking,and because I would be able to work with otherresearchers in fields close to my own, I chose to join
NTT.

NTT Technical Review

R&D Spirits

E: What do you consider most interesting about yourresearch?
N: My research theme is very close to my own life.
Anyone with normal senses can easily experiencevisual illusions, and can enjoy them the instant they
see them. Although we use volunteers in our laboratory experiments, we often also take measurements
using ourselves as subjects. At such times we areinvestigating how we ourselves see things, and we
gradually come to understand our own visual capabilities. As well, human vision is a research field thatis widely open. It is as if nothing is understood forcertain yet. In fact, in terms of understanding thehuman vision, our visual neuroscientists are probably
no more than half a step ahead of those reading thisarticle. It can be said that our research is in an area
where one wonders if it is truly possible to findanswers. So while our area of research is very friendly and familiar, since it deals with the human sense ofsight, there is no end to the work that must be done.
Those are some of the reasons why our research is sointeresting.

NTT engages in wide-ranging joint
international research
E: What is your group currently doing in the area ofinternational activities?
N: Because my papers are only published overseas,
perhaps all my research activities can be calledinternational. Other than my personal research I am
also involved a bit with the work of the Asian Association for Vision Research. But I have not yet
become a member of any committee, nor have I
served as an editor of the international scientific journal. Perhaps I have not reached that age yet. (laughter) I do receive many papers for review, however; in2002, for example, I reviewed about ten papers. This
is not many, since researchers in other countries saythey review many more than that. For a Japanese,
however, writing referee’s reports in addition to journal papers in English requires quite an effort compared to researchers native to English. I view thateffort, however, as a part of my research.
E: Does your group participate in joint research withoverseas researchers?
N: Even in the context of the overall NTT organization, our group is quite busy with joint overseas
Vol. 1 No. 2 May 2003

research. One area of such research is the revolving
windmill visual illusion introduced earlier, as well asstudies related to the asynchrony of color and motion.
While I was a visiting researcher in England at University College London for a year from 1998, I studied with researchers there and was able to propose
several new ideas. There is also an ongoing joint
research project between NTT and MIT. In that project, Professor Adelson of MIT, who developed thegray visual illusion I mentioned earlier, is myresearch partner. Other joint research in our group is
being conducted with the Psychophysics Lab of Dr.
Shinsuke Shimojo, one of our research professors, atCalifornia Institute of Technology. Our group is alsoworking jointly with researchers in Australia. With
Massachusetts General Hospital, meanwhile, we arecooperating in functional brain imaging. Being able
to work with well-known professors in various fields
is a valuable experience for everyone in our groupand we learn much in the process.
E: How is NTT’s work in the field of visual scienceevaluated overseas?
N: NTT’s research in the field of visual science is still
fairly new, and we have not yet developed to the levelof institutions such as MIT. We are proud to say, however, that in Japan we rank among the most activeresearch groups in the field. Actually, we sometimes
hear comments from overseas about the work ourgroup is doing. Some people are concerned that as aprivate-sector research institute we might follow thesame path that Bell Labs took. Very definitely, the situation in Japan differs from that in the U.S., whereindustry and academia are linked closely together. If
an organization like NTT does not conduct basic
research in various areas the country itself will suffer
severe losses. Our group manager, however, says
clearly, “NTT should not make the same mistakes
AT&T made.” The members of our group have taken
those words to heart by relating that message to theoutside at every opportunity and by continuing to
make efforts to realize good results from ourresearch.
E: Please share with us your thoughts about thefuture.
N: When I was a bit younger, I did not have a clear
view on how to relate my basic scientific studies withdevelopment of telecommunications technology. In
recent years, however, perhaps because I gained7

R&D Spirits

Paradigm change in brain science

Time
VisionSomatosensory

SpaceVision
MotionAudition

Shape

Color

Somatosensory

ShapeAudition

Separate processing for eachsensory modality and attribute

MotionColor

Cross-modality/attributeintegration

Development of new telecommunications
technology for humans

General scope of the study. Our brain may process different sensory modalities and attributes in an integrative fashion on
the unified spatiotemporal coordinates, rather than analysing them separately in distinct modules, as has been widelybelieved. Proving this idea will lead to a paradigm change in brain science, which in turn will provide basis for thedevelopment of novel telecommunications technologies for human.

greater leeway, I continue my studies not only
because I enjoy the work but also because I hope itwill serve some higher purpose in society. In otherwords, my research has significance not only in science but also for how it might influence future technological developments. That, of course, is not aneasy thing to do, but it is the stance I want to take in
my future studies.
E: How do you personally view the role of NTT
Communication Science Laboratories?
N: Although the NTT Communication Science Laboratories are part of a private enterprise, the membersof our group are not forced to conduct application
8studies. We are pretty much afforded the freedom toconduct the research we want to, and I am quite
happy to be associated with the labs. Being given
much freedom, however, does not mean we can study
anything we like. It merely means we can conduct ourresearch at the pace we determine to be optimal. And
we are rewarded for our successes. In order to support
that status we are required to be fully responsible for,and sufficiently capable of, completing our research.
Although I am sometimes not 100 percent certain that
I have such capabilities, I always conduct myresearch while being fully aware of its social and scientific significance, and I hope I can continue providing acceptable research results.

NTT Technical Review

R&D Spirits

Interviewee Profile
1985 – Graduated from Kyoto University, Faculty ofPhilosophy, with B.A. in Psychology
1990 – Finished the Ph. D. course in psychology fromKyoto University, Faculty of Literature
1990 – Joined ATR Auditory and Visual Perception
Laboratories as researcher
1992 – Joined NTT
1996 – Earned Ph. D. in literature from Kyoto University
1997 – Invited to University College London asresearcher
1999 – Current positionSelected Publications
S. Nishida and T. Sato, “Motion aftereffect with flickering test patterns reveals higher stages of motion processing,” Vision Research, Vol. 35, pp. 477-490,
1995.
S. Nishida, T. Ledgeway, and M. Edwards, “Dualmultiple-scale processing for motion in the humanvisual system,” Vision Research, Vol. 37, pp. 2685-

Vol. 1 No. 2 May 2003

2698, 1997.
S. Nishida and M. Shinya, “Use of image-based information in judgments of surface reflectance properties,” Journal of the Optical Society of America, Am,
Vol. 15, pp. 2951-2965, 1998.
S. Nishida and A. Johnston, “Influence of motion signals on the perceived position of spatial pattern,”
Nature, 397, pp. 610-612, 1999.
S. Nishida and H. Ashida, “A hierarchical structure ofmotion system revealed by interocular transfer offlicker motion aftereffects,” Vision Research, Vol. 40,
pp. 265-278, 2000.
S. Shimojo, Y. Kamitani, and S. Nishida, “Afterimageof perceptually filled-in surface,” Science, 293, pp.
1677-1680, 2001.
S. Nishida, I. Motoyoshi, R. A. Andersen, and S. Shimojo, “Gaze modulation of visual aftereffects,”
Vision Research, Vol. 43, pp. 639-649, 2003.

Membership in Professional Organizations
Association for Research in Vision and Ophthalmology and the Optical Society of America

9