AI Magazine Volume 27 Number 4 (2006) (© AAAI)
Articles

AAAI Presidential Address(AA)AI
More than the
Sum of Its Parts
Ronald J. Brachman■ Recent thinking has it that AI, 25 years ago a unified field with a shared vision of creating intelligent machines, has devolved into a loosely connected set of distinct specialty areas with little
communication or mutual interest between them.
To the extent that this is true, it certainly appears
to lessen the value of a centralized AI organization
like AAAI and of traditional grand-scale AI conferences. But, I argue, the consequences are actually
far worse: because of the very nature of intelligence, the centrifugal force on the field could
thwart the very mission that drives it by leaving noplace for the study of the interaction and synergyof the many coupled components that individually in isolation are not intelligent but, when working together, yield intelligent behavior. To raiseawareness of the need to reintegrate AI, I contemplate the role of systems integration and the value
and challenge of architecture. Illustrating somereason for optimism, I briefly outline some promising developments in large projects that are helping to increase the centripetal force on AI. I conclude by discussing how it is critical that the field
focus its attention back on its original mission, led
by a heavy dose of integrated systems thinking andgrand challenges, and why after its first quarter
century, AAAI is more essential than ever.

O

ne of the privileges afforded the American Association for Artificial Intelligence (AAAI) president is the chance tostand in front of the entire membership of theorganization and speak in an unfiltered way
about whatever he or she has on his or her
mind. This is a wonderful opportunity, yet adaunting one. Since one is keenly aware of thecommitment to speak very far in advance, one
can muse about the speech at many odd

moments over a long stretch of time. This
allows the jotting of notes and the collectionof meandering thoughts over quite a protracted period. But because of the sheer length ofadvance-warning time, it encourages one to be
expansive and to note virtually anything one
would like to opine about in a large forum. In
my case, this freedom led to a great deal of random thinking and a fairly large pile of notes.
But as the time drew near to speak, and I
looked over what I had written, I found that
there was almost no coherence to my many
minor brainstorms. There were numerous specific things and a variety of independent
research directions to consider, but no big picture. Then it occurred to me that this might
actually be symptomatic of a fundamental
problem that we are facing as a field and that
AAAI is facing as an organization, and that thelack of a strongly unifying force might itself bea worthy theme for the address.
Between 2002 and 2005, I had the privilegeof working at the Defense Advanced Research
Projects Agency (DARPA) in a position that is
uniquely important to the history of AI: I was
honored to be able to serve as director of theInformation Processing Technology Office(IPTO). In that role I had the opportunity tomeet a very large variety of people with greatideas in all aspects of artificial intelligence and,more broadly, across all of computer science.
While my ability to get into technical depth
was limited by the sheer volume of conversations and visits, the breadth one sees in such a
position is very hard to match in any other.
The global perspective accrued through suchextensive interactions with the community
also afforded me the opportunity to contem-

Copyright © 2006, American Association for Artificial Intelligence. All rights reserved. ISSN 0738-4602

WINTER 2006

19

Articles

The First AAAI Conference was Held at Stanford University.

plate the big picture and, perhaps more importantly (and consonant with the nature of thejob at DARPA), to identify gaps in our national
computing research agenda. It also occurred tome that that perspective was a very special
asset to use in drafting this presidential
address.
So, instead of addressing a technical topic in
depth or picking on a single new direction—
often the fodder for AAAI presidential addresses—I want to raise a broad issue and consider
some larger questions regarding the nature of
the field itself and the role that AAAI as anorganization plays in AI. My hope is to encourage thinking about some things that I believe
are very important to the future of the field asa whole and perhaps to start a dialogue about
research directions and collaborations that in
the end might bring us all back together andallow us to take advantage of the opportunitythat AAAI affords all of us as AI practitioners.

A Wonderful Timeto Mark Progress
The year 2005 was a momentous one in artificial intelligence, at least in the United States. It
marked the 25th anniversary of the foundingof the American Association for Artificial Intelligence (celebrated in the Winter 2005 issue of
AI Magazine), and AAAI-05 was our 20th conference. The organization was started in 1980
in response to vibrant interest in the field,

20 AI MAGAZINE

which back then was served mainly by an
International Joint Conferences on Artificial
Intelligence (IJCAI) conference held only every
two years. The first AAAI conference was heldat Stanford University; it was very much a
research conference, a scientific event that generated a lot of excitement. The conference wassmall and intimate, with few parallel sessions.
There were excellent opportunities for us totalk to one another. AAAI-80 gave real substance to the organization, clearly getting AAAI
off on the right foot, and it gave new identity
and cohesiveness to the field.
This year—2006—has also been a big year,
celebrating the 50th anniversary of the original
meeting at Dartmouth College, where thename “artificial intelligence” first came into
common use. Numerous events around theworld, including a celebratory symposium at
Dartmouth and an AAAI Fellows Symposium
associated with AAAI-05, have marked thisimportant milestone in the history of the field.
Progress since our first AAAI conference has
been substantial. While each year’s results may
have seemed incremental, when we look backover the entire period we see some truly amazing things. For example, a computer program
finally beat the world’s chess champion. In
hindsight this may no longer look so exciting(purists will say that it was not an “AI” systemthat beat Garry Kasparov but rather a highly
engineered special-purpose machine largely
made possible by Moore’s Law), but it is worth
contemplating from the point of view of
1980—or, even more dramatically, from that of1956. Looking forward from back then, no
matter how Deep Blue actually worked, playing
chess well was clearly an AI problem—in fact, aclassical one—and our success was historic.
More recently, a robotic vehicle from Stanford
University, using machine learning technology, conquered the 2005 DARPA Grand Challenge, successfully managing a course of morethan 140 miles over difficult terrain in lessthan 10 hours without any human intervention. By any measure this was an incrediblefeat.
Another notable aspect of AI life over the lastquarter-century was the broad rise of excitement and financial investment in the field.
That ranged from a wave of startups that lasted
into the 1980s to significant diffusion of ourtechnology and practitioners throughout
worldwide industry. The influence of the AI
community in current large and critical commercial enterprises—through, for example, text
processing, speech and language processing,robotics, machine learning, data mining,knowledge management, and a host of appli-

Articles

cations of the sort that the Innovative Applications of Artificial Intelligence (IAAI) conference
has highlighted for years—while accomplishedperhaps with little fanfare, has been undeniable. Important systems with significant AI
contributions are deployed and operating daily
in virtually every industry. Small robots havesaved lives, both on the battlefield and in difficult search and rescue situations. AI systems are
flying in space. And through Internet searchengines, every day AI directly touches millions
and millions of people.
Perhaps the zenith of our field’s popularity
was in 1985, when we held our joint conference with IJCAI in Los Angeles on the University of California, Los Angeles (UCLA) campus.
That conference had more than 5,000 attendees. The excitement in the community waspalpable, and the interest from the outside,both in the commercial sector and in the press,
was extraordinary. There was a great deal ofnational and international attention. For
example, Woody Bledsoe (then president of
AAAI) and I were asked to appear on a national
radio talk show, where we debated various
aspects of AI. Hector Levesque gave the Computers and Thought lecture in a crowded
Pauley Pavilion, the home of UCLA basketball—quite an exciting experience. We probably haven’t seen anything like that since, but it
was a wonderful time.
Since then, we’ve had an “AI Winter,” with adramatic drop in funding for the field, followed by many years of reasonable growth andsignificant thaw. One small indicator of thecurrent improved situation, based on our workat DARPA: starting with the fiscal year 2006
budget, there has appeared a line item (a “Program Element”) in the U.S. federal budget that
explicitly calls out “Cognitive Computing Systems”—very much an AI agenda. This expressed the government’s intent to fund a significant budget item directly focused on artificial
intelligence research, as well as at least a modest suggestion of longevity for the item. Since
then, several hundred million dollars of funding were approved and spent for this area. And
this represents money coming from one funding agency—it does not include current andprospective funding from places like the
National Science Foundation (NSF) andNational Institutes of Health (NIH) and themilitary services’ research organizations. AI hasreached the level of explicit high-level line
items in the U.S. budget.
Finally, there is the undeniable general infiltration of AI-related ideas into the public consciousness. Prior to 1980 AI was an interesting
topic for science-fiction writers and media pro-

ducers, with the HAL 9000 Computer in 2001:
A Space Odyssey, Star Trek, and the early Star
Wars movies, which together brought robotics
and the idea of intelligent androids to Hollywood. But since then, we’ve seen substantial
growth in the public vision of interesting possible futures for artificial intelligence, including
the recent film I, Robot, based loosely on the
Isaac Asimov stories, and Steven Spielberg’s
direction of a provocative film that was
expressly called Artificial Intelligence. Whoamong us, 25 years ago, would ever have imagined a Hollywood blockbuster, created by oneof the great directors of our time, with the very
title, Artificial Intelligence? Quite remarkable.
While it is unclear how much the specific story
had to do with what we do every day in our
research labs, nevertheless, there it was, frontand center in international popular culture:what might the far future of AI be?
On the other end of the spectrum—and perhaps more importantly—we have the wellloved Roomba robot vacuum cleaner, from iRobot. Through a remarkably modest invention,
in a way that we may never have foreseen, AI
has finally begun to enter the homes of tens of
thousands of regular people who have no idea
what artificial intelligence is about, but who
know they benefit from what it brings to theirlives.
In virtually every respect, our field has comea long way in the last 25 years.

Centrifugal Intellectual ForceOne of the natural consequences of the growthof interest in AI in the last 25 years has been an
explosive rise in the number of venues inwhich AI researchers can present their work.
We have seen a large number of conferencesdevelop, as well as a large number of new publications, in all of the specialized areas that at
one point were all considered simply core AI.
These include, among others, machine learning, knowledge representation, automated reasoning, agents, robotics, vision, planning,
uncertainty in artificial intelligence, computational linguistics, and data mining and knowledge discovery. In fact, as mentioned at AAAI05, there may be as many as 30 “sister
conferences” that we can draw from for our sister conference report sessions. For quite a longtime in the earlier days of this expansion, interest continued in the core conference—in AAAI
itself—with people also spending time at theirown specialized area conferences. But overtime this seems to have changed. With strong
specialized program committees to review submissions, and a greater concentration of people

WINTER 2006 21

Articles… the specialization and the kind of centrifugal
force pulling people away from AAAI as anorganization and a conference might actually behaving a much more challenging effect on ourability to do the very work of our field.

in the specialized disciplines attending these
conferences, the more narrowly defined meetings have become more attractive. If funding
becomes tight or if people are doing too muchtraveling, AAAI and perhaps IJCAI conferences
are left off of the itinerary; by attending the
Knowledge Representation (KR) conference or
the International Conference on Machine
Learning (ICML), or the International Conference on Autonomous Agents and Multiagent
Systems (AAMAS), many discipline-focusedresearchers are still getting to see most of thecolleagues that they would like to talk to andhear about the latest technical developmentsin their areas.
As a consequence of this, what we hear often
in AAAI organizational circles are concerns
about the future of our own organization:
What is its role? Why are people not coming to
the AAAI conference? Should we bother tohave the conference at all? In fact, if you wereto look back over the last several years’ worthof statements written by candidates nominatedfor AAAI Executive Council positions, you
would probably find a majority who havefocused on the issue of specialization in thefield detracting from AAAI as an organizationand possibly decreasing clarity of the meaningof our central conference. This is a common
theme that has been discussed often in AAAI
leadership circles over the last few years. There
are good reasons for this evolution, and I think
it is a natural phenomenon; it can be seen inother fields and in some ways it may not besomething to be worried about.
But my concern here—and the main themeof my talk—is around the fact that it may be
the case that in our field—in contrast withmany other fields—the specialization and thekind of centrifugal force pulling people away fromAAAI as an organization and a conference might
actually be having a much more challenging effecton our ability to do the very work of our field.
While a natural consequence of the maturity of
the field is the “loss” of some of our subfields,
this specialization and spinning off of areas ofresearch presents us with a deep and fundamental problem.

22

AI MAGAZINE

We have been seeing great progress along
specialized lines in each of the areas of AI, andsome have made extraordinary gains over thelast 10 years (machine learning being one of
the most obvious cases). But in general itappears that while we are getting greater technical depth and greater achievements in our
specialized fields, we do not seem to be getting
any closer to what people might consider “trueAI”—the goal that caused us to start AAAI in1980 and that gave birth to the field in the
1950s and that, to be honest, got most of us
very excited about getting into the field when
we were even back in high school—the grandvision of building a truly intelligent humanlevel AI system. As a whole, the field justdoesn’t seem to be making a lot of progress inthat direction, even while we make tremendous progress in our specialized areas. In myview, this is a consequence of too much focus
on AI-related “components” and not on artificial intelligence itself, which is not simply thesum of its component parts.
Another way to look at this (to pick somewhat unfairly on one subfield) is to consider
the fact that, no matter how outstanding ourlearning algorithms get, it’s hard to call any ofthem “intelligent.” The world’s greatest support vector machine is not an intelligent system that has the robustness, breadth, and common sense that even a three-year old child has—nor will it ever be. In a way, the peculiarity of
our current situation with respect to AI as awhole is exemplified by a simple phrase that iscommonly used in the learning area: we seemto have made a lot of progress in “knowledgefree learning.” Think about that: imagine talking to an average person on the street about AI,
trying to explain that we want to build intelligent humanlike robots, and that we are making
tremendous progress in “knowledge-free”
learning. I suspect that that would be at least
puzzling for a lay person, if not downrightbizarre. Why would you handicap your system
by expressly discarding anything that it hadlearned in the past or knowledge it had gained
our could infer through other means? How
could you even have the concept of knowledge-free learning in a fully integrated AI system? Yet this is common terminology, exemplary of a common point of view.
Similarly, as much as I am personally fond ofwork on the semantic web, not to mention thatmuch of my own technical work was involved
in description logics that provide a foundation
for web ontology languages, no matter how far
we go down that road—no matter how wonderful our description logics become—they
won’t by themselves lead us to artificial intelli-Articles

gence. As with isolated learning algorithms,
they may have fantastic applications, they may
have great intellectual depth, and they may
even make some people a lot of money, but I
do not believe that such specialized tracks willlead us to AI. The same can be said for virtually
all of our subdisciplines, for example, machine
perception and natural language processing.
They may produce wonderful artifacts andgreat scientific breakthroughs, but they don’t
seem individually to be leading us to an artificial intelligence. So my concern goes way
beyond the fact that specialized conferences
are drawing attendees away from AAAI conferences. The centrifugal acceleration that wehave been seeing, with more and more people
spending more and more of their concentrated
time in the subareas and at their own conferences and in their separate journals, may actually be sabotaging the future of our own field at
its very heart.

Intelligence Is a
Many-Splendored Thing
It has been hypothesized that whatever intelligence is (and we obviously have not been able
to fully define it so far), it is a multidimensional thing. Natural human intelligence appears to
be multifaceted. “Human-level intelligence”
does not lie at the end of the road that focuses
only on machine learning algorithms, nor will
it be the result of the world’s greatest natural
language processing machinery taken in isolation. It is somehow a combination of thesethings and more. Let’s think about this commonsensically for a moment: imagine that you
could build a system that was fantastic at all
the kinds of reasoning that people do, but
without the ability to learn. Such a system,
even with all of the knowledge in the world put
into it right now, would no doubt appear smart
in the first instant, but without the ability toadapt as the world changes and knowledge
needs to evolve, it will simply become more
and more stupid over time. That is, systems
that are knowledgeable at one moment, buthave no adaptive capability, ultimately become
stupid because they just stay set in their ways
and hold onto beliefs that are increasingly outof touch with reality. Similarly, systems that“learn” without knowing anything—theknowledge-free learning idea—will ultimately
become very good test takers, being able to spit
back just what their teachers have told them,but they will not be able to apply what they
have learned to truly novel situations, which infact is the essence of the versatility thathumans exhibit every moment of their lives.

Even an average human can learn something in one context and then later use it in a
new one; as a result, the average person is able
to cope with many things that he or she never
specifically anticipated and is not directly prepared for. But you can’t to do that if you justhave a learner that doesn’t know how to store,
represent, and reason with the knowledge thatit learns. In a similar fashion, perceptual systems that may appear to be excellent at dealing
with pure data-driven, bottom-up data collection tasks, but that are not guided by cognition, will not work very well in many complex
situations. When the input is multidimensional and full of interesting tidbits, they will get
distracted by things that are not important or
are not material to the task at hand. They may
not recognize that something is an anomaly, oron the flipside, they may end up paying attention to something that is a completely unimportant anomaly. When you look at humanand animal systems, perception is guided verystrongly by mechanisms in the higher parts ofthe brain in some magical way that we currently don’t understand, but that guidance is very
critical. Despite the fact that our sensors are
independently quite competent, they are notjust data-driven transducers doing blind pattern recognition—they are much more effective when working together and, in fact, whenguided by common underlying cognitive
mechanisms that can synthesize their various
outputs and use expectations to cut throughnoisy, distracting data.
Note that success or failure is not a matter ofthe quality of the individual components. We
are able to use flawed parts—our vision andhearing systems are less than ideal, our natural
language generation systems are imperfect, ourjudgment is flawed—and yet, nevertheless, as awhole, as something that in a way is more than
the sum of its parts, we do very well getting
around in an unpredictable world that throws
things at us that we can never anticipate. So,even the ultimate end products of each of our
specialized disciplines done perfectly well are
not likely to yield an intelligent system without us doing something fundamentally different. This is perhaps the key point of this treatise: AI work done in separate subdisciplines—
even that resulting in practically flawless components—is not itself “AI.” We must consider
the integration and synergies of components
in an overall system to really approach someform of artificial intelligence.
Now the idea that intelligence is actuallybuilt out of many parts and is not a single,
monolithic thing is not a new idea. Many people have talked about this over the years. One

WINTER 2006 23

Articles

Flying in Formationof the things that we did during my recent
tenure at DARPA was to look at ways to test
intelligent systems that have multifaceted
capabilities. It occurred to us—and I want tothank both Dave Gunning and Paul Cohen forthis idea—that we may want to test intelligence skills and capabilities in machines like
we do very broad athletic abilities in humans.
Someone might be the world’s greatest shotputter, but world-leading performance in the
shot put alone doesn’t seem to be indicative ofwhether or not the person is, in fact, a fantastic
athlete. They might be the physical equivalentof an idiot savant and have only a single skill
that doesn’t translate well into any other formof athletic performance. To determine who isthe best overall athlete—the most “athletic”—
we typically use the decathlon. We run a set of
events that themselves aren’t connected in any
obvious way (throwing the javelin may havenothing to do on the surface with running hurdles), but nevertheless, there are muscles andcoordination and general athletic conditioningthat somehow support all of these. Therefore,
by testing a handful of different skills, we cometo the belief that someone who may not evenbe best in any individual event, but somehow
is best in the aggregate, is truly the world’s
greatest athlete, exemplifying what we consider to be athleticism. It may be that at the
moment, given our crude understanding of
natural intelligence, this is perhaps the best (ormaybe the only) way to test intelligence. Being“intelligent” may be the same kind of amorphous mix that being “athletic” is, or alternatively, given our poor understanding of it, amultifaceted test may be the only way toapproach the measurement of intelligence.

24

AI MAGAZINE

Many others, largely from the psychology community, have talked about intelligence as amultifaceted thing, and there are “multiple
intelligence” theories. Similarly, some in AI (forexample, Marvin Minsky in Society of Mind
(New York: Simon & Schuster, 1985) and subsequent writings) have talked about intelligence
being multifaceted. Let us grant for a momentthat indeed intelligence is not a single, monolithic capability. There is one more criticalpoint: intelligence is not created by just mixing
together the individual facets. Indeed, intelligence takes many different individual, special
capabilities—like reading and thinking andlearning and seeing and hearing and talking—but, even if we have all of those pieces eachdone extraordinarily well, if we just toss them
together we are not likely to get an intelligent
system. My sense is that intelligent systems are
not simply the mere compositional sum of
their parts. Individual senses and cognitive
abilities don’t exist separate from one anotherand don’t simply drop their independent conclusions into a centralized data store that they
can all read from independently (the appeal of
blackboard architectures notwithstanding). If
you start to examine what people actually do,
you see very deep and complex connections
between these different capabilities.
Unfortunately, these relationships are thethings that are often missed in our specialized
conferences because in order to analyze andunderstand them, specialists in one area would
need to reach out to communities that havetheir own specialized conferences that are
intellectually quite far away. For example, natural language understanding in humans,
which is often addressed in computational linguistics conferences, really does seem toinvolve understanding—as tautologous as thatsounds, the focus of most natural language
processing work these days is not on understanding, but on doing as much as one canwithout positing an understanding component.
Understanding involves world knowledge, reasoning, and the ability to do inference and see
the consequences of things that you learn andapply them to new situations, which soon getsquite out of the realm of what you typically see
at a computational linguistics conference. As I
have said several times now, learning produces
knowledge as an output and that knowledge
needs to be used in reasoning components;
without this, reasoning systems can chug alongand do work with what they’ve got, but they
end up being pretty stupid unless they can takeadvantage of newly generated knowledge fromlearning mechanisms. Additionally, real-world

Articles

learning in the kind of normal everyday situations in which we find ourselves is heavily
dependent on what we already know, or
expressly seek to learn because of other things
we are thinking about. Further, as I have alsosaid, perception doesn’t really operate independent of cognitive state.
As we examine more and more of the pieces
of an intelligent system—the separate events inthe cognitive decathlon if you will—we findthat we still need to account for the magicalway in which they all have to fit together inorder to create something that is more than thesum of its parts. Perhaps the best way to get
this point across is to quote someone I used towork with (Dado Vrsalovic), who, in a different
but perhaps comparable context, was fond ofsaying the following: “100,000 spare parts flying in close formation is not the same as an airplane.” I am not sure how you launch 100,000
spare parts to fly in close formation, but imagine taking all the parts in a Boeing 777 and putting them really, really close to one another,but not totally integrated in the way we like to
see our airplanes integrated when we fly on
them. Now when you launch this thing, it isn’t
going to stay up in the air very well and, in fact,
it isn’t really an airplane. There is no architecture—an integration that needs to be taken into
account to take the parts and turn the whole
into something that really serves a broader
function.
So somehow, even in the decathlon, while
we test skills separately, they are all being doneby the same person with much overlap in thesubsystems that are used in the various events.
Running hurdles and doing the long jumpclearly use related leg muscles; throwing thejavelin and pole-vaulting both use a running
start and significant arm action. Imagine
though, a field that had emerging specialized
journals and conferences that focused totallyon the mechanics of javelin-throwing. Members of the Hurdle Society and attendees at theInternational Pole-Vaulting Conference would
no doubt be in the dark about what came to be
discussed at the javelin meetings—and in fact
might declare those fields to be so far fromtheir interests that they were no longer interested in attending general athletics conferenceswhere there would be boring and irrelevant
papers on javelin-throwing, sprinting, andhigh-jumping. If the experts in those specialized areas were interested only in creatingmachines that could star in their individual
events, they might be able to get away withthis separatism. Even then, of course, thejavelin folks might miss an important development in the discus-throwing literature that

could provide critical insight and might evenhelp their subfield take a quantum leap ahead.
But much more importantly, if the genesis of
the separate subareas was a once-exciting fieldthat had originally aspired to create a generalartificial athlete, able to compete competently
in many different events—including novel athletic competitions that had not previously
been on the program—then the subareas going
their own separate ways would fly directly inthe face of the original overarching goal. It
seems to me that the way to build a competitive artificial decathlete is not to lose the focus
on the athlete as a whole by working independently on the subsystems needed for specialized events. Certainly the leg and arm systems would need to be studied in detail andemulated in machinery; but their interactions
to produce the rhythm of a long-distance runner or the coordination necessary to achieve a
successful pole vault would need to be a mainstream concern of the field as a whole. Without
attempting to study and build a whole bodywith integrated musculature, where would balance, strength, endurance, and coordination
come from?
Now this is admittedly a naïve view, and I
am no physiologist, but I would suspect that if
you looked closely at how athletic skills are
implemented in natural systems, you would
really need to pay attention to the cross-coupling of different muscle and skeletal components, not to mention the overall cardiovascular infrastructure—and the key point here isthat it is the same thing with intelligent systems. “Learning” isn’t really learning in intelligent systems until it is tightly connected withreasoning. Perception doesn’t mean much
unless it is coordinated with cognition. Language essentially involves thought and understanding.
The implications for AI should be obvious. If
we simply keep our separate components flying in close formation, we won’t ever really get
an AI airplane. And if we build machines thatwin races in specialized areas, we won’t get a
decathlon winner or an athlete capable of succeeding at a newly invented event. We need tothink about how all the parts of an artificial
intelligence should work together, and how
they need to be connected, or we simply won’t
get one. So to my mind, what we really need to
do as a field is to spend a lot more time focusing on comprehensive, totally integrated systems. It doesn’t look like we are going to getthere from here if we keep going in the direction we are going, especially given the centrifugal force that is drawing people away from one
another into their separate subfields.

WINTER 2006 25

Articlesin the system can ultimately affect everythingelse. And that makes building and debugging
the system very hard.

Three Focal Points
Before I get into specifics about some role model efforts, I want to talk briefly about three relatively underdiscussed issues that I believe areimportant. One is the notion of architecture.
Another is what we might call knowledge-richlearning—somewhat the opposite of the knowledge-free learning that I mentioned earlier. The
third is the nasty, unpopular topic of metrics
and evaluation.

Architecture

Mutual Influences
Now, why hasn’t there been a lot of work on
integrated systems? In fact there have beensome wonderful role model projects, but by
and large, if you look at AI, most of it isexplored in specialized venues or in narrow,specialized sessions at conferences. As a result
we don’t spend a lot of time talking aboutpulling the pieces together and creating themagic glue that turns them into intelligent systems rather than just sets of pieces. First andforemost, it is clear that building full, integrated AI systems is extremely difficult. It takes a
lot of talent with a lot of breadth. It requires
sizable teams, software engineering skills, andeven project management skills—the kinds ofskills that are not deciding factors in hiring
university professors. Graduate students arereally not taught this, and it is often the casethat many engineering and architecture considerations are deemed “below” some scientists. It is notoriously difficult to get AI systems
papers published in key places.
Nevertheless, these things are absolutely critical to our ultimate success. As a field we don’tnecessarily have a lot of these skills around.
Building large-scale integrated systems is really,
really difficult. It is also very messy just becauseof the essence of what’s involved. Everything

26

AI MAGAZINE

We are all familiar with architectures in AI systems. We often see block-and-arrow diagramsin papers, and in fact, there is a pretty substantial amount of work largely driven from thepsychological side of things around what are
called “cognitive architectures.” In my view,
however, there is still significant room for workin this area as applied to integrated AI systems.
The cognitive architecture literature has tended
to focus strongly on psychological validity,
matching what goes on in the human brain.
Since our concern in AI is not just the replication or understanding of humans, but in fact a
broader space of possible minds (thanks to
Aaron Sloman, a philosopher and long-time
researcher in the field, for that description), I
think we need to expand our thinking aboutarchitectures that are adequate to support integrated AI systems.
A potentially fruitful question to ask iswhether we can learn from architecture workin other disciplines. If you have ever been
involved with a large-scale software systemeffort, you’ll know that architecture and systems engineering are critical parts of the equation. These arts are practiced by specialists with
substantial technical education in the area, andhard-won experience is crucial to success.
Architectures are not just boxes and arrows put
together by the seat of the pants. System architecture as a discipline has its own principles,
and we may be able to learn a lot by taking abroad look at the literature on software systems
architecture, hardware architecture, and evenbuilding architecture. It would be interesting tosee how well-known principles of good architecture might apply to AI.
For example, here are a few principles (courtesy of Gregg Vesonder) that you might learn
about good architecture if you were taking aclass in software engineering:
A good architecture is one that is amenable toArticles
implementation of a skeletal system with minimum functionality overall in order to test thecommunication paths between the components. One way to get started is to build stubs
for all of the parts of the system without making them rich and fully functional; this allows
you to figure out whether the APIs and thecommunication paths are right before youinvest in full implementation of the parts.
A good architecture would feature well-definedmodules with functional responsibilities allocated on the principles of information-hiding
and separation of concerns with well-definedinterfaces.
A good architecture should feature a very small
number of interaction patterns. Modules that
produce data should remain separate frommodules that consume data. Good architectures avoid doing the same thing in different
ways and in different places.

Principles like these are familiar to computerscientists, but the kind of formal architecture
reviews necessary to successful large software
projects are not common in AI work. I think wehave some important things to learn from theway architecture discipline is applied in moremainstream software projects.
On the other hand, we also need to consider
how building an intelligent system might bedifferent from building a more conventional
software system. It certainly appears to be thecase in the human brain that the architecture
doesn’t look like it does when we construct systems with simple components and do information-hiding and separation of tasks and only
do one thing each in one place in a system.
Neural architectures hypothesized by brain scientists look quite different from those of conventional software systems. One critical element here is the fact that the one role modelthat exists in the world that we’ve been looking
to emulate was not a designed artifact; thebrain evolved over millennia, and in fact, what
you get with an evolutionary approach is very
different and potentially much messier thansomething you would do with a very goodengineering design. So this leaves us with aninteresting conundrum: as we build large integrated AI systems, it would make sense to follow conventional architecture discipline, but
we generally look for inspiration to somethingthat evolved in an organic way and whose currently discernible architecture is very different
from that of the systems we typically design.

Knowledge and Learning
Humans learn, it is clear, in many different
ways. We learn by being taught, we learn by trial and error, we learn by mental simulation andexplanation, we learn by reading, and we learn

in a host of other ways that may or may not all
be implemented differently. When we think
about learning, though, the goal is to produce
knowledge or resources that we can use in dealing with novel situations later in life. So what
we tried to do when I was at DARPA, and what
I think is important for the field in general, wasto move back to the more traditional AI view oflearning. Here I’m not talking about Machine
Learning with a capital “M” and “L” as a subfield, but good old learning as done in core AI
systems, where, for example, we might havesystems as capable as humans are of learningfrom very few examples (including just a singleone). Learning also is profoundly important ingetting around in life when we use it in what
we might call a “transfer” way—for example,
absorbing things we’re taught in a classroom
and actually finding out many years later thatthey apply in unanticipated ways to real life.
We learn things in one context that work out
to be very useful in a different time and context.
Another phenomenon in learning is “laddered learning” or what Dan Oblinger has
called “bootstrap learning,” where learning
works best when principles are presented in a
structured approach, so you learn one “rung”
at a time. We know that when things are presented to us in school in the right order, we canactually learn incredibly complex concepts,
whereas if we just jumped to them right away,
we could never learn them. So we know pedagogically it is very important to ladder things,
that is, to master one rung before we jump to
the next rung, yet that’s not how we are currently approaching research in machine learning.
Consider also the difference between bottom-up, or data-driven knowledge acquisition,and top-down, or goal-driven simulation andacquisition of knowledge. These two approaches come together in learning. For example,
when reading you don’t just read the words;
rather, you read with expectations about what
will come next. You can complete a sentenceeven if you hide the last word in the sentence
and your interpretation of the words you see is
influenced by what you’ve just read. Your forward-looking expectations influence how you
interpret the next sentence, and what topics
might be presented in the next paragraph. Thisimplies a potentially complicated architecture
underlying reading that we ought to include in
our learning research.
A final issue on the subject of learning systems is how to tell whether a system haslearned and whether its learning can be
improved.

WINTER 2006 27

Articles

tions are requirements for the future of the
field, both for our own sake and also for thesake of those who oversee our funding. In particular, in the United States it is very important
to show Congress that we are actually making
forward progress—even if our goals are 20, 50,
or 100 years in the future. It no longer matterswhat our subjective description is of how muchbetter things are than they were five years ago.
Rather, it is important to understand in a measurable and repeatable way where we are heading and how we will get there.
Now this is very challenging. It’s hardenough to apply meaningful evaluation designs to AI component technologies, just given
the nature of research. But the difficulty is magnified when you take to heart the message that
we need more work on integrated, comprehensive AI systems. How will we evaluate intelligence in a machine when we don’t even know
how to measure intelligence in people? But thischallenge alone should prod us as a scientificcommunity to come up with new ideas andapproaches to the issue of evaluation and metrics.

Some Role Models

Deep Space One Encounters an Asteroid.

Photo courtesy, NASA.

All in all, these notions of learning are quite
different from the main current thrust ofmachine learning research. This is ironic in away, since as far back as the first AAAI conference these concepts were posited as the basic
foundations for learning.

A Word on Evaluation
Before working in the government, I, like manyof us in the field, really hated the idea of metrics. It was too difficult to figure out quantitative metrics for knowledge representation andreasoning or concept learning, and when you
did, they tended to distract from the main lineof work. But since I started working with otherson larger integrated AI projects, and securing
funding for the field, I have found that evaluation is an absolutely critical ingredient of ourresearch program. I believe metrics and evalua-28

AI MAGAZINE

I am encouraged to note that there are somegood examples of the integrated, systems
approach to AI. I want to highlight briefly acouple of projects that I think could becomerole models. One involves work in the area ofspace exploration at the National Aeronautics
and Space Administration (NASA) and at the
Jet Propulsion Laboratory (JPL). The other is a
DARPA project called the Personalized Assistant That Learns.
One of the most exciting premises of NASAstyle space exploration projects is the fact thatthe environmental constraints and mission
goals are such that AI must be a fundamental
ingredient. Simply put, you can’t teleoperate aremote vehicle in dangerous or rapidly changing circumstances when there is a 30-second (orworse) radio transmission lag. In these situations the vehicle has to care for itself in a fundamental way. Dan Clancy presented an invited talk at AAAI-04 in which he talked about thevehicles we are sending to Mars and deep space.
Dan described how these machines monitor
and maintain their health on a constant basis.
They may get hit by meteorites, they may experience power failures, they may pick up dust in
moving parts, and they may have faulty electronics—a host of things that require real-timeobservation and response by the vehicle itself,
because there may not be time to get a signal
back to earth before it’s too late.

Articles

In addition to keeping track of their own
health and other prosaic things, these vehicles
need to assure the success of their science missions, following plans for collecting data andgetting them back to earth—and changing
those plans on the fly. Here we have a set ofinteresting constraints from a nasty set of realworld problems where AI—integrated AI—isfundamentally necessary. There are some reallyfantastic projects completed or imagined, ranging from the remote agent mission launchedon the Deep Space One probe a few years ago
to something called the “Autonomous Sciencecraft Experiment,” which has been running on
a continuous basis for some time now. It isclear in this kind of application that planning,
execution, perception, and action are integrated and multilayered. Further, because subsystems can override each other and opportunities
for conflict abound, the architecture of theintegrated system and how it carries out its
intelligent activity are of absolutely fundamental concern. You can’t send a great camera, agreat pattern recognition system, a great set ofwheels and motor controls, and a host of otherseparate parts into space and expect those“spare parts” flying in formation to carry out
the mission. They must be deeply integrated.
One has to think in advance about all criticaland dependent interactions between the partsof the system. In other words, the architectureis the overriding concern. And it is fantastic tosee deployed AI systems of the sort that we allknow and love actually running in real time,
live on $100,000,000 NASA space missions.
At JPL, Steve Chien and others have pointed
out how these issues come into play in thedesign and implementation of their systems.
Steve notes that having multilayered systems,
coupled with the fact that there are different
priorities at different times, means that weneed to worry about redundancy, safety checks,
real-time considerations, priorities, and constant change and adaptation of plans based on
the real world’s effects on the mission. And as
if that were not hard enough, systems likethese can’t really be fully tested until they arelaunched. To run a realistic test on earth to simulate what the mission would be like in spacefor four months or five years is totally impractical, if even possible to design. And so we havesome very interesting challenges to face when
we build a true AI system that’s going to be
deployed in a real-world, real-time, missioncritical environment.
The second example involves some major
efforts under the Personalized Assistant that
Learns Program at DARPA. One of them is SRIInternational’s Cognitive Agent that Learns

and Organizes (CALO)—an artificial secretary,
one that aspires to do virtually all of the kindsof things that a good human secretary can doin the office. What CALO is trying to emulate
is not a Ph.D-level physicist, but rather, theseemingly mundane tracking, learning, andreminding aspects of a good secretary who canadapt to real-world circumstances, improveover time, become personalized to the person
he or she is supporting, and take into accountthe many small things that really make everyday life challenging. This is of course a very different context than putting a rover on Mars,but equally challenging and exciting from acore AI perspective. The CALO team has identified six very high-level areas of competency
in which the system must learn and succeed:
organizing and managing information, preparing information products, observing and managing interactions, scheduling and organizingin time, monitoring and managing tasks, andacquiring and allocating resources. The multiinstitutional team has been working for several
years now on a very wide variety of AI subjects,
trying to put this all together into an integrated
system.
To build a CALO assistant, SRI International
has served as the systems integrator for a number of technology specialists. Importantly,
from the very beginning SRI instituted a systems approach that required, among otherthings, software engineering discipline, a chiefarchitect, and a release schedule for softwarethat will be tested as a system and used as a prototype in DARPA’s evaluations. Of course, agreat deal of fundamental research work forthis project is being tackled in many wonderful
universities. But CALO is more than the sum ofthese individual research efforts. It starts withan architecture and a design approach thatbrings all of these pieces together in an integrated AI system. The architecture allows specialized learning methods, created by individual scientists at different institutions, to beinserted by using common APIs and built from
common core learning algorithms. These modules interact in a well thought out, well-engineered way using a knowledge base, reasoning
methods, perceptual apparatus, and otherthings that go into a complete integrated AI
system. It is the architecture for bringing thesethings together where they can learn from one
another, interoperate, and deal with inconsistencies that is the essence of this project, andone of its greatest challenges.
The results so far are very promising. Afterthe first year of the project, the SRI team developed a flexible, true AI architecture with many
components that could interact in at least a

WINTER 2006 29

Articles

Illustration courtesy, SRI International.

CALO: Cognitive Assistant that Learns and Organizes.
minimal way to accomplish a very wide varietyof things—ranging from moderating meeting
activity and taking notes, to preparing briefing
packets that you can take in advance to a meeting. There are also obvious things such as helping with putting mail in folders and dealing
with calendars, and so on—the kinds of things
you would expect from an office assistant. But
the exciting thing to me about this project isthe deep integration between the many components and the different types of capabilities
that CALO brings to the table. Tom Dietterich
recently noted some specific ways in which the
CALO model has driven new types of research,outlining to me some really interesting novel
challenges to people doing work in machinelearning. In fact, one of the more exciting
things about this project is the way that
research in machine learning has been affectedby the requirement of building this integrated
system. One of Dietterich’s observations is thefact that learning and reasoning must be combined in a tight fashion in CALO. This means
that, especially when you’ve got multiple
learning systems, when a prediction is madeand the world either verifies or refutes that prediction, or the user provides additional feedback (“No, that was the wrong document togive me for that meeting”), the system must
keep track of the new knowledge and assign
credit to the component or set of componentsthat collaborated to make that recommendation. But there may not be an obvious pathback, and in fact, different components cancollaborate to come up with a recommenda-

30

AI MAGAZINE

tion, making it very difficult to assign credit. So
here is an important, somewhat novel problem
in machine learning that might not have surfaced if the focus were simply on a boostingalgorithm or support vector machine. Similarly, this kind of integrated, long-lived systemneeds to formulate its own learning problems.
Because it is running autonomously and is trying to be helpful, and because it knows it needs
to improve, it needs a plan for acquiring, in an
unobtrusive fashion, a set of labeled training
examples and to evoke feedback from the user
without being too annoying. These are things
that people just do naturally; CALO has to figure out how to do this using deep interaction
between a planning component, perception
and action, learning, and reasoning technology that are all part of a complex, always-running, always-evolving system.
Not too long ago, Dietterich observed to me
that in working on CALO, he was now talking
with people he hasn’t spoken with in 25 years.
In 1980 at the first AAAI conference, we were
all working on the same thing—it was AI.
While we were individually focused on different pieces, we were at the same time working
together and communicating. Twenty years
ago, learning people didn’t totally divorce
themselves from knowledge representation
people, and natural language people didn’t go
off and spend their time at their own conferences. Today, projects like CALO and theNASA-related projects have rekindled the kindof collaboration and communication that we
had at the beginning. I hope this is a harbingerof the future—a way to bring the field back
together and help us develop true artificial
intelligence systems.

Versatility, Purposeful
Perception, and Doing
Something Reasonable
One of the historic frustrations in our field isthat when we have built something that we
believe is a successful example of AI, as it getspress and gets analyzed, it seems no longer to
be AI. Instead, it becomes some form of engineering or just a tour de force. I believe that
this is not really a great mystery, but that we
usually bring it on ourselves by solving specialized problems with components of intelligence,not complete, integrated intelligent systems.
Deep Blue is a classic example: what succeededin the very specialized domain of chess wasn’tan AI system embodying general intelligencewith perception, knowledge, reasoning andlearning, but a set of very special purpose algorithms that allowed the system to succeed at aArticles

specialized task. We see this all the time withthe specialized tasks that AI addresses. (It’s fairto say that chess is such a specialized problem
that in the end we discovered it didn’t reallyneed general intelligence. But if so, we certainly need to find some challenge problems forthe field that do need true intelligence.) So weneed to ask the question: “what really is AI andwhat is intelligence about?”
To my mind the key word may be versatility.
(I’d like to thank Rich Doyle for first getting me
to think about versatile systems when I was at
DARPA.) Humans are versatile: they can takeon new missions all the time; they can learn todo new things. Nils Nilsson, in a paper talking
about human-level AI and grand challenges forthe field, talks about educable, or what he calls“habile,” systems. Versatile or habile systems
embody the real value that true intelligencebrings to the table—that is, the ability to doalmost anything, maybe not brilliantly or wonderfully, but at least adequately well, and to be
trainable to do new, previously unanticipatedthings. I think we need to pay more attention
to this notion of versatility in our field. When
the Rover completes its planned mission on
Mars and yet is still functioning, it would be
wonderful if it could adapt to a new mission—
be “re-purposed,” if you will—merely by us
explaining to it what was needed or educating
it didactically on a different class of missions. A
human could easily do that. However, this isalmost impossible to do with a classically structured NASA spacecraft because the mission isbuilt into the architecture of the system.
General intelligence, or real AI, would allow
us to describe, in a declarative way, a brandnew mission and a recommended approach tothat mission that the system would understandand could then take on. In fact, one of the earliest thoughts on a true artificially intelligent
system was McCarthy’s proposal of an “advice
taker,” back in the late 1950s.1 While we’ve
never really built such a system, McCarthy’s
original idea remains much more about true AI
than most of the “specialized AI” systems wehave built since then.
Another area that I believe is well worthy of
more attention is what Manuela Veloso andothers have called “purposeful perception.”
This is not passive absorption and purely bottom-up processing of sensor inputs, but rather
perception guided by and influenced by higher-level cognitive processes and current needs.
What in the perceptual field needs paying
attention to, and how sensor inputs should beinterpreted, is clearly driven at least partially ina top-down way in natural systems. This kindof architecture is worth understanding andemulating. In the CALO project, for example,
the perception subsystem should be influenced
in an appropriate way by the overall current
priorities, upcoming tasks, things that havebeen learned, and similar cognitive aspects ofbeing intelligent.
Finally, a great mystery of naturally intelligent systems is that when presented with new
situations, we are almost always capable ofdoing something reasonable. Our responses
may not be ideal, but we don’t simply go catatonic when we are faced with a new situation.
We, generally speaking, muddle our way
through and most often arrive at an adequate
outcome. Afterwards we can reflect on what
we’ve done and learn from our experience anddo things better the second time. This is a concept we can all understand, at least informally,
but reasonableness hasn’t been thought about
a lot in AI. Why is this? As with the other issues
just discussed, I believe this escapes serious
study because it is another virtue of an integrated, comprehensive intelligent system, notjust a collection of spare parts hanging around
together. And we’re still paying almost total
attention to the spare parts. Versatility, purposeful perception, and reasonableness are all
exemplary characteristics of real, honest-togoodness intelligent systems, and all seem torequire the kind of integration that I have beentalking about.

Measuring Progress
Once we build an intelligent system, how do weknow if it’s any good? How do we tell if it isdoing well, especially if its behavior is kind of
reasonable, it is not great at any one thing, andevery piece of the system influences every otherpiece? How do we actually measure progress?
Again, the CALO experience points in a promising direction. For the project, we hired a fullfledged, independent evaluation team to identify the task challenges for the system and thendecide on the right measurements to be appliedyear after year—thus giving essentially the sametest each year, but without the system being
able to cheat because it saw the test the year
before. In a way, the project has developed an
SAT-like test for the system by creating a set ofparameterized questions, or question templates,that can be varied from year to year, but that
require precisely the same capabilities and intelligence to answer each year. For CALO, we can
ask about a new set of meetings or tasks for theoffice assistant each year but use the same testing to end up with a quantitative score of performance and track progress from one year to
the next. What really matters is whether we can

WINTER 2006 31

Articles

show that a system can learn, that it will ultimately do much better than an engineered system because its knowledge growth and its
improvement in skills just outstrips what thedesign team can do as programmers.

AI Grand Challenges
If our goal is to build a machine that has someform of true, versatile intelligence, and yet thefield is not progressing much in the right direction, how can we stimulate more of the rightkind of work? Certainly having a system “situated” and interacting with the real world is an
important element, partially because whatever
AI system we build will need to work in a realcontext. But there is more. The real world is
messy and unpredictable, so I think we need tospend more time building systems that we
experiment with in the wild, not just in thelab—we need our AI systems out there dealing
with the messiness of the world, dealing withnoise and misdirection and things that we wish
we generally didn’t have to deal with.
Another important element of a refocusingof the field is to move AI to be thought of moreas a system science—not a handful of disconnected subdisciplines, but an overarching viewof how those disciplines and the resulting technologies they create interrelate. In fact, I think
we need to find room in our conferences for all
aspects of computer science that can contribute to and benefit from the needs of complex AI systems. How can we do this, given ourcurrent commitments to our universities andcompanies, our students, and our careers,which tend to demand simple allegiance tofields defined by existing conferences and journals? What kind of game-changing event
might create the motivation we need toleapfrog to a new paradigm? I suggest that welaunch a new series of “grand challenges.”
Such challenges, like President Kennedy’s boldpromise in 1961 to put a man on the moon(and return him safely) by 1969, can grab theattention of a broad group of people. They
present hard problems to be solved—hardenough to require lots of work by lots of people, and when defined right, they move fields
in significant leaps in the right direction. The
really good ones over the years are interesting
because they have really demanded the kind of
integration that I have been advocating. In myview, with appropriate sets of challenges, wemight actually generate a tremendous amountof new excitement in the field of AI systems,
which doesn’t really have its own separate subarea or conference.

32

AI MAGAZINE

Criteria for Successful
Grand Challenges
There are aspects of creating such challenges
that require some serious thought. First andforemost, we need a problem that demands aclear and compelling demonstration of AI. Suchproblems, as I’ve intimated above, will need tostress versatility, adaptation, learning, reasoning, and the interaction of many aspects ofintelligence. Next, a successful challenge cannot be gameable in a way that somebody could
win the prize with very specialized engineeringthat doesn’t get us closer to an AI system. Thisrequires extensive thinking about the groundrules for the challenge. Another thing that isreally critical to a successful grand challenge isthat the measurements—the success criteria—
are well-defined, easy to articulate, and understandable to any lay person who might participate. For example, the success criterion for the
DARPA Grand Challenge autonomous vehicle
competition in the desert was simple: travel on
the ground along a prespecified route frompoint A to point B in less than 10 hours, withno human intervention—period. You don’t
have to say much more—people know what
you want and what it takes to succeed.
Another key aspect of a good grand challenge is to make the problem hard enoughsuch that it is not clear that it can be solved
and to allow partial successes to lift the generalstate of the art each year. Here, failures along
the way can point the direction we should go
to improve things—a critical aspect of good
grand challenges. (Thanks to Paul Cohen for
the notion of failure being “diagnostic” as an
aspect of a good grand challenge.)
Of course, the challenge has to be ambitious
and visionary, like putting a man on the moonor cracking the human genome or beating theworld’s chess champion, but it can’t be unrealistic. As much as I would like to put out thechallenge of building a Star Trek–like transporter, I don’t believe such a thing would bepossible in my lifetime (if ever). People will get
frustrated if they don’t see their way clear topossibly solving the problem and winning the
prize. We can debate what the right time horizon is, but we certainly want to have solid faiththat someone will succeed in somewhere
between, say, 5 and 15 years, with reasonable
doubt about solving it in the first 5 years.
That’s what makes it really fun.
A successful AI grand challenge would need
to be compelling to the general public, generating good press and building general excitement. The public has to find it compelling andexciting enough to follow the stories in thepress and be rooting for the AI researchers toArticles

succeed. Finally, a successful challenge has tobe strongly motivating for the research community, one that gets specialists talking witheach other and exploring new collaborative
paths to success, asking new questions andquestioning past beliefs.
The aforementioned criteria should beapplied rigorously to grand challenge suggestions. Hopefully we will end up with not just alaundry list of possibilities but rather a compelling, exciting quest that will attract
researchers and lay people to find a solution. I
believe AAAI could play a very important parthere.

Some Examples
What specific problems might qualify as highquality AI Grand Challenges? Back when I was
at DARPA we held a workshop to identify,
debate, and refine some ideas. Here, briefly, aresome of the ideas that were still on the board at
the end of the workshop.
Truly read and understand a book. Here, thechallenge would be to build a system that canread a book (a textbook, say) and successfully
answer the questions at the end of a chapter or,
better yet, sequential chapters. To make such achallenge compelling, it probably has to be theequivalent of what a student does in a highschool science course, with some details andground rules appropriate to computer programs. But pretty much, that’s it. We would
have to think through the ground rules andvery carefully plan a nongameable way toadminister the test. But the overall problem is
intuitive and challenging. We could easily
imagine a poster announcing the challenge.
Take the Scholastic Aptitude Test (SAT). Hereeach system would take the SAT on the same
Saturday morning that human students do invirtually the same context and would be
required to end up in the 90th percentile ofhigh school student performance. Again in thiscase, it is very simple to describe the challenge
and it would be very clear to even the generalpublic how monumental an achievement thiswould be. Many people complain that the SAT
is not a “good” or adequate test for humans ofhigh school age. But no matter how flawed weall think the SAT is, we use it, our kids take it,
and it matters for getting into college. It is fairly broad and involves a substantial amount ofbackground knowledge and the ability to apply
learned skills in novel settings. Wouldn’t it beexciting if a computer program could, under
the same ground rules as the students, actually
do well?
Write an essay that is graded well by a highschool teacher. Picture your typical school

research paper assignment: write a short essayon an assigned topic, first doing some researchusing defined information resources, and thenproducing a coherent paper in a finite periodof time. For example, I may not know muchabout the French revolution, but if I were given
two hours and access to the world wide web, I
could probably put together at least a plausible
two-page essay on the causes of the French Revolution, or what happened at the Bastille, or
the origins of “Let them eat cake!” Whyshouldn’t a machine be able to do this? Why
don’t we have computers try to write essays
just the way we have kids do it, have the essays
graded by human teachers in exactly the sameway, and then as soon as one of them gets an“A,” celebrate a monumental step in the history of AI?

WINTER 2006 33

Articles

Be a “search engine” in the real world. Why
don’t we build search engines for the physicalworld? In fact, why don’t we create a grandchallenge treasure-hunt adventure? Imagine
we are going to take mobile robots entered into
the competition to a previously undisclosed
physical location. It could be a house, but one
that the entrants have never seen before. We
tell them to find a $1,000,000 treasure in achest in one of the rooms (or some other challenging physical and mental quest). Make thecircumstances difficult enough such that it
would test a human. In the end, the treasure
would be much more than watching the robot
bring out the chest—the real treasure would be
the advances in AI systems and science that arebrought to the world.
Of course, the sky is the limit here. You canimagine a very wide variety of challenges,
which would be successively harder and insome cases would ultimately appear to take truly an integrated AI system to actually solve a
problem like this and win the grand prize. Forget a vehicle that can drive itself across thedesert to Las Vegas—let’s challenge our colleagues to build intelligent machines to drive
to Las Vegas from anywhere in North America,
find a reasonable hotel for a night, check in,and then win some of its designers’ money
back in a casino, walking to the cashier with its
chips, and depositing money in the right bankaccount….

Conclusion: The AAAI ChallengeThis somewhat meandering discussion leads
me to a very simple and obvious conclusion.
Who will lead the way to these new, integrated
systems? Who will bring together our apparently divergent research specialties? And who
will lead the charge in creating, motivating,and running a set of AI Grand Challenges?
Clearly, if we are going to focus more on integrated systems work, what you might call true
AI, we still need a venue for exploring and sharing our results, talking to one another, and also
discussing and running these grand challenges.
I don’t particularly think that this should be agovernment activity, although if we can convince DARPA to sponsor another grand challenge that demands more integrated intelligence, that would be great. I think these things
should be the province of the researchers andscientists and leaders in the field.
The obvious place to do this is, of course,
AAAI. In some ways, this is exactly what theorganization was founded to do more than 25
years ago: to be the central place where all of us
come together, in a conference setting, in

34

AI MAGAZINE

print, and as an organization. It should sponsor
activities to bring together people who ultimately need to collaborate, and create an
atmosphere where those researchers could
work together closely to build truly integrated
AI systems. This movement back to the roots of
AAAI has to start with the AAAI leadership, andwith each of you.2 We all need to do what Tom
Dietterich observed that he was forced into
doing: all of us should in fact start talking
much more to colleagues we haven’t spoken toin 5, 10, or 25 years. Only through such personal, substantive collaboration, facilitatedthrough AAAI’s operations and meetings, will
we be able to reach our goal of building thekind of AI system that was the heart and soulof the field when AAAI was founded in 1980,
and was the dream of the founders of the fielda half-century ago.

Note
1. See John McCarthy’s “Programs with Common
Sense,” Teddington Conference on the Mechanization of Though Processes, December, 1950 (www-formal.stanford.edu/jmc/mcc59/mcc59.html).
2. I am pleased to note that AI Magazine (27[2]:2006)
recently featured a special issue with the theme,“Achieving Human-Level Intelligence through Integrated Systems and Research.”
Ron Brachman is the vice president of worldwide research operations at Yahoo! Research, theadvanced research arm of theworldwide leader in Internet services. He earned his B.S.E.E. degreefrom Princeton University (1971),
and his S.M. (1972) and Ph.D.
(1977) degrees from Harvard University. Brachman has made numerous important
contributions to the area of knowledge representation, including Knowledge Representation and Reasoning, a recent textbook written with Hector Levesque.
Brachman started his career at Bolt Beranek & Newman, spent several years at Fairchild/Schlumberger’s
Lab for AI Research, and, having developed a worldclass AI group at AT&T Bell Laboratories, moved into
senior research management jobs at Bell Labs andAT&T Labs. He served as the director of DARPA’s
Information Processing Technology Office from 2002
to 2005, and there developed IPTO’s Cognitive Systems initiative, which brought hundreds of millionsof dollars to the national research community. Brachman was program chair for AAAI-84, and was thepresident of AAAI from 2003–2005. He is a founding
AAAI Fellow and a Fellow of the Association for Computing Machinery (ACM). At the International Joint
Conference on Artificial Intelligence in January of2007 he will be awarded the Donald E. Walker Distinguished Service Award.