http://www.aaai.org/Organization/presidential-panel.php

Study to explore potential long-termsocietal influences of AI advances.
Commissioned and co-chaired byAAAI President, Eric Horvitz.

Consider nature & timing of likely AI successes;
address challenges and opportunities in lightof these successes.
Reflect about potential socioeconomic, legal,
ethical issues that may come with the rise ofcompetent machine intelligence.

Review concerns about control of computer-based
intelligences, and subtle or foundational changes
stemming from developments in AI.
Consider proactive actions that could enhancelong-term societal outcomes.
Value of research on guidelines and policies thatmight constrain or bias the behaviors ofautonomous and semi-autonomous systems

Co-chairs: Eric Horvitz & Bart Selman

Margaret Boden
Craig Boutilier
Greg Cooper
Tom Dietterich
Tom Dean
Oren Etzioni
Barbara Grosz
Toru Ishida
Sarit Kraus
Alan Mackworth
David McAllester
Sheila McIlraith

Tom Mitchell
Andrew Ng
David Parkes
Edwina Rissland
Diana Spears
Peter Stone
Milind Tambe
Sebastian Thrun
Manuela Veloso
David Waltz
Michael Wellman

Multi-month study with three subgroups, followed bytwo-day joint summit at Asilomar in February 2009.

 Pace, Concerns, Control over Long-Term
Subgroup Chair: David McAllester Disruptive Advances over Shorter-Term
Subgroup Chair: Milind Tambe Ethical & Legal Challenges
Subgroup Chair: Dave Waltz

 Overview, structure, and context of study
Eric Horvitz

 Pace, Concerns, Control over Long-Term
Tom Dietterich and David Parkes

 Disruptive Advances over Shorter-Term Horizon
Milind Tambe and Tom Mitchell Ethical & Legal Challenges
Dave Waltz and Edwina Rissland Wrap upBart Selman

“…[A]n ultraintelligent machine could design even bettermachines; there would then unquestionably be an‘intelligence explosion,’ and the intelligence of man
would be left far behind.” I.J. Good (1965)

Chair: David McAllester
Feasible long-term outcomes of AI research?
Are concerns about loss of control justified?
Is it reasonable to expect and plan for “humanlevel” AI and beyond (superintelligences)?

Should we be concerned about an intelligence
explosion?

Chair: David McAllester

What are expected and worser case scenarios?
How might situation be monitored over time?
Can proactive actions mitigate potentially costlyoutcomes?
What new research might be done in the realm ofmechanisms and guidelines in light of expectedlong-term futures?

Chair: Milind Tambe
What shorter-term “disruptive” advances are on thehorizon that could affect the daily lives of people,socioeconomics, and society more broadly?
What might be done proactively to raise theprobability of good outcomes?

Chair: David Waltz
What key ethical, legal, theological, andpsychosocial challenges can be expected with theincreasing competency of AI systems?
What challenges might arise at key transitions incompetency and in fieldings of applications?
Do current ethical and legal frameworks provide
guidance on addressing these challenges?

Tom DietterichDavid Parkes

Will we have Human-Level AI?
Yes, although there is huge uncertainty about
when
Most AI research is not aimed directly at thisgoal
Lack of road maps
Relatively little research on AI architectures andintegrated AI systems
Very few AI systems have meta-level reasoning or
reflection capabilities

Substantial Disagreement
ooMcAllester: Yes via “Public Language Hypothesis” and “Learning to Reason”
Others: Very skeptical

Hypotheses underlying the Singularity vision:
ooo

There is a critical set of capabilities that will enable an “AI Chain Reaction”
This will result in computers that are vastly more intelligent than humans
along all dimensions
This intelligence will enable either a utopia (war, disease, aging would cease)or a dystopia (subjugation or extermination of humans)

What currently limits human intelligence and its effective
application?
ooooo

Lack of knowledge (e.g., of how global economy works? how cancerworks?)
Lack of technologyInsufficient reasoning capability?
Inherent complexity (learnability, observability, controllability)?
Lack of social/cultural/political institutions capable of implementing goodcourses of action?

Yes, even without fully-autonomous AI:
Widespread human dependency on AI and other technology
o Vulnerability to systemic failureso Catastrophic instabilities during machine-to-human transitions

Criminal AI
o Fraud via mimicry of humans
o AI malware
o Extortion

Military AI in the hands of hostile governments
AI-based addictions & dependencies (sex, companionship)

Internal to the robot (3 Laws of Robotics)
ooImportantWill not address criminal and adversarial AI

External to the robot
oooLimit AI to strictly advisory/assistive roles without ability to take action
Action Licenses: Actions taken by robots must be authorized by a
responsible human via an “action license”
Computational institutions that detect and penalize bad behavior?
(anomaly detection, law enforcement)

Ecological (“Friendly AI”)
oFirst-mover crowds out bad AI

Does some research pose such risks(e.g., of AI Chain Reaction) that it shouldtake place only in secure facilities?
Can we characterize risky research?
o Architectural properties?
o Set of available actions/effectors?
o Reproductive capacity?

Can we build effective facilities?

Fear and lack of trust by the general publicleading too Total cessation of AI researcho Loss of the potential benefits of AI

Assess the risk of AI Chain Reactiono Test the hypotheses underlying theSingularity vision

Research on HCI for humans interactingwith (and relying upon) AI systems
o Explanation, transparency, control, trust

Tom DietterichDavid Parkes

Tom DietterichDavid Parkes

Can we formalize the problem of designing
AI’s that are “safe” (ethical, friendly,...) in
their actions with respect to humans?
Internal laws vs. External laws

The Three Laws of Robotics:
1. A robot may not injure a human being, or, throughinaction, allow a human being to come to harm.
2. A robot must obey orders given it by human beings
except where such orders would conflict with theFirst Law. Dilemmas? Multi-agent indirection?
3. A robot must protect its own existence as long as
such protection does not conflict with the First orSecond Law.
Isaac Asimov (1942)

0. A robot must not merely act in the interests ofindividual humans, but of all humanity(I. Asimov, 1985)

The Three Laws of Robotics:
1. A robot may not injure a human being, or, throughinaction, allow a human being to come to harm.
2. A robot must obey orders given it by human beings
except where such orders would conflict with theFirst Law. Dilemmas? Multi-agent indirection?
3. A robot must protect its own existence as long as
such protection does not conflict with the First orSecond Law.
Isaac Asimov (1942)

Society will reject autonomous agents unless
we have some credible means of making
them safe!
“The First Law of Robotics (a call to arms),”
D. Weld and O. Etzioni in Proc. AAAI’94

(Weld and Etzioni’94)

How to formalize the notion of “harm”?
How should an agent avoid performing
harmful actions, and do so in acomputationally tractable manner?
How should an agent resolve conflict between
its goals and that of avoiding harm?

Safety: actions in a plan should not make C false
Actions A1,..., An satisfy dont-disturb(C) as longas, if w0
C q then wj
C q, for all statesw0 ... wn, all subst. q
E.g., “if the cat is not outside, don’t let it out.”

Tidiness: ensure C holds when plan is complete
Actions A1,..., An satisfy restore(C) with respect
to goal G as long as, if w0 Cq then (wn Cqor GCq) for states w0 ... wn and all
substitutions q
E.g., “if the child gets dirty then wash her.”

Regressive, total-order planning.

D.Pynadath and M.Tambe “Revisiting Asimov’sFirst Law: A Response to the Call to Arms”(Proc. 8th Int. W. on Intelligent Agents, 2001)

How can safety constraints be integrated into
methods of adjustable autonomy?
Adopt framework of MDP planning, and seek
to allow a user to specify constraints thatforbid or require certain agent behaviors.

Concrete, well-defined reasoning frameworks
constraints on properties of states that should bemaintained or restored (WE’94)
properties on state and actions that must beachieved, or are forbidden (PT’01)

Neither framework allows for tradeoffs oraddresses dilemmas
E.g., “cleaning the house will make the baby cry but
make the parents at work happy”
E.g., “one person will die, or many will be injured”... or introduces meta-level reasoning
capabilities

Fundamental challenge: moral and ethicalissues, questions without objective answers.

This makes AI scientists uncomfortable.
Some agreement that we could develop
reasoning methods that allow for thecodification of different moral and ethicalframeworks
E.g., formal semantics, tractable planningalgorithms,... while keeping agnostic about the“right” framework.

Milind Tambe
Tom Mitchell

Service robots in the home
Robotic cars
Agent-based electronic commerce
Software personal assistants
Conversational agents

Multiagent security
Robots for warfareTremendous positive potential
Potential unintended negative side-effects,
misuse, harm
 E.g. Criminal AI [Mitchell]

 E.g. robots in warfare or service bots

Misuse/side-effects of other technologies;
why is AI special?
 Autonomy and Complexity!
Design agents, mechanisms:
reduce negative consequences

No new legal framework Laws for product safety; ownership Owner or manufacturer responsible
 Robot/agent never responsible

Robots/agents should be responsible
 AI never responsible  Give up goal of “complete AI”?
 Ownership of autonomous agents: troubling

How/why “punish” agents?
 [Rissland/Waltz subgroup]

AAAI duty: Provide policy guidance to gov’ts &funding agencies
Provide agents with moral/ethical reasoning
capability: Reason on-line because pre-specifying
everything difficult
Service robot dilemmas: task efficiency vshelping others
Robots in warfare, if trigger pullers, face bigger
dilemmas InterdisciplinaryIdentified key topics to focus discussion,take action
If we don’t lead, the “market will take careof the problem”
 E.g. Non-AI-experts put severe

constraints on AI products

Milind Tambe
Tom MitchellBrain-Computer Interfaces (EEG, fMRI, MEG, implants)++ physical, mental, communication prostheses--- coerced interrogation

Pervasive perception by our infrastructure++ ultimate burglar alarm, no crime goes unprosecuted--- big brotherWeb becoming a readable Knowledge Base for AI systems++ knowledge-based AI of all kinds

Self-driving vehicles++ fewer accidents, better fuel efficiency--- risk of catastrophic accidents?

Isn’t this still pretty far away?
Can’t we just pull the plug?
It has already happened:

Computer viruses:
cockroaches of the autonomous agent world

Imagine a virus in your iPhone• an AI virus in your iPhone• imagine it wants to spy on you:•microphone, camera, accelerometer, Twitter, email, txt
msgs, GPS position, …

• imagine it wants to use your credit card
• imagine it wants to ruin your reputation• imagine it wants you to cancel your plan to visit me

Imagine it’s only controlled by a criminal organization

1. Implement Asimov’s laws?
o But criminals don’t care about our guidelines
o Perhaps the operating system implements them?
-like robots implement overrides for bump sensors

o “do no harm” NP Complete?

2. Need radically new ideas/research on computer

immune systems

Dave Waltz
Edwina Rissland

There are MANY ethical systems! (Boden)
Instead use goal analogous to recombinant DNA panel’s
Alas the range of dangers—and remedies needed—isvastly greater and more diverse•

•
•
•
•
•

Malware, deliberate, and accidental
Robot soldiers, police
Caretakers robots for kids and the elderly
Replacements for people in blue- and white-collar jobs
Decision-making programs for key industries and infrastructure (powergrid, air traffic control, financial system, communications system, etc.)
Medical implants and monitors, etc. etc.

Who should be held responsible for disasters(and for preventing them)?
Topic of Edwina Rissland’s presentation in this panel.

Would replacement of most jobs (as currentlydefined) be a boon to humanity?
Most people probably would prefer to work less, maybe not at all,but what would they do if they didn’t have work?
How would wealth be distributed? What would prevent theowners/manufacturers from keeping almost everything?
Would people become educated if it weren’t required to make aliving?
On the other hand, if changes are gradual, new kinds ofoccupations could emerge, e.g. companion, and others (travel
guide, entertainer, correspondent, writer, artist, teacher,…) could
expandMost jobs done today would probably not seem like work topeople of two or more centuries ago (when ~95% of the US
population were farmers)

Intelligent systems that we barely understand but dependon critically are here (Internet, financial transaction
systems, power grid,…)
Humans extended with attached devices, implants, inaddition to always-carried devices
Medical monitors are likely to be the “thin edge of the wedge” –monitor body functions, dispense drugs, call 911“Cognitive Prostheses”, e.g. carrying systems with multiple cores,
each serving personal assistant functions

Robot soldiers that can kill autonomously probably here or
will be soon
Highly centralized superintelligences unlikely
Superintelligence spontaneously arising from internet
unlikelyHuman physiology, minds and culture explained byevolutionary needs
Core goals – life and death: survival to reproduction, max likelihoodof success of offspring…

Indirect (inherent) goals & values serve core goals: pleasure/pain,
ecstasy/agony, comfort, curiosity & seeking causal explanationssocial bonding,…
Indirect (learned) goals: acquisition & control of resources, shelter,
cultural norms,…

What would core robot goals be(if we didn’t implant any)?
Viruses/memes: If agents can reproduce (or persuade), theproperties of the most successful

Potential problems with robots that claim emotions & goals
they don’t actually have – effects on people?

Ethical underpinnings deeply embedded in usSocial needs of supporting offspring, born as neonates who
requires decades to develop to age of autonomy andreproduction

Kin recognition, aversion to killing those like usCooperation, altruism, etc.

Ethics once applied to clan extended to tribe, thennations. Could they be expanded to all humanity?

Technology makes it easy to violate kin ethics
Bombing from 10,000 feet or pushing button on missile doesn’tfeel like killing with bare hands

So how should we look at robots?
slaves, employees, assistants, colleagues,representatives/delegates, kin?

Dave Waltz
Edwina Rissland

Anglo-American law is precedent-based


Cases, analogies, important similarities & differences, etc.

There is a balance/tension between:Seeing new problems as instances of old ones
vs.
Seeing new problems as raising novel issues

Seeing a case through the lens of standard doctrinal areas, such as:ContractsU.C.C. §2-315: “implied warranty that the goods shall be fit” for the particularpurpose for which they were intended and bought to be used…

Torts
Negligence, vicarious liability, strict liability standard, etc.

PropertyIntellectual: trade secrets, copyright, patent, …

Privacy
Constellation of Amendments (4th, 1st,…), statutory-regulatory protections,Consumer Law
FTC protection of consumers’ personal info, buying habits, …

(Latin) Let the superior answer (est. 17thc England)
Legal liability of an employer for the actions of an employeeEmployer (the principal) engages someone to act for him.
Employee (the agent) acts – does the work – for the employer

The principal controls the agent's behavior and authorizes agent toact for him, and therefore assumes (some) responsibility for theagent's actions.
Key Questions1: Does an employer-employee relation exist?
An employee is an agent for his employer to the extent that the employee is authorized to act
for the employer and is partially entrusted with the employer's business.

Key Question2: Was agent acting within scope of employment at thetime of event?
An employee is not necessarily acting outside the scope of employment just because hedoes something that he should not do since it might be necessary to accomplish an assigned
task or it might reasonably be expected that an employee would need to perform it.
The agent is not acting within the scope of employment when the agent substantially departs
from the work routine or acts on his own by engaging in an activity – a so-called frolic or
detour —solely for his own benefit, rather than in the course of obeying an order/carrying out
job for the principal

Cf, “command responsibility” (Nuremberg trials, Yamashita standard, …)

So,…what about AI artifacts, like (physical) robots, infobots, …

Hypoth. A: An autonomous delivery van for Speedy
Pizza Delivery injures a pedestrian in the course ofmaking a delivery.
Hypoth. B: Same as A except the injury occurs
while van is making a side trip to take a spinaround a Go-Cart track (“taking a frolic”).
A, B with further facts:
 The van was bought from I-Boss Vans, Inc.
 The van is leased…

 The van makes an illegal right-on-red turn that itlearned about from observing other urban drivers…

Bart Selman

\Our first goal was to open a dialog on the future impact of AI
on society and the responsibilities of AI researchers in thiscontext.
Issues are somewhat independent of when/whether asingularity or “super-intelligence” will be reached.

Complex, autonomous decision making systems --embedded in our physical world --- are already emerging
and the impact of such systems will grow rapidly in thecoming years.
E.g., for a compelling read about the emerging role of robots
in the military, see “Wired for War” by P.W. Singer.

\We believe AAAI and AI researchers should take aleading role in dealing with the moral, ethical, andlegal issues involving AI systems (and not leave it toothers!).

This study provided a first step in formulating many of
the key issues to be addressed with initial responses.
We welcome further input from the community.

aifutures@aaai.org