**Turning Numbers Into Knowledge: The art of analysis in the Information
Age**

Jonathan G. Koomey, Ph. D.

6429 Harwood Ave, Oakland, CA 94618

Get a PO Box

510/654-9634

(phone and fax)

\"\...I\'d always believed that a life of quality, enjoyment, and wisdom
were my human birthright and would be automatically bestowed upon me as
time passed. I never suspected that I would have to learn how to live --
that there were specific disciplines and ways of seeing the world I had
to master before I could awaken to a simple, happy, uncomplicated
life.\"

\--Dan Millman[^1]\--

# Author\'s biography

Jonathan G. Koomey is currently a staff scientist at one of the
nation\'s foremost research laboratories. He holds M.S. and Ph.D.
degrees from the Energy and Resources Group at the University of
California at Berkeley, and an A.B. (Cum Laude) in History of Science
from Harvard University. He is the co-author of six books and author or
co-author of more than eighty articles and reports on energy
conservation technology, energy economics, energy policy, environmental
externalities, and global climate change. He has been studying issues of
technology and society since 1980. In 1993, his report, titled
*Cost-Effectiveness of Fuel Economy Improvements in 1992 Honda Civic
Hatchbacks* won the Fred Burgraff Award for Excellence in Transportation
Research from the National Research Council\'s Transportation Research
Board.

Dr. Koomey has been fascinated by numbers since he was a small boy. In
graduate school, he amazed and annoyed his friends with his love of
back-of-the-envelope calculations. In his spare time, he enjoys Aikido,
hiking, cycling, running, and playing classical contrabass.

# Forward

Holdren should write this (or Clifford Stoll or Peter Schwartz). Think
about audience.

# Blurb for cover

This is a book about numbers, but not about math. An award-winning young
scientist guides you through the other side of learning about analysis,
revealing tools, tricks, and heretofore unwritten rules that the best
real-world problem solvers know by heart.

# Preface and Acknowledgments

**Analysis**. The word itself exudes boredom, but the task it describes
is at the very heart of your efforts to understand the world around you.
Analysis is the process by which we take *numbers* and transform them
into *knowledge*, using our instincts and experience to fill in where
data are sparse or non-existent.

In the course of a typical day, most people are confronted by at least a
handful of claims and assertions, backed up by numbers or calculations.
This book will help you sort through those claims, and give you tricks
and techniques for evaluating them. There is no \"silver bullet\" for
separating fact from fiction, but a little training can go a long way.

After reading this book, you will be far better equipped to make
independent judgments about analysis used by others. You will know which
key questions to ask, so you need never again be at the mercy of those
who traffic in \"proof by vigorous assertion\".[^2] You will also be
much more effective at conducting and presenting your own analyses, no
matter what the topic.

The technical aspects of analysis are taught at many universities, but
the *art* of analysis is rarely discussed and even more rarely written
down. The book in your hands is the first to break with that tradition
and to present this art for a broader audience. This book won\'t teach
you how to solve particular math problems, because other books do this
just fine. It will, however, teach you the formerly unwritten rules for
becoming a first-rate analyst, and will help you apply those rules in
your daily life.

Analysis is not just for scientists. No matter what your occupation, you
rely on analysis of greater or lesser complexity each and every day.
Whether it is to estimate the duration of a work task or to budget your
money to last through the final days of a trip, you must conduct
analysis to make reasoned choices. This art is so fundamental to
citizens of a modern technological society that I believe it should be
taught at the *high school level*.

Make no mistake: it takes effort to understand the many subtleties in
conducting analysis. Becoming sufficiently fluent in these techniques is
not, however, beyond the reach of the average person. Complex higher
math is ***not*** required to understand much of what\'s happening in
the world. Most people just need simple tools in combination with their
inherent common sense to be vastly more effective in their personal and
professional lives.

#### How to use this book

This book is written for the person who has a little experience with
analysis and yearns to learn more. Total novices should read the first
few chapters and obtain the references listed there to get up to speed.
Analysis sophisticates can profitably skip from section to section or
read the book straight through.

Like Strunk and White\'s *Elements of Style*, this book is a good one to
reread periodically as your tasks and goals change. I have worked hard
so that you will discover something new each time you read it.

Each small chapter is compact and self-contained, with quotations and
cartoons that amuse and edify along the way. Where there are \"links\"
to other chapters, there will (in the final document at least) be some
kind of graphical signpost telling the reader what page to investigate
for that link.

The *Further Reading* section in each chapter does not attempt to be
comprehensive. Rather, it contains the sources I liked best as I
conducted my extensive research for this book.

I separate the book into four sections. The first is a brief
introduction to the *Information Explosion,* which explains the context
in which. To prepare you for what follows, I then present *Things to
know* about yourself and about the world, which are essential
prerequisites. In the next section, I explore *Things to do* to become a
first-rate analyst. I\'ve separated this major section into four major
subsections (Be Prepared, Assess their Analysis, Create Your Analysis,
and Show Your Stuff), each of which is composed of six to twelve
self-contained small chapters. Finally, I have a short section on
*Creating the Future*, which gives some perspective on why we use
analysis in the first place.

Enjoy!

Thank Calwell, Cooper, Erik, Amey, Kim, Tod, F, Karen, Flo, Hoffman,
Mark Morland, nathan, steve G., Laura, Jim McMahon, Alan Meier, Mark
Levine, steve wiel, Polly, Marja, Chris K., Gregor, Dad, Mom

> \"Our networks are awash in data. A little of it is information. A
> smidgen of this shows up as knowledge. Combined with ideas, some of
> that is actually useful. Mix in experience, context, compassion,
> discipline, humor, tolerance, and humility, and perhaps knowledge
> becomes wisdom.\"
>
> \--Clifford Stoll\--[^3]

# Table of contents

**Author\'s biography i**

Forward i

Blurb for cover i

Preface and Acknowledgments iii

Table of contents v

The Information Explosion 1

Part 1: Things to Know 7

About yourself 9

How to conquer fear 11

Don\'t be intimidated 15

Beginner\'s mind 17

About the world 19

The way things work 21

All numbers are not created equal 31

Scientists are people too 33

Numbers aren\'t everything 39

Part 2: Things to Do 41

Be prepared 43

Explore your ideology 45

Get organized 49

Establish a filing system 51

Build a toolbox 53

Put facts at your fingertips 57

Assess their analysis 65

Question authority 67

Don\'t believe everything you read 73

Go back to the questions 77

Reading tables and graphs 79

Distinguish facts from values 81

Create your analysis 85

Reflect 87

Inquire 91

Be a detective 95

Create consistent comparisons 97

Tell a good story 99

Dig into the numbers 103

Recycling old envelopes 109

Hear both sides 113

The value of models 115

Understanding forecasts 117

The uncertainty principle and the mass media 123

Get real! 125

Show your stuff 127

Familiarity breeds interest 129

Document, document, document 131

Let the tables and graphs do the work 139

Designing compelling graphs and figures 143

Designing good tables 145

Using numbers effectively in oral presentations 147

Some Parting Advice: Creating the future 149

Appendix A: Data Sheets 151

Appendix B: annotated bibliography 152

# The Information Explosion

![](media/image1.wmf){width="7.388888888888889in"
height="5.958333333333333in"}

In 1815, Thomas Jefferson\'s library of 6500 volumes represented an
exemplary collection of works by the world\'s great thinkers. A diligent
student who read two and a half books a day, 300 days per year, could
peruse the entire collection in less than nine years. At the time,
Jefferson\'s collection was a large one, and someone who read the
library in its entirety could reasonably assert that they had
familiarity with the most important knowledge of human civilization.

The same diligent student would have to spend a bit more time to read
all the books in the Library of Congress today. At the same rate of 750
books per year, it would take more than 26,000 years to read them
all![^4]

The two centuries since Jefferson have been extraordinary ones for human
knowledge, particularly the last half century. The pace of change has
quickened noticeably, and the rate of information production has
increased by many orders of magnitude.[^5] For the first time in human
history, information is being created at a rate far faster than humans
can assess and use it.

In this section, I examine some of the forces that have caused this
\'information explosion\'. Understanding these recent developments can
help give context to the advice contained in the rest of the book, and
will (perhaps) motivate you to read further. We all face information
overload, and the art of analysis is one important tool for controlling
it.

#### So what \'s going on?

There are several important trends giving rise to the information
explosion:

\(1\) The speed of information *transmission* has increased manyfold.
News could travel at the rate of a person on a horse or a boat two
hundred years ago, but today it travels instantly in several different
media. The rate of information transmission has increased for a given
mode (i.e. letter delivery has become faster and more reliable), but
there are also many new modes of transmitting information, some of which
are vastly faster than the older modes.

Consider how fast a book could be transported from a firm in New York to
a firm in Boston two centuries ago. On a ship that sailed between the
two cities in favorable conditions (and with a stop or two along the
way), it might take 48 hours, including time on either end for delivery
to and from the ship. For a book containing half a million
characters[^6] , that\'s a transmission rate of about three characters
per second (cps). The latest data transmission technology in widespread
use by large institutions is called a \"T3\" line, which transmits 4.5
million cps[^7], and is thus about a million and a half times faster
than the best that could be done two centuries ago. To transmit the
entire book would take less than one second on a dedicated T3 line.

An additional factor is that data transmission speeds now are largely
independent of geography. Information can now be sent instantly to any
point on the earth, limited only by the technical characteristics of the
recipient\'s receiver. Two organizations with interlinked T3 lines would
have essentially the same transmission rate from New York to Boston as
from New York to Buenos Aires or Bangladesh.

The increased speed and accessibility of information transmission
accelerates information production (the faster and more easily
scientists and businesses can communicate, the fewer lags there are in
producing technological and institutional innovations). As the number of
participants in an interconnected communications network grows,
information generation grows even more quickly. Bob Metcalfe believes
that \"the value of a network grows as the square of the number of
users\".[^8] If the network triples in size (as measured by the number
of users), the value of the network increases by a factor of 32 or 9.
Speed of information production is one of the beneficiaries of this
relationship.

The shift from analog data (radio, conventional television) to digital
media (CDs, laser discs, DVDs, modems, computer networks, high
definition television) also accelerates these trends. It is cheaper and
easier to reproduce and transmit digital data than analog signals,
especially as computer processing power and bandwidth increase.

\(2\) Information *processing* capabilities have increased by many
orders of magnitude in the post-WWII period (mainly because of increased
computer speeds). The number of transistors on a silicon chip, which is
a crude measure of computer processing power, has increased by about
four thousand (4 x 103) times[^9] since the first microprocessor was
created in 1971, and about 10 million (10 x 106) times since the
transistor entered large scale commercial use in the late 1950s[^10].
The total number of transistors shipped in a single year is now about 15
quadrillion (15 x 1015), and the cost of processing power has fallen at
a furious pace (Scientific American 1997).

These changes are the result of rapid technological innovation in the
design of microprocessors. Moore\'s law, coined by Intel founder Gordon
Moore, states that processing power roughly doubles every 18 months, and
this law has held for about the last three decades. Eventually, physical
limits will slow this progress, but each time possible limits appeared
in the past, chip designers overcame them.

This change both generates more information more quickly and helps
people understand more. Developments in microprocessors and other
electronics also help improve data gathering capabilities. Understanding
always lags information production and data gathering, however, because
understanding requires thought, while information production can be
partly or completely automated using computer technology.

\(3\) Information *gathering* has become easier, cheaper, and more
efficient, which accelerates the production of information, but also
helps people understand more about the world. Improvements in electronic
sensors and integration with microprocessor technology has allowed
automated data gathering to become commonplace. For example, it is now
easy for supermarkets to collect data on purchases of thousands of
different products using those grocery checkout scanners that so
surprised President George Bush during the 1992 presidential campaign.

\(4\) All this information generation would not be possible without
commensurate improvements in *information storage* technology. The
density of information storage per square centimeter of disk area has
increased by more than 1800 times since 1973, with a comparable rate of
improvement likely for the next five years. Costs have also declined
during this period at a pace reminiscent of microprocessor
technologies.[^11]

\(5\) In part because of the rapid improvements in data transmission and
processing speeds, the effort and incremental cost of sending out
messages has declined to almost zero (because of email and, to a lesser
extent, voice mail). Email generally doesn\'t cost the sender any more
to mail to a thousand people than to one person, and it is easy to
broadcast messages to dozens or hundreds of people with a few
keystrokes.

\(6\) The increased *specialization* of people in a post-industrial
society (reflected in part in increasing education levels) means that
they are more efficient in doing many activities, including producing
information, so the production of information is much more rapid now
than two centuries ago. The increased speed of information transmission,
gathering, and processing also accelerates the tendency towards
specialization, so these trends feed back on themselves. Specialization
also makes it harder for people to understand implications of
developments outside their increasingly narrow areas of expertise, which
accentuates feelings of information overload.[^12]

These changes are a logical outgrowth and extension of the advent of
mechanical type and printing presses five centuries ago. The trends
reinforced by the creation of printing included rapid, error-free
reproduction of texts, the specialization of knowledge and the
responsibility of scholars for its accuracy, the first steps towards our
current conception of intellectual property, and a sharp decline in the
cost of reproducing and distributing information (Burke 1978).

As these developments have accelerated the pace of our lives, a
concomitant growth in people\'s expectations has reinforced these
trends. It is now expected that people will respond much more quickly to
business or personal queries, which accentuates the feeling that events
are speeding up. Fast transmission of information also conveys data that
the sender didn\'t have before. If you fax a page to your accountant or
lawyer, you can be sure it is received instantly. Two hundred years ago,
delays in their response could be attributed to the mail system or other
intermediaries. Now there is no such excuse for delays, and you gain
information about how efficient they are in processing your request (any
delays are *their* responsibility). Thus, the power relationship between
clients and professionals has been irrevocably altered, in a way that
makes people even more frenzied.

#### Responses to information overload

As shown above, people have good reason to feel overwhelmed by the pace
of change in modern societies. There are two extreme reactions to this
information overload:

• *Keep reading and never shut down the flow of information*--Something
new may emerge that can help you solve the problem at hand. In this
case, people can be paralyzed by too much information.[^13]

• *Shut out new information because it\'s overwhelming*--People
afflicted with this reaction are handicapped by too little information.
The rely on old knowledge and ideology, and make bad decisions because
they close off the flow.

My goal for this book is to help you plot a middle course between these
two extremes, giving you tools and tricks to help you face the onslaught
of new information with equanimity, confident that you can make sense of
it and put it to good use.

#### The role of analysis

In simplest terms, analysis is systematic thinking, using logic and
facts to infer more about the world around you. Analysis helps you
evaluate the assertions of others and develop sensible assertions of
your own. It can help you separate useful information from junk, and is
one key defense against information overload. The rest of this book will
help you hone your analytical instincts and prepare you to prosper in
this information-glutted world.

#### Further reading

Burke, James. 1978. *Connections*. Boston: Little, Brown and Company. An
outstanding book that discusses the history of technology in a brilliant
and accessible way. Among other things, Burke explores the key role of
the printing press in fostering later technological and scientific
developments.

de Sola Poole, Ithiel. 1983. *Technologies of Freedom*. Cambridge, MA:
MIT Press. This book made a first attempt to quantify the accelerating
pace of information generation in modern society. It\'s a bit hard to
find, but a large public library will probably have a copy.

Scientific American. 1997. *Scientific American, Special issue: The
Solid-State Century\--The Past, Present, and Future of the Transistor*.
vol. 8, No. 1. A useful compilation of recent information on
developments in electronics and information technology.

Shenk, David. 1997. *Data Smog: Surviving the Information Glut*. New
York, NY: HarperCollins Publishers, Inc. This book documents in a
journalistic fashion the symptoms of information overload. It\'s short
on solutions, but worth a read just for the anecdotes.

> \"By and large, over our long history, people have been able to
> examine and consider information about as quickly as it could be
> created and circulated. This equipoise lasted through an astonishing
> range of communications media--the drum, smoke signal, cave painting,
> horse, town crier, carrier pigeon, newspaper, photograph, telegraph,
> telephone, radio, and film.
>
> But in the mid-twentieth century this graceful synchrony was abruptly
> knocked off track with the introduction of computers, microwave
> transmissions, television, and satellites. These hyper-production and
> hyper-distribution mechanisms surged ahead of human processing
> ability, leaving us with a permanent processing deficit, what Finnish
> sociologist Jaako Lehtonen calls an \'information discrepancy\'.
>
> In this way, in a very short span of natural history, we have vaulted
> from a state of information scarcity to one of information
> surplus--from drought to flood in the geological blink of an eye.\"
>
> \--David Shenk\--

# Part 1: Things to Know

Before beginning our explorations, there are a few preliminary items to
work through. In particular, there are things to know about yourself and
about the world that can help guide you effectively in becoming
proficient with analysis.

#### How to use this section

Read these chapters straight through, in preparation for the Things to
Do section. If you don\'t feel comfortable with some of the concepts
presented here, track down some of the further reading and delve into
it.

> \"Mr. Casals, you are 95 and the greatest cellist that ever lived,\" a
> young reporter commented. \"Why do you still practice six hours a
> day?\" Casals replied, \"Because I think I\'m making progress.\"

## About yourself

There are three main things to know about yourself to be an effective
analyst:

*How to Conquer Fear*: Everyone faces fear when confronting a new
challenge. If you find analysis and numbers unfamiliar and daunting,
read through this section and its associated references.

*Don\'t be intimidated*: The difference between success and failure
often depends on whether you are intimidated or not. By consciously
refusing to be intimidated, you can stack the odds in your favor.

*Beginner\'s mind*: Start fresh and approach any problem like a beginner
would, and you\'ll surely see things that others will miss.

Each of these short chapters offers key insights to help you get
started.

> \"Don\'t allow yourself to be intimidated by know-it-alls who thrive
> on bestowing their knowledge on insecure people. Put cotton in your
> ears and blinders next to your eyes, and trudge ahead with the
> confidence that whether or not someone else \'knows it all\' isn\'t
> really relevant; the only thing that\'s relevant is what *you* know
> and what *you* do.\"
>
> \--Robert Ringer\--

### How to conquer fear

Fear inhibits accomplishment and restricts your life. It is perhaps the
most common roadblock to people achieving great things, but few realize
that fear can be consistently beaten.

Anyone who has studied a martial art or participated in a high risk
sport knows the symptoms: Sweaty palms, nervousness, butterflies in the
stomach, and an inability to focus on ostensibly simple movements. I
have studied Aikido now for more than a decade, and have learned in that
time to face fear as a matter of course in both my Aikido training and
in other parts of my life.

I began my Aikido studies with Lorraine Dianne Sensei, who was then a
fourth degree black belt. Soon after I began, I tried to learn a
particularly challenging type of fall. For whatever reason, these falls
terrified me. When being thrown in this way I stiffened instead of
relaxing, which greatly increased my chance of injury.

I recognized this problem and asked Dianne Sensei how to overcome it.
\"Everyone has fear\", she said. \"The best way to move past it is to
practice the scary movement slowly and deliberately until it seems
natural, then gradually speed it up\".

This same method is effective in conquering other sources of fear. Most
people fear numbers as something too difficult for them to master, so
they avoid analysis, even when it affects something as important as
their own retirement planning. By starting slowly, reading widely, and
attacking relatively simple problems first, virtually anyone can build
up sufficient proficiency to be called \"numerate\". It only takes
common sense and the ability to add, subtract, multiply, and divide.
These are all learned skills, and none of them are particularly
complicated.

Abandon your preconceptions about how hard it is to understand technical
matters. It is possible to figure out just about anything if it is
important enough. The best way to become more proficient with numbers is
to choose a topic you care deeply about that involves some relatively
simple numerical aspect. Explore it by reading and finding examples of
how other people used numbers to treat the topic. Then copy their method
and apply it to the problem at hand. Imitation is more than the
sincerest form of flattery; it is often the most effective way to learn
something new. Your purpose is to solve the problem, and copying a
method of solution is a time-honored way to do so.[^14]

Many people consciously exclude numbers from their existence, or at
least they try to do so. I once met a community college professor who,
when I asked how many students her college had, could not even give me
an estimate. Her specialty was communications, and she stated
matter-of-factly \"I don\'t do numbers\". The idea of tallying the size
of the college using a number had not even crossed her mind. In fact,
she was curious as to why I wanted this information. I explained that
knowing the number was a way of generalizing so I could compare the size
of the school to others I had visited. She merely thought of the school
as a \"mid-sized, rural community college\", which had no meaning to me,
since I had never visited such a school.

When I pointed out to her that she used numbers when deciding what speed
to drive, in grading her courses, in balancing her checkbook, and for
cooking, she was forced to acknowledge that she used numbers, but only
sometimes. She *chose* to exclude certain numbers from her life, and I
suspect she is not that different from most other people. You need not
make the same choice, and your life will be enriched by choosing another
path.

#### Further reading

Clawson, Calvin C. 1992. *Conquering Math Phobia: A Painless Primer*.
New York, NY: John Wiley and Sons. This book is one of the best
introductions to basic math anywhere. It is eminently approachable and
clear.

Dewdney, A. K. 1993. *200% of Nothing: An Eye-Opening Tour through the
Twists and Turns of Math Abuse and Innumeracy*. New York, NY: John Wiley
& Sons, Inc. Another nice text giving examples of what NOT to do.

Howett, Jerry. 1984. *Number Power 1: The Real World of Adult
Math\--Addition, Subtraction, Multiplication, and Division*. Chicago,
IL: Contemporary Books, Inc. A practical introduction to very basic
math.

Kadesch, Robert R. 1970. *Math Menagerie*. New York, NY: Harper and Row:
A great book with easy projects for people to do to help them learn math
concepts like probability, mapping, and binary arithmetic.

Mitchell, Robert and Donald Prickel. 1983. *Number Power 5: The Real
World of Adult Math\--Graphs, Tables, Schedules, and Maps*. Chicago, IL:
Contemporary Books, Inc. I found this book quite helpful, interesting,
and practical.

Newell, Patricia J. 1996. *Test Yourself: Basic Mathematics*.
Lincolnwood, IL: NTC Publishing Group. Has modest amounts of
explanation, but lots of problems to work on to develop basic skills.
Covers whole numbers, fractions, decimals, ratios and proportions,
percentages, measurements and geometry, and signed numbers

Ritchhart, Ron. 1994. *Making Numbers Make Sense: A Sourcebook for
Developing Numeracy*. Menlo Park, CA: Addison-Wesley Publishing Company.
A fine book for elementary and junior-high school teachers who want to
promote numerical abilities among their students (parents could also use
it). It contains more than thirty different lessons for teachers to
adapt to their own classrooms.

Swartz, Clifford E. 1973. *Used Math: for the First Two Years of College
Science*. Englewood Cliffs, New Jersey: Prentice-Hall, Inc. This is one
of the best concise summaries of more advanced math concepts. Don\'t
tackle this one until have some math classes under your belt!

Tamarkin, Kenneth. 1983. *Number Power 6: The Real World of Adult
Math\--Word Problems*. Chicago, IL: Contemporary Books, Inc. A helpful
introduction to simple word problems (e.g., Joe can paint a room in 2
hours, and Mary can paint it in 1 hour. How long will it take if they
both work on painting the room?).

Tobias, Sheila. 1993. *Overcoming Math Anxiety* *(Revised and
Expanded)*. NY, NY: W.W. Norton and Co. This book focuses specifically
on why people are \"no good with numbers\", and proposes ways for people
to improve their skills. It is an important and extremely useful book
for those seeking to become comfortable in the world of analysis.

Tobias, Sheila. 1987. *Succeed with Math: Every Student\'s Guide to
Conquering Math Anxiety*. NY, NY: College Entrance Examination Board. A
very nice book that is just what it says it is.

> \"I am here today to prove that it can be done; that the power gained
> by understanding and appreciating the world around us can be obtained
> by anyone. That belief is what has motivated me for the last 75 years
> to get this degree.\"
>
> \--Mary Fasano, who at age 89 fulfilled a lifelong dream by graduating
> from Harvard University with an undergraduate degree in liberal arts
> (she began high school at age 71)\--

### Don\'t be intimidated

Most of us have experienced \"know-it-alls\" who confidently toss out
statistics to bolster their arguments. While there are those who speak
only when they are sure of their facts, there are many more who express
their opinions even when these opinions are ill-founded. They do so
secure in the knowledge that most people will not see through their
little game.

In 1974, Robert J. Ringer first published his classic book titled
\"Winning Through Intimidation\". In it, he described how he developed
his own philosophy for success in his chosen field of real-estate sales.
The core of this philosophy is what Ringer calls \"The Theory of
Intimidation\", which has wide application to every field of human
endeavor. In summary, it states that

> ***the rewards a person obtains are inversely proportional to the
> degree to which that person is intimidated.***

To avoid being intimidated, Ringer developed a strong posture. He first
created a professional image that would prevent others from intimidating
him (for example, he created a lavish \"calling card\" that was so
nicely produced that anybody receiving it would immediately be convinced
that Ringer was \"somebody\"). He then perfected the use of certain
legal tools that would ensure his rights to claim rewards for his
efforts. Finally, he was so good at his profession that no one could
legitimately conclude that his performance was anything but excellent.

Ringer\'s refusal to be intimidated resulted in spectacular financial
rewards relatively quickly. The important lessons to take from his book
are that

1\) many people use intimidation to accomplish their goals, even when
they are not in the right;

2\) these people often use analysis to commit these acts of
intimidation, and

3\) someone who knows the right question to ask about how such analysis
is created can often deflect acts of intimidation before any damage is
done.

Brenda Ueland, in her classic treatise on creative writing (1987),
explicitly advocates mentally thumbing your nose at \"all know-it-alls,
jeerers, critics, doubters,\" several times a day. These intimidators
murder talent and crush creativity, and you shouldn\'t let them get away
with it.

#### Further reading

Ringer, Robert J. 1974. *Winning Through Intimidation*. New York: Funk &
Wagnalls. This book is one of my very favorites. It\'s amusing,
entertaining, interesting, and timeless. It\'s also a quick read, and is
available in most big bookstores (it still sells well).

Ueland, Brenda. 1987. *If You Want to Write: A Book About Art,
Independence, and Spirit*. St. Paul, Minnesota: Graywolf Press. THE
classic book on creativity. Ueland writes about writing, but her advice
applies equally well to most forms of human creation. It\'s beautifully
written and easy to read.

![](media/image2.wmf){width="4.291666666666667in"
height="4.041666666666667in"}

> \"Why do people who know the least know it the loudest?\"
>
> \--George Carlin\--

### Beginner\'s mind

![](media/image3.wmf){width="0.5277777777777778in"
height="0.4861111111111111in"}

Most people attack a new problem by relying heavily on the tools and
skills that are most familiar to them. While this approach can work well
for problems that are similar to those previously solved, it often
fails, and fails miserably, when a new problem is particularly novel or
vexing. In these circumstances, it is best to assume nothing and treat
the problem as if you have never seen anything like it before.

In martial arts, this sense of looking freshly at something is known as
\"Beginner\'s mind\". Beginners to a new art don\'t know what is
important and what is irrelevant, so they try to pay attention to every
little detail. Experienced martial artists use their experience as a
filter to separate the essential from the irrelevant. When that filter
mistakenly screens out something essential, then even seasoned masters
can err.

The phrase \"Beginner\'s mind\" is a reminder that experience is a
two-edged sword. It serves to eliminate unneeded detail and allow a
focus on the ostensibly most important elements of a problem. It can
also lead us astray when a new problem is sufficiently outside our
experience. The art is in knowing when experience is no longer
applicable to the problem at hand.

Try to combine the curiosity and the unjudgemental observation of a
beginner with the experience and wisdom of a senior analyst. Together,
these skills can be an immensely powerful combination.

#### Further reading

Hyams, Joe. 1982. *Zen in the Martial Arts*. New York, NY: Bantam Books.
The story of Joe Hyams\' explorations into different martial arts is one
of the inspirations for the book you are now reading. It\'s broken up
into short sections, each with an important lesson he learned over the
years. Another quick read.

> \"In the course of my life I have often had to eat my words, and I
> must confess I have always found it a wholesome diet\"
>
> \--Winston Churchill\--
>
> \"A Japanese Zen master received a university professor who came to
> inquire about Zen. It was obvious to the master from the start of the
> conversation that the professor was not so much interested in learning
> about Zen as he was in impressing the master with his own opinions and
> knowledge. The master listened patiently and finally suggested they
> have some tea. The master poured his visitor\'s cup full and then kept
> on pouring. The professor watched the cup overflowing until he could
> no longer restrain himself. \'The cup is overfull, no more will go
> in.\'
>
> \'Like this cup\', the master said, \'you are full of your own
> opinions and speculations. How can I show you Zen unless you first
> empty your cup?\'.\"
>
> \--Bruce Lee\--[^15]

![](media/image4.wmf){width="6.555555555555555in"
height="5.180555555555555in"}

## About the world

In the following short chapters, I explore four key things to understand
about the world before you read the main part of the book.

*The way things work*: I describe in this chapter how humans respond to
events, exploring the connections between what we measure, what we
assume, and what we choose to do.

*All numbers are not created equal*: Numbers and calculations
characterizing the physical world are almost always more certain than
those describing human behavior . For example, many analysts wrongly
imply that modeling results based on economic data are just as solid as
science. They aren\'t, so be forewarned.

*Scientists are people too*: Progress in science can be subject to human
frailty, just like any other human endeavor. The end-result, however, is
something you can count on.

*Numbers aren\'t everything*: Not everything that matters can be
quantified, so make sure the unmeasurable doesn\'t fall through the
cracks.

You may need to read these chapters a couple of times, but they are
worth mastering. If you don\'t feel up to attacking them head on, read
the first chapter, then read the introduction and conclusions of each of
the three remaining chapters.

> Math was always my bad subject. I couldn\'t convince my teachers that
> many of my answers were meant ironically\".
>
> \--Calvin Trillin\--[^16]

### The way things work

Everything in life follows what I call \"The Cycle of Action\" shown
below. *Every* event results in choices about how to measure and
interpret that event, followed by your own response to it. Further
events follow as a consequence of your choices, leading to an endless
spiral that continues as long as you live and breathe. External events
also influence this cycle and can propel it in unexpected ways.

Norman (1990) first introduced the concept of the Cycle of Action, and
applied it to how people interact with technology.[^17] He separates
human response to events into Evaluation and Execution. Evaluation is
the process of observing external events and determining whether those
events are consonant with our goals. Execution is the process of
influencing external events so that they move more closely to our
desired outcome. Norman applies the Cycle to the choices of individuals,
but the Cycle also applies to the process by which institutions interact
with the external world.

######### The Cycle of Action

![](media/image5.wmf){width="4.6375in" height="3.332638888888889in"}

#### The Evaluation Process in Detail

The Evaluation part of this cycle can be split further into three
components, which are illustrated in the figure below:

a\) perceiving the state of the external world, which involves
observation, direct measurement, or indirect data collection;

b\) interpreting the perception, which requires that we apply our
knowledge and logic to understanding what we have perceived; and

c\) evaluating the interpretation and comparing with our goals. This
latter process is the one where value judgments and assumptions are most
forceful in exerting their influence.

######### The Evaluation Part of the Cycle of Action

![](media/image6.wmf){width="4.735416666666667in" height="4.56875in"}

##### Perceiving the state of the external world

Some aspects of the external world can be measured, but others cannot
(link to NUMBERS AREN\'T EVERYTHING). Perceiving the external world may
include both intuitive and quantifiable knowledge, and people typically
draw on both types of information when assessing their current
situation.

People measure those things that are easiest to measure. Like the man
looking for his keys under the street lamp even though he lost his key
chain down the street, we are likely to be led astray if what we\'re
measuring is only peripherally related to what we care about. If the
people charged with collecting the data are too far removed from the
actual decision and don\'t know how the data will ultimately be used,
the results can be bad or even disastrous. Sam Walton, the founder of
WalMart, did his own research on the competition as he was building his
business because *he* was the one who ultimately had to make the
decisions, and he didn\'t want strategic data collection choices made by
people in his organization who did not understand the big picture
(Kawasaki 1995).

Information technology has made it easier to collect data on events we
could only dream about tracking twenty years ago. Peter Coffee at PC
week recently advocated that businesses collect data on sales of
products by day of the week and even *time of day*.[^18] The benefits of
such tracking include improved inventory management, cash flow, and
staff management, but it is only been in the past decade that such data
collection has become possible on a large scale.

##### Interpreting the perception

Once data are collected, they virtually always require \"massaging\" to
put them into a useful form. This step can involve as little as minor
reformatting or as much as detailed calculations and analysis. The
choice of how much data manipulation to do is as much art as science,
and it relies heavily on the instincts of the researcher. For the
\"sales by time of day\" example above, the sales data might be viewed
for individual products or bundled into groups of products, to help
discern patterns and trends. The data could also be viewed by hour, by
day of the week, by month, or by year.

If the pending decisions are to be made by people other than those
collecting and massaging the data (as is often the case), the data must
then be summarized in the form of text, tables, and graphs for the
decision maker\'s use (if the information is for my own use, I usually
make the tables and graphs and skip the text). This step also involves
judgments about which data to emphasize. Poor choices here can make
irrelevant the most useful data, and can lead to erroneous conclusions.
Hourly sales data, for example, might be overwhelming if presented en
masse\--some filtering is required to make it useful. Such filtering
might include focusing on results for one particularly important product
as an example.

Bad choices in presenting the information can even be fatal, as the
unfortunate crew of the space shuttle Challenger found out. Edward Tufte
documents how the poor graphical and tabular presentation of data by
Morton Thiokol\'s project engineers failed to convince NASA to delay the
launch of the shuttle (Tufte 1997). Tufte shows that the relationship
between cold weather and damage to the O-rings in the shuttle\'s solid
rocket boosters would have been obvious to anyone presenting the
information in a clear way. In the hours leading up to the launch,
however, the information was never presented in a way that demonstrated
the clear causal link between cold weather and O-ring damage. Bad
information display can have deadly consequences.

##### Evaluating the interpretation

The evaluation of data involves drawing conclusions and developing
assertions to emphasize. These assertions can take several forms (see
below), but the choice of which assertions to make from a given set of
data is at the discretion of the analysts or the people presenting
information to the decisionmaker. Even among the set of assertions that
can be logically supported by a particular set of data, there is no one
\"right\" choice here.

#### The Execution Process in Detail

The Execution part of this cycle can also be split into three pieces, as
shown below:

a\) our intention and desire to achieve the goals, which exists because
of our human needs for closure, fulfillment, status, etc.;

b\) a list of actions (the \"action sequence\") that we plan to
undertake, which is developed based on our evaluation of the situation,
our goals, and our experience; and

c\) the actual execution of the action sequence, successful completion
of which relies on our capabilities.

######### The Execution Part of the Cycle of Action

![](media/image7.wmf){width="4.3180555555555555in"
height="4.596527777777778in"}

##### An intention to act

Execution cannot take place without a intention to act. Before deciding
what to do, we first need to decide to do *something*. From this
intention flows the next step (a decision about *which* action to take).
If you decide not to act, the Cycle stops here.

##### Developing the action sequence

Accurate data collection and analysis in the Evaluation Stage of the
Cycle make it more likely that the decisions made are the best possible,
given current realities. In virtually all cases, no one can have
complete knowledge of all relevant factors, so the decisionmaker relies
on instinct, experience, ideology, and whatever relevant data have been
brought to bear for her use. Any decision is usually one of dozens that
she will make on a given day, and there are few such decisions that do
not also face time constraints. Poor judgment can result as much from
time constraints that are too tight as from data that are compiled
inaccurately.

Because all information is uncertain in one way or another, it is
important to develop strategies to cope with that uncertainty when
formulating the action sequence (Morgan and Henrion 1992). One of the
most effective coping strategies is called \"Scenario Building\", which
is explored further in the chapter \"Tell a good story\" (LINK).

Experience and education both affect how information is used in
decisionmaking: \"a manager who has been through years of price wars
will interpret an opponent\'s price reduction differently from a Harvard
MBA, who\'s never been closer to a price war than a lecture hall, or a
Wharton MBA, who has to formulate an econometric model to buy a pack of
gum\" (Kawasaki 1995). When evaluating how information was used in a
particular instance, always explore the interests and training of the
people involved. Such an exploration will almost certainly be fruitful.

Finally, all decisions about action involve value choices, whether
implicit or explicit. No matter how technical the topic, a choice made
by any human being affects human life and thus is inescapably linked
with human values (LINK TO FACTS AND VALUES SECTION). To pretend
otherwise is both foolhardy and shortsighted.

##### Executing the action sequence

Once we decide on what to do, we must follow through with action. Good
data documentation practice (LINK), accurate analysis, and clear
thinking can help ensure that the outcome of execution is what we
intend. The perception of others can be key to success, and a
decisionmaker backed up by credible analysis can influence that
perception to her advantage.

#### Piercing the Bubble

The nice rational process I\'ve sketched out above can be subverted by
the complexities of office politics. The busiest decisionmakers (CEOs,
Presidents, Cabinet Secretaries) are often protected from new ideas by
their legions of \"handlers\", from their Chief of Staff to their
secretary. Robert Reich discovered this truth when he became Secretary
of Labor in 1993:

\"My cavernous office is becoming one of those hermetically-sealed,
germ-free bubbles they place around children born with immune
deficiencies. Whatever gets through to me is carefully sanitized.
Telephone calls are prescreened, letters are filtered, memos are
reviewed. Those that don\'t get through are diverted elsewhere. (Reich
1997, p. 73)\"

The Cycle of Action can thus become the Cycle of Inaction when
information is diverted from its intended audience. Never forget that
within institutions, the most important decisionmakers virtually always
face some variant of Reich\'s \"bubble\". Reich himself tried to
puncture the bubble now and then, to the consternation of his staff.

#### The bottom line

I use the Cycle of Action as a reminder that making any decision or
taking any action relies on choices about which data to ignore and which
to address explicitly. It also illustrates that unknown and unknowable
data, as well as assumptions and time constraints, affect *every*
choice. No decision is immune from these facts, and you would do well to
keep them in mind.

Time constraints can influence all the steps in the Cycle of Action.
Rarely do decisionmakers have enough time to evaluate all the options,
so they rely on judgment and ideology to help them assess a situation.
Execution of plans also occurs in a time-constrained environment,
because other people and institutions are making changes that can negate
the beneficial effects of your own changes, so there\'s pressure to act
quickly.

Use the summary figure below whenever your choices make events go awry.
It applies to every event in your life, from decisions about what
clothing to wear to those about which college to attend. By thinking
systematically about how you use information to make decisions, your
decisionmaking will improve in the future.

######### The Cycle of Action in Detail View

![](media/image8.wmf){width="4.777083333333334in"
height="5.304861111111111in"}

#### Assertions in everyday life

One of the symptoms of the information glut is that unscrupulous or
unknowing purveyors of misinformation can make ridiculous assertions
with impunity. Such assertions are part of the external world to which
you are responding, and they can affect your perception of that world.
These purveyors of bad information (I call them the \"untruth squad\")
are protected by the sheer volume of assertions floating
around\--misinformation can be created much faster than the truth can
stomp it out. I separate the types of assertions commonly encountered in
everyday life into four categories: normative, rhetorical, explanatory,
and predictive. I illustrate each of these in **Table 1**.

*Normative* assertions promote a certain action, such as purchasing a
product or voting for a candidate. The word \"should\" appears
prominently in any normative assertion, followed by the action being
promoted. Such assertions pop up in advertisements most often, but also
are found on newspaper editorial pages and in conversations between
people.

*Rhetorical* assertions promote a point of view or a way of thinking.
They are distinct from normative assertions because they are focused on
altering thoughts, not on changing behavior.[^19] The word \"should\"
will also appear in rhetorical assertions, followed by the point of view
being promoted. Rhetorical assertions are commonly found on newspaper
editorial pages and in conversations between people.

*Explanatory* assertions describe why events transpired they way they
did, while *Predictive* assertions attempt to predict how the future
will unfold. Both are commonly found in newspapers, magazines and books
of all sorts.

######## Table 1: Types of Assertions in Common Use

+-----------------------------------+-----------------------------------+
| *Type of assertion*               | *Example*                         |
+-----------------------------------+-----------------------------------+
| Normative                         | You should buy this widget        |
|                                   | because\...                       |
|                                   |                                   |
|                                   | You should vote for me because..  |
+-----------------------------------+-----------------------------------+
| Rhetorical                        | You should adopt this opinion     |
|                                   | because\...                       |
+-----------------------------------+-----------------------------------+
| Explanatory                       | This event occurred for this      |
|                                   | reason                            |
+-----------------------------------+-----------------------------------+
| Predictive                        | Events will unfold in this way    |
|                                   | because\...                       |
+-----------------------------------+-----------------------------------+

It is helpful when evaluating the validity of a particular assertion to
determine which category it occupies. As an exercise, purchase today\'s
edition of your local newspaper and identify the assertions you find on
the front page, the business page, and the editorial page. Put an \"N\"
next to a normative assertion, an \"R\" next to a rhetorical one, and so
on. You will be edified by the number and type of the assertions you
discover.

#### Conclusions

The collection and use of data are human endeavors, and as such are
subject to all the vagaries of individual and institutional behavior. At
any point in the Cycle of Action, human frailties can change the course
of events. This realization should not inspire fear or resignation.
Rather, you should refer back to the Cycle of Action whenever you want
to improve how you use information or to explain how a decisionmaking
process broke down in a particular instance. Identify which parts of
this cycle were the result of your own choices, those of others, or of
external events. Such a systematic assessment can help you use
information more skillfully and make better decisions in the future.

#### Further reading

Kawasaki, Guy. 1995. *How to Drive Your Competition Crazy: Creating
Disruption for Fun and Profit*. New York, NY: Hyperion. Kawasaki harkens
back to his time as chief evangelist for the upstart Apple Computer in
the late 1970s and early 1980s, and gives some lessons for marketing
products in the face of competition from huge and well-funded rivals.
The book is loaded with great anecdotes.

Morgan, M. Granger and Max Henrion. 1992. *Uncertainty: A Guide to
Dealing with Uncertainty in Quantitative Risk and Policy Analysis*. New
York, NY: Cambridge University Press. A somewhat technical book, but for
those interested in assessing uncertainty in a quantitative way, it has
no equal.

Norman, Donald A. 1990. *The Design of Everyday Things*. New York, NY:
Doubleday/Currency. This fascinating book explores how humans interact
with technology, and gives guidance to designers on how to make their
products a joy to use.

Reich, Robert. 1997. *Locked in the Cabinet*. New York, NY: Alfred A.
Knopf. Regardless of your political leanings, you\'ll enjoy Reich\'s
tales of political intrigue, error, and innuendo.

Schwartz, Peter. 1996. *The Art of the Long View: Planning for the
Future in an Uncertain World*. New York, NY: Doubleday. Lively and
realistic scenarios are crucial tools for decisionmakers, and Schwartz
describes how to create them in this classic book.

Tufte, Edward R. 1997. *Visual Explanations: Images and Quantities,
Evidence and Narrative*. Cheshire, CT: Graphics Press. Chapter 2, titled
\"Visual and statistical thinking: displays of evidence for making
decisions\", contains the space shuttle Challenger example.

######### One approach\...

![](media/image9.wmf){width="6.555555555555555in"
height="2.138888888888889in"}Need to get permission to reproduce this
cartoon and pay any royalty due.

> \"It is not the critic who counts, not the man who points out how the
> strong man stumbled, or where the doer of deeds could have done them
> better. The credit belongs to the man who is actually in the arena;
> whose face is marred by dust and sweat and blood; who strives
> valiantly; who errs and comes short again and again; who knows the
> great enthusiasms, the great devotions, and spends himself in a worthy
> cause; who, at the best, knows in the end the triumph of high
> achievement; and who, at worst, if he fails, at least fails while
> daring greatly, so that his place shall never be with those cold and
> timid souls who know neither victory nor defeat.\"
>
> \--Theodore Roosevelt \--

### All numbers are not created equal

A common fallacy to which even sophisticated analysts sometimes fall
prey is to look for immutable laws of human behavior, much like the
physicist discovers such laws through experiment. Such generalizations
about human and economic systems fail because these systems are
adaptable in ways that physical systems are not.

Physicists conduct replicable experiments to uncover fixed physical
laws. A scientist measuring the speed of light, for example, would find
the velocity the same in the U.S. or Tahiti. Another scientist, if she
conducted a similarly accurate experiment in Russia, would produce the
same result.

On the other hand, relationships between cause and effect for
individuals and human institutions are dependent on institutional,
social, and economic context. Further, these relationships change over
time. A market researcher attempting to predict consumer acceptance for
a new toothpaste would find that a market test in San Francisco would
yield quite different results than in Tahiti, even though the speed of
light remains the same in both places. In addition, if the same
experiment were conducted in the 1950s, the results would presumably
vary wildly from those of the current day.

In the mid-1970s and early 1980s, the conventional wisdom held that
modern societies could not reduce energy use without harming their
economies. For example, Gordon Corey, vice-chairman of Chicago\'s
Commonwealth Edison, stated in 1981 that \"there is an unbreakable tie
between economic prosperity and energy use\". Similarly, the Chase
Manhattan Bank stated, in their 1976 *Energy Report* that

there is no documented evidence that indicates the long-lasting,
consistent relationship between energy use and GDP will change in the
future. There is no sound, proven basis for believing a billion dollars
of GDP can be generated with less energy in the future.[^20]

Believers in an unbreakable link between energy use and GDP found their
belief shattered by events. From 1973 to 1986, U.S. primary energy
consumption stayed flat, but GDP rose 35% in real (inflation-adjusted)
terms. These believers had forgotten that human institutions can adapt
to new realities, and historically derived relationships (like the
apparent link between energy use and GDP that held up for more than two
decades in the post World War II period) can become invalid when events
(like the 1973 oil embargo) overtake them.

Most economic computer models embody historical experience through
relationships that are derived statistically, and then use those
relationships to forecast the future. These models are often used to
assess the potential effects of proposed changes in government policy or
business strategy. The models embody history, but cannot give an
accurate picture of a world in which the fundamental relationships upon
which they are based are in flux. If the historically-derived
relationships embedded in such a model are the very ones that would be
affected by choices or events, then those relationships must be modified
in the analysis, or else the results are suspect.

For example, many economists trying to assess the costs of reducing
carbon emissions assume that the only way to reduce these emissions is
to impose a large carbon tax, which would raise the price of coal, oil
and natural gas substantially. They further compound this error by using
models that embody past historical relationships to do long-term
forecasts.[^21]

Achieving a low-carbon world presupposes massive behavioral and
institutional changes that render past relationships between energy use
and economic activity largely irrelevant (just like after 1973). It also
presupposes alterations in government policies. It is simply laughable
to use computer models based on historical relationships for one hundred
year forecasts in the face of such massive changes. Instead, scenario
analysis (See section \"Tell a good story\") is the appropriate tool to
use to explore the key relationships and how the world might evolve if
those relationships change (Schwartz 1996).

If you are confronted by results of computer forecasts, always ask
whether the analysts changed the key historical relationships to reflect
the possibility that those relationships will be affected by
institutional choices or other developments. If not, then you\'ve
identified a key failing of the analysis, and the results are suspect.
LINK TO UNDERSTANDING FORECASTS.

I have noticed that physical scientists who have changed fields,
computer modelers, and economists are especially susceptible to this
kind of mistake. I am not certain why practitioners of these disciplines
fall prey to this pitfall, but I have noticed it often. It may be what
my social science-oriented friends call \"physics envy\", or it may be
that most analyses are conducted in a mechanical way without significant
reflection. In any case, once forewarned you need not let them get away
with it.

#### Further reading

Schwartz, Peter. 1996. *The Art of the Long View: Planning for the
Future in an Uncertain World*. New York, NY: Doubleday. The classic book
on scenario analysis. A \"must read\" for anyone thinking systematically
about the future.

> \"It is fashionable today to assume that any figures about the future
> are better than none. To produce figures about the unknown, the
> current method is to make a guess about something or other - called an
> \"assumption\" - and to derive an estimate from it by subtle
> calculation. The estimate is then presented as the result of
> scientific reasoning, something far superior to mere guesswork. This
> is a pernicious practice which can only lead to the most colossal
> planning errors, because it offers a bogus answer where, in fact, an
> entrepreneurial judgment is required.\"
>
> \--E.F. Schumacher \--[^22]

### Scientists are people too

Science is a human process that proceeds in fits and starts. The end
result is generally something you can count on, but the intermediate
steps to that result do not advance in a linear progression, as science
textbooks might lead you to believe (Kuhn 1970).

Most people (especially nonscientists) don\'t recognize the large role
that intuition and instinct play in scientific discovery, particularly
in the process of developing hypotheses (Holton 1973, Holton 1986,
Pirsig 1981). A scientist\'s desires for recognition and status, her
enthusiasm for solving a problem, and her attitudes towards authority
are all motivations rooted in human needs and emotions.

These non-rational inclinations all affect the progress of science, but
the end result of the scientific process is one that is rational to the
core. Science has an internal coherence and predictive power that is
unique among human endeavors. Holton (1996, p.111-112) identifies the
\"apparent contradiction between the seemingly illogical nature of
actual personal discovery and the logical nature of well-developed
scientific concepts\".

This stark contrast between the messy business of scientific discovery
and the end result, which we generally regard as an accurate description
of how the physical world operates, comes about in part because of a
process known as \"peer review\". It is important for nonscientists to
understand how this process works, because accurate peer review, more
than any other part of the scientific process, determines whether a
particular set of research results is credible.

The formal process of peer review is usually conducted for articles
submitted to scholarly (\"peer reviewed\") journals or for research
proposals submitted to funding agencies such as the National Science
Foundation. In journal reviews, the author submits the article to the
journal editor, then the editor distributes the article to one to four
reviewers. The reviews are sometimes \"blind\" so that the reviewers
don\'t know the names of the authors and their affiliations (it is
however often possible for knowledgeable reviewers to infer who the
authors are from the paper\'s citations and the approach the authors
take in presenting the results). Both the number of reviewers and
whether the reviews are blind are at the discretion of the editor. For
scientific project proposals, reviews are more often blind and are
usually more formalized than are journal reviews.

The purpose of the review is to judge whether the work is based on
principles and judgments that represent the current scientific
consensus. When such a consensus exists in a particular field, it
reflects the paradigm accepted by scientists in that field (Kuhn 1970)
and gives practitioners a standard set of tools with which to evaluate
new ideas. When a paradigm is challenged by anomalous observations and
experiments, a crisis ensues. The crisis continues until a new paradigm
emerges that encompasses the anomalous results. In periods of crisis, or
in non-scientific fields, peer review is generally less reliable than in
\"hard\" sciences like physics, because the foundations upon which the
reviewers judge research results in that field are still being
questioned in fundamental ways.

######### The Course of Science

![](media/image10.wmf){width="6.166666666666667in"
height="6.458333333333333in"}Source: Unknown

Peer review is not a guarantee of accuracy. In fact, papers containing
major scientific blunders have been peer reviewed.[^23] However, the
formal peer review process does make it more likely that a paper avoids
major flaws. Be skeptical of \"research\" results that are not peer
reviewed. There are some \"scientists\" who announce their results to
the media but do not publish in peer reviewed journals. Usually, such
announcements are funded by particular organizations with an ax to
grind, and have little to do with science.

There are two kinds of proof accepted in scientific research. The first,
based on deductive logic, relies on assumptions and general principles
to derive implications and predictions of specific events.[^24] If I
leave my bicycle outside and it rains, I deduce from my experience and
the laws of chemistry that parts of the bicycle will rust. Deductive
inferences rely on a set of initial assumptions (\"axioms\") which are
analyzed using rules of logic. If the initial assumptions are correct,
the rules of logic are followed, and no logical contradictions arise in
the analysis, then the conclusions must be correct.

Inductive logic, on which many inferences in science and other forms of
human endeavors are based, relies on compilations of specific instances
and infers general laws from the specifics (Hughes 1997). If I leave my
bicycle unlocked outside for a hundred days and it is never stolen. I
could use inductive logic to infer that it will not be stolen tomorrow,
but I cannot be sure. I can only say that it is *unlikely* to be stolen,
based on past experience. Inductive proof is less certain than deductive
proof (in rare instances it can be *as* certain, but never more).

The *Encyclopedia of Philosophy*[^25] defines inductive argument to
\"cover all cases of nondemonstrative argument, in which the truth of
the premises, while not entailing the truth of the conclusion, purports
to be a good reason for belief in it\". In plain English, an inductive
proof shows that the conclusion is highly probable, but not demonstrably
true in the same way that a correct deductive proof is. Induction is the
link between mathematics, deductive logic, and our experience of the
physical world.

Deduction by itself is impotent because it depends only on assumptions
and its own internal logic. Without induction to enrich the set of
assumptions upon which deductions can be based, the world of deduction
would be a spare one, indeed. When inductive logic was first formalized,
the goal of its devotees was to show that the results that followed from
it were demonstrably *true*, in the same way conclusions based on
deductive logic were true. According to the *Encyclopedia of
Philosophy,* \"not until the end of the 19th Century did a more modest
conception of inductive argument and scientific method, directed toward
acquiring probability rather than certainty, begin to prevail\".

The limitations of induction are offset in part by the peer review
process. Outside reviewers uncover errors of fact, logic, and omission,
and dutifully report them to the authors and the journal. Obvious errors
are weeded out quickly, while more fundamental flaws may take years for
the process to uncover. Eventually, though, there are sufficient numbers
of devoted scientists that even these errors will be discovered as those
scientists attempt to build on previous work. Serious flaws will result
in logical inconsistencies (discovered through deduction) that will
inevitably surface and be corrected.

#### The scientific outlook

Science is predicated on two uncommon attitudes:

1\) being willing to accept what you find; and

2\) being willing to discover that you are wrong.

Human nature being what it is, these two attitudes are rare. Next time
you find yourself resisting a new idea, take a deep breath and try to
see the other point of view. Ask yourself why this idea might make you
uncomfortable. If you can step back from a situation in this way, you
will have achieved what I like to call the true \"scientific outlook\".

#### Conclusions

Remember always that science is a human endeavor. Learn about the
process so you can better appreciate how to interpret scientific
findings. Finally, always give more weight to peer-reviewed research
than to results announced solely in the media. Your own understanding of
the world will benefit from this simple advice.

#### Further reading

Edwards, Paul, ed. 1972. *The Encyclopedia of Philosophy*. New York, NY:
Macmillan Publishing Co., Inc. & the Free Press. Volumes 1 through 8.
This encyclopedia is the standard reference source for anyone
investigating philosophical concepts. It contains articles by leading
scholars on everything from logic to epistemology, energy to truth,
liberty to idealism, ethics to virtue, and almost anything else you can
think of.

Holton, Gerald. 1973. *Thematic Origins of Scientific Thought: Kepler to
Einstein*. Cambridge, MA: Harvard University Press. This book contains
Holton\'s investigations of how science is created in the real world.
It\'s a clearly written and fascinating excursion into the history of
science.

Holton, Gerald. 1996. *Einstein, History, and Other Passions: The
Rebellion Against Science at the End of the Twentieth Century*. New
York, NY: Addison-Wesley Publishing Company. Holton turns in this book
to the role of science in modern society, drawing liberally on
historical references to inform his views.

Hughes, William. 1997. *Critical Thinking: An Introduction to the Basic
Skills*. 2d Peterborough, Ontario: Broadview Press. This clearly written
and accessible book is a marvelous introduction to logical argument.
Hughes uses myriad examples to illustrate the logical fallacies that
bedevil most arguments. He also clearly describes the difference between
inductive and deductive logic.

Kuhn, Thomas S. 1970. **International Encyclopedia of Unified Science**.
Vol. II, No. 2. *The Structure of Scientific Revolutions*. 2d Chicago:
University of Chicago Press. The classic book that redefined
contemporary views of how science progresses. I believe this book is one
of the most important of the twentieth century.

Pirsig, Robert M. 1981. *Zen and the Art of Motorcycle Maintenance*. NY:
Bantam Books. This book is a popular excursion into the relationship
between science and art. While not always philosophically rigorous, the
book is still entertaining, thoughtful, and provocative.

Sokal, Alan D. 1997. \"A Plea for Reason, Evidence, and Logic.\" *New
Politics*. vol. 6, no. 2. p. 126. This short transcript of a talk by
Sokal is a brilliantly clear and concise exposition of the scientific
world view, in the context of his now famous parody of
\"post-modernist\" relativism. Sokal created a controversy in 1996 by
submitting an article that he knew was total nonsense to a journal
called *Social Text*, and the journal published it. Sokal suspected
correctly that by liberally sprinkling the article with phrases that
flattered the editors\' predispositions, he would make publication
likely. He then exposed the hoax.

> \"Man tries to make for himself in the fashion that suits him best a
> simplified and intelligible picture of the world. He then tries to
> some extent to substitute this cosmos of his for the world of
> experience, and thus to overcome it\....He makes this cosmos and its
> construction the pivot of his emotional life in order to find in this
> way the peace and serenity which he cannot find in the narrow
> whirlpool of personal experience\....The supreme task \... is to
> arrive at those universal elementary laws from which the cosmos can be
> build up by pure deduction. There is no logical path to these laws;
> only intuition, resting on sympathetic understanding of experience,
> can reach them.\"
>
> \--Albert Einstein\--[^26]

### Numbers aren\'t everything

Don\'t ever confuse things that are merely countable with those things
that really count.[^27] Some analysts systematically ignore things that
can\'t be turned into a number, and assume, because they can\'t be
measured, that they are unimportant. These analysts also commonly assume
the converse, that things that can be measured must be important. Both
mistakes are common, especially among the more quantitative among us,
but these are errors to which you need not succumb.

There are four categories of things affecting decisions in the world:

\(1\) some things can be measured directly and with confidence before
decisions have to be made. The amount of money in your bank account at
the time of your last statement falls into this category.

\(2\) some (perhaps most) things are not directly measurable, but can be
estimated accurately in time to influence decisions. Estimation is less
precise than measurement, but it is still accurate enough in most cases.
You cannot know precisely how much money you will have in your bank
account in a month, but you can estimate it in a crude way by making
assumptions about your expenditures, based on past spending habits.

\(3\) some things cannot be measured or estimated accurately within the
time frame in which decisions have to be made. Given time, some
reasonable estimate might be possible for such factors, but time is
often limited. For example, time constraints may prevent you from doing
a complete analysis of your financial resources before rushing to put a
bid down on a house.

\(4\) some things cannot in principle be measured or estimated under any
circumstances. Sometimes there are physical reasons for this inability
(e.g., the Heisenberg uncertainty principle (LINK) limits the accuracy
of measurements of the velocity and position of subatomic particles) and
other times it is because things that really matter to people\'s lives
(like loyalty, love, honesty, and integrity) are simply not quantifiable
in any meaningful way. It is not possible to estimate the value of a
warm summer\'s day or a species saved from extinction, except in terms
that have nothing at all to do with numbers.

Events in categories (3) or (4) fall in the Cycle of Action (LINK) under
\"unknown or unknowable aspects of events\". To analyze such situations,
we must make assumptions and value judgments, using our ideology and
instinct as guides to fill in the facts that we can\'t obtain with
measurement or estimation.

It is items in category (4) that are commonly ignored, but often are of
the most importance to human life. Since many things simply can\'t be
quantified, we can only insist that those decision factors that defy
quantification be listed and described in as complete a form as is
possible given current knowledge. Then at least decisionmakers are aware
of the other factors that may relate to the decision at hand.
Decisionmakers can then weigh these factors based on their values,
instincts, experience, ideology, and knowledge.

#### Further reading

Bella, David A. 1979. \"Technological Constraints on Technological
Optimism.\" *Technological Forecasting and Social Change*. vol. 14, p.
15. An article that uses the methods of science and mathematics to show
why social and environmental problems generated by technology may be
difficult to solve.

Jonas, Hans. 1973. \"Technology and Responsibility: Reflections on the
New Task of Ethics.\" *Social Research*. vol. 40, no. 1. p. 31. This
classic article is one of the most important in recent times. It
addresses ethical issues that affect modern technological societies that
for the first time have the power to effect changes in the natural
environment in a global way. As technology changes what it means to be
human, Jonas argues that contemporary ethics must keep pace.

######### What\'s the value of the earth?

![](media/image11.wmf){width="6.777083333333334in"
height="2.4298611111111112in"}Need to get permission to reproduce this
cartoon and pay any royalty due.

> SUCCESS
>
> \"To laugh often and much; to win the respect of intelligent people
> and the affection of children; to earn the appreciation of honest
> critics and endure the betrayal of false friends; to appreciate
> beauty; to find the best in others; to leave the world a bit better,
> whether by a healthy child, a garden patch or a redeemed social
> condition; to know even one life has breathed easier because you have
> lived. This is to have succeeded.\"
>
> ---Ralph Waldo Emerson\--

# Part 2: Things to Do

Now comes the fun part. In the following thirty or so short chapters, I
describe some important lessons I\'ve learned over the years. I separate
these chapters into four groups:

*Be prepared*: A key determinant of your effectiveness is the quality of
your preparation. Whether you\'re building a house or chairing a
meeting, preparation for the analysis tasks at hand can turn a potential
disaster into a triumph.

*Assess their analysis*: When faced with the assertions of others, it\'s
good to know the right questions to ask. These chapters summarize
hard-won knowledge about deciphering other people\'s analyses.

*Create your analysis*: Each person develops their own techniques for
creating cogent analyses, and I summarize those I\'ve learned here. The
importance of organization, clear thinking, careful definitions,
systematic exposition, scrupulous documentation, and consistent
comparisons cannot be overestimated, and you\'ll learn about each of
these in this section.

*Show your stuff*: Once you\'ve done good work, you\'ll want to present
it effectively to readers or listeners. The short chapters in this
section give insights into making your results \"grab\" your audience,
designing good tables and figures, and using those tables and figures to
convey your key points.

#### How to use this section

As above, each chapter is self-contained, with links to related
chapters. Feel free to skip from chapter to chapter. They need not be
read in the order they appear.

> \"Like all other arts, the science of deduction and analysis is one
> which can only be acquired by long and patient study.\"
>
> \--Sherlock Holmes\--[^28]

## Be prepared

Few people are so brilliant that they need never prepare, and even they
would do far better with a little work beforehand. The six short
chapters in this section describe important preparations that any good
analyst should consider.

*Explore your ideology*: Ideology provides a simplified model of the
world that reflects our values and experiences, and prevents paralysis
in the face of the many choices we make every day. Make sure you know
your own and that of others.

*Get organized* Working in a chaotic office is like running a marathon
with your feet tied together. Get your workspace in shape and keep it
that way.

*Establish a filing system*: Few mistakes are more maddening than
knowing you have seen an article that\'s relevant to the task at hand,
but not being able to find it. By creating a good filing system, you can
prevent this annoyance from ever happening again.

*Build a toolbox*: My analytical toolbox is the set of tricks and
techniques that I can pull out to solve a particular problem. In this
chapter, I describe some key tools to consider for your own.

*Put facts at your fingertips*: Every analysis requires data. Unless
you\'ve memorized the encyclopedia, you\'ll still want to keep some key
reference sources within easy reach. This chapter describes the ones I
find most useful.

*Value your time*: Remember that your time is your life, and that if
someone is wasting your time, they are stealing your life. Keep this in
mind always.

Prepare well, and you\'ll never regret it.

> \"If I had eight hours to cut down a tree, I\'d spend six hours
> sharpening my axe.\"
>
> \--Abraham Lincoln\--

### Explore your ideology

Anyone who states that ideology has no effect on their decisions is
either lying or deluded. Everyone has an ideology, whether explicit or
not. Ideology provides a simplified model of the world that reflects our
values and experiences. It helps people make decisions in the face of
imperfect knowledge.

Flew\'s Dictionary of Philosophy defines ideology as \"any system of
ideas and norms directing political and social action\", so it links
ideas and choices. Ideologies are often explicitly political
(libertarian, conservative, liberal), but may also relate to preferences
about material things.

Even technical subjects can be affected by ideology. For example,
certain hackers with a libertarian ideology actually protected a
recently released version of the Macintosh operating system from being
illegally copied (even booby trapping or removing copies from public
news groups) because they wanted to strike a blow against Microsoft, the
dominant company in the computer industry.[^29]

Computer users each see themselves in a certain way, which can reflect
their ideologies. Macintosh users, for example, usually see themselves
as going against the conventional wisdom\--resisting the crowd mentality
by choosing a computer they regard as technically superior. Windows
users, on the other hand, see themselves as realists who are adopting
the standard because they perceive its complete dominance to be
inevitable, because they believe it is cheaper,[^30] and because they
prefer to use the system that most other computer users also use.

Ideology is one connection between the world of ideas and the world of
choices. It impinges upon the cycle of action (LINK) in the assessment
of unknown or unknowable aspects of events and the interpretation of
data for those events that can be measured. Ideology guides the
assumptions and value choices you make in the course of any analysis,
and it can be particularly helpful (or particularly misleading) in
situations where little is known.

There\'s no guarantee that your ideology, which has been developed in an
iterative process between your predispositions and your experience, will
work well in the future, even if it\'s worked well in the past. That\'s
why ideologies that maintain their relevance over time must allow for
adaptation and evolution.

My own ideology, based on reason, pragmatism, environmental
consciousness, and social responsibility, has not changed much over the
past fifteen years. One core part of this ideology is, in the physicist
Alan Sokal\'s words, \"a respect for evidence and logic, and for the
incessant confrontation of theories with the real world\". My ideology
is constantly challenged by left-wing friends who find my tolerance for
market mechanisms distasteful, but thus far I have not been convinced to
modify my ideology in a significant way. Over the years, I have changed
positions on particular issues, but my core belief in logic and evidence
has never wavered.

The ideology held by an individual *can* change radically over the
course of a lifetime. Eldridge Cleaver, one of the original Black
Panthers in the 1960s, became a Republican as he grew older. Such shifts
can be triggered by a watershed event, or can happen gradually as a
person\'s experience affects their outlook.

It is essential to think through your ideology so it will be consistent.
You can check its *external* consistency by examining how its
assumptions and conclusions stack up against facts you know for sure.
Sometimes there is no way to check external consistency, and we say in
this case that the assumptions behind the ideology are not falsifiable,
testable, or provable. We can always test *internal* consistency,
however. If there are internal inconsistencies, we know there\'s
something wrong.

To help think about your ideology, ask yourself these questions[^31]:

• Where do you get your information?

• Whom do you trust?

• With whose opinions do you normally agree? Do you agree with all their
opinions?

• To whom do you give the benefit of the doubt?

• Are there events that outrage you? Why do they make you mad?

• What motivates you to work hard?

• In what situations would one part of your ideology lead you to take
actions that would be inconsistent with another part of your ideology?

Ideology is the filter through which we view the world. By exploring
your own and that of others, you will be better prepared to use numbers
responsibly and effectively in your analyses, writing, and
presentations.

> \"Only the supremely wise and the ignorant do not alter\"
>
> \--Confucius\--

######### Apple appeals to ideology

> ![](media/image12.wmf){width="3.9715277777777778in"
> height="8.569444444444445in"}

### Get organized

Many people garner high scores on standardized tests but are unable to
apply themselves in a way that gives results. I distinguish here between
*raw intelligence*, which is the ability to think logically, and *useful
intelligence*, which is raw intelligence as enhanced by hard work,
common sense, instinct, experience, discipline, and organization. A
person of moderate raw intelligence who possesses high useful
intelligence will virtually always be more productive than a brilliant
person who lacks the personality traits needed to make that brilliance
useful in the world.

Organization is the single most important factor in creating useful
intelligence, and it is a *learned* skill. By consciously becoming more
organized, you can substantially increase your effectiveness in
virtually any endeavor.

There are a few key elements to effective personal organization:

• *Control information flows*: Decide which information streams are
relevant and which are detrimental to your professional effectiveness.
Cancel magazine and newspaper subscriptions except those you use
regularly. Limit your use of the World Wide Web to get news
updates\--with few exceptions, checking more than once a day is a waste
of time and effort. Set aside specific times to answer mail and email.

• *Buy a paper or electronic organizer*: Track to-do tasks, addresses,
phone numbers, and notes in an orderly way. Get rid of those myriad
little slips of paper that confuse and confound.

• *Clean your office*: Throw out extraneous stuff and store rarely used
books. Make frequently-used materials easily accessible. Keep only
current work on your desk and put other materials in drawers.

• *Develop a filing system (LINK)*: Create a filing system that helps
you retrieve stored information in a timely fashion. If you can\'t find
an article or electronic file, it doesn\'t matter if you have it
somewhere or not\--it\'s worse than useless, because you waste time
looking for it. Be selective about the items you file.

• *Take time to reflect (LINK)*: Make sure you give yourself at least a
half hour per day of solid thinking time, with no interruptions. Unplug
the phone and turn off the radio.

• *Prioritize tasks and focus on the essential (LINK to VALUE YOUR
TIME)*: Too often, your day is eaten up by urgent tasks that are not
important. For at least part of the day, focus on the important tasks
that are not urgent.

• *Document, document, document (LINK)*: Documentation creates a trail
for you and others to follow after the fact. It is essential to any
intellectual work, but is oft neglected. By making it a priority, you
ensure that your intellectual contribution will remain important and
useful for years to come.

You have control over each of these steps . Do them, and you\'ll make
the most of every opportunity. Avoid them, and you\'ll never be as
effective as you could be.

#### 

#### Further reading 

Covey, Stephen R. 1990. *The 7 Habits of Highly Effective People*. NY,
NY: Simon & Schuster. Covey got me to think systematically about how I
organized my time and my work.

Culp, Stephanie. 1990. *Conquering the Paper Pile-up: How to Sort,
Organize, File, and Story Every Piece of Paper in Your Home and Office*.
Cincinnati, Ohio: Writer\'s Digest Books. Practical advice for getting
organized.

Lively, Lynn. 1996. **The WorkSmart Series**. *Managing Information
Overload*. New York, NY: AMACOM/American Management Association. This
book is brief but important. Its advice on controlling information flows
is especially pertinent in the information age.

Pollar, Odette. 1996. *365 Ways to Simplify Your Work Life*. Chicago,
IL: Dearborn Trade Publisher (800/829-7934). More practical advice to
increase your effectiveness and reduce stress.

> \"Chance favors the prepared mind.\"
>
> \--Albert Einstein\--

### Establish a filing system

A key part of being organized is creating a filing system that helps you
retrieve stored information in a timely fashion. Such a system is
critical for both printed documents and (increasingly) computer files.
If you can\'t find a file, it doesn\'t matter if you have it somewhere
or not\--it\'s worse than useless, because you waste time looking for
it.

Be selective about the items you file. For years I clipped articles from
newspapers on subjects in which I maintained an active interest,
thinking that someday I might write papers on those topics. In most
cases I never did, and all that effort was for naught. There was one
example where the files came in handy for a research project, but I can
only think of that one. I stopped clipping most newspaper articles years
ago, but I still save the articles and reports that I find particularly
valuable.

Peter Schwartz (1996) also gave up an elaborate filing system because he
never used it. His approach is one to emulate: \"I concentrate on
educating myself; on passing information through my mind so it affects
my outlook; on tuning my attention as if it were an instrument\". He
recognizes that this approach may have the drawback of not being able to
draw upon previously compiled materials, but that can be a benefit in
disguise: \"Sometimes I must go back and re-create all the research I
did several years before. But that, in itself, is valuable, because in
the fields I care about, the facts have changed since I last went to
look for them\". Plus, he saves time by only filing those key items that
are most crucial to his work.

Paper should pass through your hands only once on the way to the file or
the recycling bin. You may never achieve this ideal in practice, but
strive for it and you will save time and be more effective.

Never clip anything you can easily find later, especially if you can
search for it electronically. As more newspaper articles and reports
have become available on the internet, my need for hard copy storage has
decreased. I now know that current events are well documented on the
World Wide Web, and I need only call up one of the many competent search
engines to get a nice listing of recent articles on particular topics.
Such electronic retrieval sure beats clipping and filing newspaper
articles! If you need to create your own database of electronic
clippings, there are many capable sheet scanners that will quickly and
efficiently put these clippings on your computer.

All new reports that my group creates are put on the World Wide Web, in
a file format that will allow all types of computers to read it
(PDF[^32]). Others can then download our reports without the time and
expense of me having to send the hard copies. I also don\'t have to
worry about where the copies of the printed reports are\--I can just
call them up in my web browser and view them. Often this is easier than
hunting around in the filing cabinet for an old report.

Order files by active projects and by major subject areas. Also have
file drawers for people, institutions, and for administrative issues.
The \"People\" drawer holds files for particular individuals who work in
my field (it is usually easier for me to remember who worked on a
particular report than to remember the subject heading where I filed
that report\--having a file for individuals gets around that problem).
The \"Institutions\" drawer is analogous to the People drawer. Create an
archive for old projects that is in one of the less accessible filing
cabinets, and put the most widely used \"active\" files near your
workspace.

A key trick to making any filing system work is to go through your files
periodically to familiarize yourself with where things are. This
occasional sorting will give you the opportunity to discard archaic
materials and refile items that belong somewhere else. No filing system
is ever finished. Periodic sorting helps keep your system current and
useful.

In a World Wide Web browser like Netscape Navigator or Microsoft
Explorer, it is easy to record \"bookmarks\" for sites that you
frequently visit. I have a couple of hundred of these bookmarks, and I
accumulate more every day, so it\'s important to organize these well.
Create a structure that parallels your hard-copy filing system. These
bookmarks are crucial to fast information retrieval, so spend the time
to organize them.

Computerized reference databases like EndNote can make any filing system
more effective. Each report is one record in the database, and it is a
trivial matter to add a sentence in the notes section of each reference
record that indicates the file drawer in which the hard copy is stored.
These databases can be searched electronically, which can be a godsend
when you\'re looking for a crucial report.

To preserve access to your stored articles and papers, DON\'T lend them
out to others unless this lending is recorded in a loose-leaf binder
devoted to this chore. I routinely reach for a document on my shelves at
work, only to find that someone has borrowed it. Such a written record
can help you track down missing reports when this happens.

Most people borrow reports and then forget they have them. You should
insist that someone borrowing a report copy the relevant pages (obeying
copyright laws, of course) and return it to you promptly. This
insistence will save you precious hours in the long run.

#### Further reading

Culp, Stephanie. 1990. *Conquering the Paper Pile-up: How to Sort,
Organize, File, and Story Every Piece of Paper in Your Home and Office*.
Cincinnati, Ohio: Writer\'s Digest Books. Recommends a 4 pile system: To
Do, To Pay, To Read, To File.

Lively, Lynn. 1996. **The WorkSmart Series**. *Managing Information
Overload*. New York, NY: AMACOM/American Management Association. Advice
on remaining organized in the face of a flood of information.

Schwartz, Peter. 1996. *The Art of the Long View: Planning for the
Future in an Uncertain World*. New York, NY: Doubleday. See the section
on \'Information hunting and gathering\'.

> \"There\'s nothing remarkable about it. All one has to do is hit the
> right keys at the right time and the instrument plays itself\"
>
> \--Johann Sebastian Bach\--[^33]

### Build a toolbox

A problem solver needs a toolbox as much as a wood worker needs a lathe.
My toolbox is the set of tricks and techniques that I have mastered and
can pull out when needed to solve a particular problem. Each person\'s
toolbox is different.

#### Language

Each field has its own language. You must at least develop working
familiarity with basic concepts in a given field before attempting to
decipher a problem. If you are delving into a field for the first time,
obtain a textbook and begin your reading there. Textbooks conveniently
summarize the core information in a given field, and usually provide a
nice glossary and lists of further references. Another good place to
begin is the list of frequently asked questions on a particular topic
that is available on the internet for virtually any topic you can name.
Talk to people you trust who know something about the topic, because
they can often point you to the best reference sources and save you time
(LINK TO INQUIRE).

#### Calculations

Darrell Huff\'s *The Complete How to Figure It*, is a beautiful
compilation of hundreds of ways to calculate numbers important to daily
life. Huff is the fellow who blessed the world with the book *How to Lie
with Statistics*. Some of the major headings in the book, just to give
you a flavor, are as follows:

Lifetime money strategy

Some personal things (like life expectancy)

Interest and saving

Investing

Other people\'s money

House planning

Building things

your car

Travel

Outdoors

Conversions

The book is chock full of specific examples (more than 350!) that you
can replicate when doing your own calculations. I recommend adding it to
your toolbox.

Another nice source of math techniques is a book called *Used Math*,
which is more advanced than Huff\'s book. It summarizes the basics of
calculus and other essential mathematical tools, mainly for students of
science. If you do find yourself needing to refresh your knowledge of
more advanced math, this book is a great place to start.

The best textbook I\'ve found on practical applications for mathematics
is called *For All Practical Purposes* (COMAP 1997). It covers five
major areas: management science, statistics (the science of data),
coding information, social choice and decisionmaking, and size and
shape. Most mathematics texts shy away from real world applications, but
this one breaks the mold.

#### Calculational Computer Tools

To investigate any particular software package, I recommend that you
browse in your local bookstore. Writing computer help books has become a
cottage industry, and within weeks of publication of a new piece of
software there is often a book on the shelves to assist you with its
use.

Spreadsheets and databases are the two most common computer tools used
for calculations. The first spreadsheet was VisiCalc, which was created
in the late 1970s, and was a great innovation compared to the
programming languages that had been commonly used for business
applications before that. I use spreadsheets every day. They are fine
for simple analyses, and powerful enough for moderately complex modeling
tasks. They are also handy for graphing results and creating final
formatted tables for reports.

Databases are useful when you have large quantities of data that have
the same format. For example, if you have a mailing list of three
thousand names, with addresses, phone numbers, and email addresses, a
database is the right tool for organizing it. A good database will allow
you to export the data in any format . You may just want the names and
addresses in one case, and the names and email addresses in another
case. Once entered into the database, you can extract data any way you
want.

A more advanced set of tools is available for statistical analysis of
data. These software packages are typically quite expensive, and are
generally for expert use only. They are worth investigating if you need
the horsepower and have the knowledge to use them effectively. They can
help you divine relationships between factors that are related in a
complex way.

#### Presentation Tools

Creating presentations in a word processor is a waste of time. There are
now presentation packages (like PowerPoint) that make designing
presentations a breeze. These software tools automate and simplify many
of the difficult aspects of creating presentation materials, such as
ensuring consistent formatting. If you do a lot of presentations, I
highly recommend that you obtain such software.

#### Further reading

Consortium for Mathematics and its Applications (COMAP). 1997. *For All
Practical Purposes*. New York, NY: W. H. Freeman and Co. Project
Director: Solomon Garfunkel. A truly outstanding textbook on using
mathematics in the real world.

Huff, Darrell. 1996. *The Complete How to Figure It*. New York, NY: W.
W. Norton & Co., Inc. An excellent sourcebook, with hundreds of examples
of how to solve different problems in the real world.

Jones, Gerald E. 1995. *How to Lie with Charts*. San Francisco, CA:
Sybex. A fine source for ideas on graphical and tabular presentation.

Swartz, Clifford E. 1973. *Used Math: for the First Two years of College
Science*. Englewood Cliffs, New Jersey: Prentice-Hall, Inc. Summarizes
mathematics that can be useful for advanced analysts.

Tufte, Edward R. 1995. *The Visual Display of Quantitative Information*.
Cheshire, CT: Graphics Press. Keep this classic book on your shelf for
periodic perusal and ready reference. Reread it every year or so.

> \"Applying computer technology is simply finding the right wrench to
> pound in the correct screw\"
>
> \--Anonymous\--

### Put facts at your fingertips

My college buddies used to object to my tactics in our numerous
discussions about public affairs, because I would be the first to pull
out a reference book to check their sometimes wild assertions.[^34]
While your friends may also think it unfair, your discussions will be
far more fruitful.

#### General

The most important single document for a U.S. based researcher in
virtually any quantitative field is the Statistical Abstract of the US
(U.S. Bureau of the Census 1997). It is available from the National
Technical Information Service (NTIS) for \$33 in paperback.[^35] For an
additional \$50, you get a CD ROM with all the data in electronic form.
It\'s a real bargain, containing about 1500 tables summarizing
everything from air pollutant emissions to population, average
temperatures to voter registration, and hunting licenses to federal
budget outlays. The CD-ROM version is especially nice because it allows
easy access to tables in spreadsheet format, giving you a head start on
your analysis.

The Statistical Abstract is the best source for those items (like
population and economic activity) needed to normalize any data you get
from other sources (for example, absolute growth in cement production
over time is not really meaningful until you normalize it on a per
capita or per \$ of GDP basis). It also contains the latest information
on inflation over time, for use in correcting any current dollar
statistics you obtain to constant (inflation adjusted) dollars (LINK TO
DOCUMENT, DOC, DOC SECTION). The data are taken from a variety of
sources, and are carefully documented. The Stat Ab is a great place to
start any investigation, and you should have a recent version on your
shelf at all times. All libraries will have the latest one.

A companion to the Stat. Ab. is a good almanac. While some of the data
will overlap, the almanac will contain a bunch of useful stuff that the
Stat. Ab doesn\'t have, like zip codes, postal rates, tax rates,
historical events, sociopolitical facts about states and countries, and
other information that you didn\'t know you really needed (but you
really do). There are a couple of common almanacs, namely The World
Almanac and Book of Facts and The Information Please Almanac.[^36] The
publisher of the latter source also produces a version of particular
interest to business people, as well as other variants. They are
available in any bookstore or library.

Another good source of data is a book called *Reading the Numbers*
(Blocksma 1989). This book summarizes the meaning of numbers we
encounter in everyday life, like the codes found on the side of tires,
or the conventions used in describing fluorescent light bulbs. It is
unfortunately out of print, but can be obtained at used book stores and
some libraries.

A similar book that is more readily available is *Numbers: How Many, How
Long, How Far, How Much* (Sutcliffe 1996). The variety of data contained
in this book is truly astounding. In one section there are estimates of
safe food storage times in the refrigerator and freezer, and in another
there is a list of the number of different types of instruments in a
typical symphony orchestra. The major sections include money, food and
nutrition, health, travel, daily life, transportation, time, weather,
people and places, the universe, and conversion tables/everyday math.
The book contains more than 600 pages of data, and it\'s well worth
having on your shelf.

WRI (1998) and Brown et al. (1998) contain essential information related
to resource consumption and environmental effects around the world,
including pollutant emissions, fresh water resources, land use,
population and health, food and agriculture, wildlife and habitat,
energy, atmosphere and climate, and policies and institutions.

One outlet for data on other countries is the PEMD Education Group in
Cloverdale, CA.[^37] PEMD sells for \$75 a CD-ROM that contains 500 data
series from the World Bank with data from 1970 to 1995 for 209
countries. The topic areas for these data are overview, people,
environment, states & markets/global links, national accounts (both in
local currency and U.S. dollars), money/balance of payments/debt, and
production/trade/government finance. If you are doing international
comparisons, this source is a useful one. The CD-ROM contains software
to help you easily maneuver among the hundreds of data files.

#### Web/internet data

More data appear on the internet every day. It is an impossible task to
report the latest developments in a book like this one (things change
too fast),[^38] but there are a few key sites of which you should be
aware:

\(1\) The Library of Congress site
(http://lcweb.loc.gov/homepage/lchp.html) is an invaluable source. You
can search many parts of the Libraries databases with little effort.
Many local libraries have also placed their catalogs on the internet,
which can save you a trip to the library next time you want to do
research.

\(2\) The U.S. Bureau of the Census site (http://www.census.gov/) is
beautifully designed and is a treasure trove of social, demographic, and
economic data. You can search by key word, by place, by clicking on
maps, or in many other ways. Most of the data on this site are free or
available at nominal cost.

\(3\) The National Technical Information Service (NTIS) is the official
source of U.S. government reports. NTIS sells almost three million
titles, and you can search on their nicely designed web site
(http://www.ntis.gov) to find the one you want.

\(4\) The U.S. Department of Energy\'s Energy Information Administration
site (http://www.eia.doe.gov) is the right place for energy and some
environmental data for the U.S. The site also has limited information on
certain international topics, like nuclear power or energy use in other
countries. Most of the data on this site are free or available at
nominal cost.

\(5\) The U.S. Environmental Protection Agency\'s site
(http://www.epa.gov/) contains huge quantities of environmental data, as
well as access to EPA\'s reports.

\(6\) The Fed Stats site is a treasure trove for U.S. economic data
(http://www.stat-usa.gov/stat-usa.html). It allows easy access via the
World Wide Web to data from more than 50 federal agencies. The downside
is that it costs \$150/year for individuals to access.

\(7\) The National Center for Health Statistics site
(http://www.cdc.gov/nchswww/index.htm) contains data on U.S. health as
well as downloadable reports on this topic.

\(8\) There are hundreds of *newsgroups* on everything from Star Trek to
statistical analysis. By posting an email to the relevant newsgroup, you
can often get answers to questions from people all over the globe in a
matter of hours or days. You can generally access these newsgroups using
your web browser. Consult your internet service provider for details.

There\'s a real art to searching for information on the internet.
Different search engines bring to you only a subset of the information
actually available, so you\'ll need to use several to be certain you\'ve
done a thorough search. For example, when I searched for the words
\"information overload\" on the Alta Vista search engine, it returned
about 1000 documents. When I used Infoseek, it returned just under 5000.
Not all of the items on the Alta Vista list were in the Infoseek list,
and vice versa. Skill is also involved in narrowing down your search so
that you have at most dozens of documents to sort through, instead of
the thousands that these engines commonly produce for you. Search
engines try to rank their results by relevance, but the algorithms for
ranking the relevance of documents also differ.

Refer to Barrett (1997) for research strategies on-line. A convenient
site where you can access the most common search engines is
http://www.lbl.gov/Glimpse/search-all.html, and a nice summary of the
characteristics of the most widely used search engines is found at
http://hplus.harvard.edu/about/srchcomp.html.[^39]

#### Context

Sometimes in presenting the results of analysis, you\'ll want to know
the intellectual history behind a certain concept. For example, if you
are giving a talk about energy policy to a general audience, you might
want to open with a brief summary of how the concept of energy was
developed, and how our modern conception differs from that common in the
1800s. In this circumstance, two essential works are *The Encyclopedia
of Philosophy* (Edwards 1972) and *The Oxford English Dictionary*
(Simpson and Weiner 1989). The former is helpful in tracing the genesis
of a particular concept through the ages, while the latter tracks the
origins of particular words. Each of these sources can be useful in
writing a speech or a paper, and are found in most libraries.

An alternative to the conventional *Encyclopedia of Philosophy* is the
Stanford Encyclopedia of Philosophy, which is a web-based encyclopedia
of comparable scope to the hard-copy version, on-line at
http://plato.stanford.edu. It is constantly being updated and evaluated
by its editorial board (they call it a dynamic encyclopedia). The
creators of this new tool felt that the old way of creating an
encyclopedia was too slow to capture the rapid developments in
philosophical thinking (particularly those developments related to
scientific concepts).

A good book of quotations is essential to liven up your writing and
presentations, and a standard dictionary and thesaurus will make your
writing more precise. These are available in myriad incarnations in any
bookstore. If you have easy access to the web, a large collection of
quotations can be found at
http://www.cyber-nation.com/victory/quotations/quotes_menu.html. You can
search Roget\'s thesaurus at
http://humanities.uchicago.edu/forms_unrest/ROGET.html or the Merriam
Webster thesaurus at http://www.m-w.com/thesaurus.htm.

#### Conversion factors

For physical constants and conversions, there\'s nothing better than the
Handbook of Chemistry and Physics. It comes out every year, but because
physical laws and constants don\'t change, even an old version of this
work is handy, so pick one up at a used book store. Virtually all
libraries have a copy of this book on the shelf.

There are certain facts that I use frequently. For example, I often need
to convert English units to metric units, and vice versa. To help this
process along, I memorize the most commonly used conversions and keep
the rest on a one page data sheet for ready reference. I recommend
compiling such a sheet for your own use that contains the conversions
most useful to your work. The hour or so of work needed to compile this
page will be repaid many times over in the convenience of not having to
dig deeply into obscure reference books whenever you need to do a
calculation.

#### Further reading

Daniel J. Barrett. 1997. *NetResearch: Finding Information Online*.
O\'Reilly and Associates. (for a summary go to
http://www.ora.com/catalog/netresearch/ and for the latest updates to
the book as well as related links, see
http://www.cs.umass.edu/\~barrett/netresearch/)

Berkman, Robert I. 1990. *Find It Fast: How to Uncover Expert
Information In Any Subject*.. New York, NY: Harper & Rowe, Publishers. A
helpful guide to digging out data from the myriad disparate sources out
in the world.

Blocksma, Mary. 1989. *Reading the Numbers: A Survival Guide to the
Measurements, Numbers, and Sizes Encountered in Everyday Life*. New
York, NY: Penguin USA. This book is somewhat hard to find, but it
explains how to interpret and use a variety of data that we encounter in
everyday life.

Brown, Lester R., Michael Renner, and Christopher Flavin. 1998. *Vital
Signs 1998: The Environmental Trends That Are Shaping Our Future*. New
York, NY: W.W. Norton and Co. Tons of good time-series data on
environmental trends. Worldwatch also sells these data in electronic
form in both Windows and Macintosh formats.

Edwards, Paul, ed. 1972. *The Encyclopedia of Philosophy*. New York, NY:
Macmillan Publishing Co., Inc. & the Free Press. vols. 1 through 8. The
best place to go when you need some background on an intellectual
concept.

Famighetti, Robert, ed. 1996. *The World Almanac and Book of Facts
1997*. Mahwah, New Jersey: World Almanac Books. A \"must have\"
reference. It comes out every year, and is chock full of useful
information.

Simpson, John and Edmund Weiner, ed. 1989. *The Oxford English
Dictionary*. Oxford, UK: Oxford University Press. 2d edition. This
reference work takes up 20 volumes and 22,000 pages! This version costs
\$3,000, but you can get a complete version with *very* small type for
about \$350 (it comes with its own powerful magnifying glass!). A better
choice is the CD-ROM version, available for both Windows and Macintosh
for about \$400.

Sutcliffe, Andrea. 1996. *Numbers: How many, How Long, How Far, How
Much*. New York, NY: Stonesong Press, Inc. Another good sourcebook for
all sorts of helpful data.

U.S. Bureau of the Census. 1997. *The Statistical Abstract of the United
States 1997* 117th edition Washington D.C.: U.S. Government Printing
Office. One of the most important data sources for any U.S. researcher.

Lide, David R., ed. 1997. *CRC Handbook of Chemistry and Physics*. 78th
ed. Boca Raton, Florida: CRC Press, Inc. The best source for physical
constants and conversions. CRC also has many other technical and
scientific reference works.

WRI (World Resources Institute). 1998. *World resources 1998-99*.
Oxford, UK: Oxford University Press. Every two years, WRI compiles key
environmental data from all over the world. This edition focuses on
indicators of environmental change and human health, but it also has the
core data on energy and environment that have been the hallmark of this
publication for many years.

> \"Data, Data, Data! I can\'t make bricks without clay! \"
>
> \--Sherlock Holmes\--

###   {#section-1 .unnumbered}

### Value your time

Americans always seem to rush around. There\'s never enough time to do
everything we want to do, see everyone we want to see, or travel
everywhere we want to go. In part, this busyness is an inherent part of
modern-day life, but it is also partly self-imposed.

Your time is your life, and it is precious. If someone is wasting it,
they are stealing it from you. Many people don\'t think about this
equation, and they allow people and institutions to monopolize their
time and sap their energy, without commensurate rewards.

The most articulate statement of this principle is found in *Your Money
or Your Life* (Dominguez and Robin 1993). In this book, the authors
describe a set of simple techniques to help people internalize this
equation and make it part of their daily decision making. They encourage
people to evaluate their spending by calculating how many hours of labor
(\"life energy\") will need to be spent for each purchase. The value of
money becomes more real after you ask yourself two key questions:

\(1\) \"Did I receive fulfillment, satisfaction, and value \[from this
purchase\] in proportion to life energy spent?\"

\(2\) \"Is the expenditure of life energy on this purchase in alignment
with my values and stated life purpose?\"

One hundred dollars has an abstract quality. It is made more real when
comparing various purchases that can be made for that amount. It becomes
most real when put in terms of the four or eight or twenty hours of
labor needed to earn the money to buy this new toy you want.

This line of argument dates back to Thoreau, who in *Walden* compared
the cost of train fare to the hours of time needed to earn the money to
pay for it, and concluded that walking was preferable for him. It has
further implications for your choices of where to work and how to spend
your personal time.

Stephen Covey (1990) describes how most people allow urgent items to
dominate their lives, when the most effective people focus significant
attention on *the important items that are not urgent*. Be selective
about the endeavors on which you spend time. Prioritize. Choose jobs
with the greatest likelihood for success and the highest potential
payoff. Working at a dead-end job can sap time and energy from a more
high valued use of your time.

Think carefully about time spent with others: Does it add to your life?
Does it make you happy? If not, then do something about it. Don\'t let
anyone steal your life!

Avoid going to committee meetings except where the agenda is well
focused and the person chairing the meeting will keep things on track.
Most meetings wander into irrelevancy, and most people who call meetings
have no respect for the time and money wasted. Large meetings should be
avoided except when absolutely essential. I knew my meetings were
valuable when my staff of 12 actually requested that our biweekly group
meetings last for a full hour instead of the 35 minutes we typically
took, to allow more in depth discussion of each person\'s current
research.

Finally, you must protect your most productive time from interruptions.
Each of us works best at particular times of day. I find that my best
writing occurs in early afternoon and early evening. My best
calculations are done in the late morning. Find your most productive
times and guard them jealously. Your life can be stolen by constant
interruptions just as well as it can by people wasting large blocks of
time in meetings.

If you are feeling unproductive, do something which takes little mental
energy or effort but will yield dividends later (like sorting your
files, reading mail, or answering phone calls). Even \"down time\" can
be useful if you match the task to your mental energy level.

#### Further reading

Covey, Stephen R. 1990. *The 7 Habits of Highly Effective People*. NY,
NY: Simon & Schuster. Covey is a highly successful \"personal growth\"
guru who doles out advice for a living. His book helps you explore how
your life patterns thwart or facilitate success.

Dominguez, Joe and Vicki Robin. 1993. *Your Money or Your Life:
Transforming Your Relationship with Money and Achieving Financial
Independence.* New York, NY: Penguin Books. Includes a clear and
persuasive statement of how money, time, and your life are intimately
connected.

Hyams, Joe. 1982. *Zen in the Martial Arts*. New York, NY: Bantam Books.
The chapter titled \"Do not disturb\" describes how Hyams and his
martial arts colleagues take command of their time.

Nhat Hanh, Thich. 1991. *Peace is Every Step: The Path of Mindfulness in
Everyday Life*. New York, NY: Bantam Books. This book advocates
awareness in every waking moment of every day, so you never forget how
amazing the world is. Sadly, most of us forget this all the time.

Thoreau, Henry David. 1971. *Thoreau: Walden and Other Writings*. New
York, NY: Bantam Books, Inc. Joseph Wood Krutch, ed. Contains Thoreau\'s
musings during his stay at Walden Pond.

FIGURE: Dilbert cartoon on pointless meetings or drawing of an hourglass

> \"Furious Activity is No Substitute For Understanding.\"
>
> \--H. H. Williams\--[^40]

## Assess their analysis

Any time you\'re exploring work others have done, you\'ll need to know
what to look for. The following six sections give some advice to make
these explorations productive ones.

*Question authority*: This catch-phrase of the 1960s is still applicable
today. Authority figures can be wrong or biased, so investigate their
assertions in the same way as you\'d examine those of someone with whom
you\'re not familiar.

*How guesses become facts*: Always remember that \"official\" statistics
are based on calculations that are often poorly documented, incorrectly
cited, or otherwise hazardous to your intellectual health.

*Don\'t believe everything you read*: Maintain a healthy skepticism,
even of well established sources. In this age of instant information
transmittal, rumor and error seem to propagate even more quickly than
truth.

*Go back to the questions*: Any time you rely on survey data to make an
important decision, refer back to the actual questionnaire upon which
the survey data are based or risk misinterpreting the data.

*Reading tables and graphs*: First check for internal consistency, then
see if the results contradict other facts you know to be true. Search
for cognitive dissonance: any discrepancy between the author\'s results
and what you already know will give you leads to investigate further.

*Distinguish facts from values*: Don\'t be fooled by technical people
who portray their advice as totally rational, completely objective, and
value-free. If they have made a choice, they have also made a value
judgment.

Your own systematic thinking can uncover logical blunders that the less
critical observer will overlook. Lieber (1944) gives many examples of
apparently obvious conclusions that are erroneous, as does Huff (1993).
Hughes (1997) explores in a more general way correct and incorrect modes
of argument. By investigating carefully and thoughtfully, you will be
able to evaluate other people\'s analysis as a matter of course, and
thus contribute to your understanding of the task at hand.

#### Further reading

Huff, Darrell. 1993. *How to Lie with Statistics*. New York, NY: W. W.
Norton & Co., Inc. There\'s no better exposé of commonly used tricks
that mislead and misrepresent than this book.

Hughes, William. 1997. *Critical Thinking: An Introduction to the Basic
Skills*. 2d Peterborough, Ontario: Broadview Press. Hughes presents
examples and exercises to explore and categorize the logical fallacies
that befall most arguments.

Lieber, Hugh Gray, and Lillian R. Lieber. 1944. *The Education of T. C.
MITS*. New York: W. W. Norton & Company, Inc. This book is an
entertaining excursion into mathematics, logic, and argument. Though
it\'s dated in some ways, it\'s still worth digging out of the public
library.

> \"Just because I use a study to refute another study does not mean my
> study is right. It just means I believe it. Caveat Emptor.\"
>
> \--Cynthia Crossen\--

### Question authority

> \"Sacred cows make the best hamburger.\"
>
> \--Mark Twain\--

\"Question Authority\" is one of the favorite T-shirt and bumper sticker
slogans in Berkeley. It dates back to the 1960s, when students engaged
in well-publicized defiance of authority figures in the campus
administration and the state government of California. While some
aspects of these emotion-charged times are best left in the past, it is
time to dust off \"Question Authority\" for the 21st Century.

There are, of course, excellent reasons for following authority in many
instances. People become authority figures by virtue of hard work,
unique experience, special insight, and long training. In some
situations, such as military maneuvers, obedience to authority is
essential to achieving the mission and preserving lives. Obedience in a
business setting can also be critical, and can allow people to act in a
concerted and collective way to achieve the common goal of increased
profit for the firm.

However, most situations in everyday life are not as dangerous as
military maneuvers. There are two main reasons why questioning authority
is often a healthy response:

• Even brilliant people affiliated with well-respected institutions are
not always right. History is littered with examples of wrong-headed
conventional wisdom and misjudgments by otherwise outstanding
individuals. Recall what a Yale University management professor said
about Fred Smith\'s paper proposing reliable overnight delivery service:
\"The concept is interesting and well-formed, but in order to earn
better than a \'C,\' the idea must be feasible\". Smith went on to found
Federal Express Corporation, which now does billions of dollars in
business carrying overnight packages every year.

• Many salespeople and marketing professionals use deference to
authority to gain advantage and influence consumer behavior. Robert
Cialdini documents dozens of amusing and sometimes disturbing examples
of how these \'compliance professionals\' use authority figures and
pseudo-authority figures to sell products. One famous example is
described in Cialdini\'s book, where the actor Robert Young, who used to
play Marcus Welby, M.D. on TV, advises people to buy Sanka coffee for
its health benefits: \"Objectively, it doesn\'t make sense to be swayed
by the comments of a man we know to be just an actor who used to play a
doctor. But, as a practical matter, that man moved the Sanka. (p.220)\"

So long as there is time to reflect and consider the dictates of
authority, it\'s healthy to question it. Cialdini proposes asking two
questions when presented with an ostensible authority figure (pp.
230-2):

• \"Is this person truly an expert?\" If not, then their advice should
be given no more weight than that of other people with no special
knowledge.

• If they are a legitimate expert, however, it then behooves us to ask a
second question, namely \"How truthful can we expect the expert to be
here?\" By asking whether the authority figure (or the institution
funding her) stands to benefit from the advice she is supplying, we gain
insight into the trustworthiness of that advice. If the expert has a
vested interest in one or more potential outcomes of your decision,
their advice should be discounted and you should seek corroborating
information. The advice may still be sound, but it is worth special care
in this case.

If the expert\'s interests coincide with yours, or the expert is
demonstrably impartial, then more weight should be ascribed to their
opinions. Even in this case, however, it is best to seek corroborating
evidence, because even experts can be mistaken.

One recent example where the interests of the researchers and their
funding institutions led to widespread (and well justified) public
skepticism is the furor over research on the health effects of smoking.
For years, the tobacco companies funded research ostensibly
demonstrating that smoking did not cause adverse health effects, all the
while knowing that it did. Investigating who is paying for the expert\'s
opinion is a particularly fruitful line of inquiry when such cynical
manipulation of research is so common.

Some authority figures unknowingly promote the messages of
self-interested institutions. In *Diet for a New America*,[^41] John
Robbins marvels at the way that the dairy and meat industries have for
decades promoted attitudes favorable to their products by contributing
educational materials to schools. All of us who grew up in the 1960s and
1970s remember the advice to drink milk and eat meat for health, but
those nice charts describing the basic \"food groups\" were part of a
brilliant marketing campaign, in spite of how official they looked.

The flood of facts and falsehoods in this data-laden age puts greater
responsibility on us to question \"information\" that appears to be from
authority figures. Information found on the internet is less subject to
institutional credibility checks than are stories published by more
established information sources like the New York Times, but even the
Times can propagate nonsense. Proceed with caution, and always
corroborate what you hear or read with multiple independent sources
before taking action.

#### Further reading

Cialdini, Robert B. 1993. *Influence: The Psychology of Persuasion*. New
York, NY: Quill\--William Morrow. An extraordinary book that documents
how \"compliance professionals\" influence people\'s opinions and
actions.

Crossen, Cynthia. 1994. *Tainted Truth: The Manipulation of Fact in
America*. New York, NY: Simon & Schuster. Another eye opener that shows
how people and institutions can and do play with numbers to convince

> \"Loyalty to petrified opinion never yet broke a chain or freed a
> human soul.\"
>
> \--Mark Twain\--

######## Conventional wisdom (sic) from well known authorities

\"But what \... is it good for?\" \--Engineer at the Advanced Computing
Systems Division of IBM, 1968, commenting on the microchip.

\"There is no reason anyone would want a computer in their home.\"
\--Ken Olson, president, chairman and founder of Digital Equipment
Corp., 1977

\"So we went to Atari and said, \'Hey, we\'ve got this amazing thing,
even built with some of your parts, and what do you think about funding
us? Or we\'ll give it to you. We just want to do it. Pay our salary,
we\'ll come work for you.\' And they said, \'No.\' So then we went to
Hewlett-Packard, and they said, \'Hey, we don\'t need you. You haven\'t
got through college yet.\'\" \--Apple Computer Inc. founder Steve Jobs
on attempts to get Atari and H-P interested in his and Steve Wozniak\'s
personal computer.

\"640K ought to be enough RAM for anybody.\" \-- Bill Gates, 1981

\"This \'telephone\' has too many shortcomings to be seriously
considered as a means of communication. The device is inherently of no
value to us.\" \--Western Union internal memo, 1876.

\"The wireless music box has no imaginable commercial value. Who would
pay for a message sent to nobody in particular?\" \--David Sarnoff\'s
associates in response to his urgings for investment in the radio in the
1920s.

\"Who the hell wants to hear actors talk?\" \--H.M. Warner, Warner
Brothers, 1927.

\"A cookie store is a bad idea. Besides, the market research reports say
America likes crispy cookies, not soft and chewy cookies like you
make.\" \--Response to Debbi Fields\' idea of starting Mrs. Fields\'
Cookies.

\"We don\'t like their sound, and guitar music is on the way out.\"
\--Decca Recording Co. rejecting the Beatles, 1962.

\"Heavier-than-air flying machines are impossible.\" \--Lord Kelvin,
president, Royal Society, 1895.

\"Professor Goddard does not know the relation between action and
reaction and the need to have something better than a vacuum against
which to react. He seems to lack the basic knowledge ladled out daily in
high schools.\" \--1921 New York Times editorial about Robert Goddard\'s
revolutionary rocket work.

\"Stocks have reached what looks like a permanently high plateau.\"
\--Irving Fisher, Professor of Economics, Yale University, 1929.

\"Airplanes are interesting toys but of no military value.\" \--Marechal
Ferdinand Foch, Professor of Strategy, Ecole Superieure de Guerre.

\"Everything that can be invented has been invented.\" \--Charles H.
Duell, Commissioner, U.S. Office of Patents, 1899.

\"Louis Pasteur\'s theory of germs is ridiculous fiction\". \--Pierre
Pachet, Professor of Physiology at Toulouse, 1872

###   {#section-2 .unnumbered}

### How guesses become facts

All data should be treated with skepticism. Here one example of how
official statistics get created, as recounted by Alan Meier, a long-time
colleague and friend:

In 1987, I (with Steve Greenberg\'s assistance) wrote an article in an
energy magazine about the rising amount of energy use that did not fit
into the traditional categories[^42]. As part of the article, we created
three tables showing ownership of these appliances and their estimated
annual energy consumption. These values were based on very limited
monitored data, back-of-the-envelope calculations, and hunches. The
tables were assembled in one evening. (Many of the envelopes with
calculations were then discarded.)

In 1989, the U.S. Department of Energy\'s Energy Information
Administration (EIA) published its official "Household Energy
Consumption and Expenditures", including the results of their 1987
survey and additional analysis.[^43] The EIA published the Meier and
Greenberg data in a new table, "U.S. End-Use Consumption of Electricity
for Selected Appliances". Whereas we published ranges in our estimates,
the EIA just calculated and printed the averages from the high and low
values. The word "estimated" appeared nowhere in the EIA table, so the
reader was led to believe that these numbers were exact. (Curiously, the
EIA is careful to give confidence bounds and other statistical
parameters for its own survey data.) To add insult to injury, the EIA
misspelled my name in the citation.

Alan\'s example is more the rule than the exception. When little
information is available about a particular topic, any moderately
credible source gets cited by everyone concerned with the topic and
becomes the new conventional wisdom. This happens frequently even though
such estimates are often based on extremely crude assumptions. I
sometimes joke that ALL estimates of energy use of appliances somehow
originate with Alan.

Another colleague, Chris Calwell, was hired in 1996 to write a research
report about the energy and safety problems with halogen torchieres,
those inexpensive floor lamps that have become so popular in recent
years. Here is his story:

In order to write the report, I needed to figure out how many halogen
torchieres had been sold. The trouble was, nobody knew. The Census
Bureau knew how many halogen bulbs had been imported, but not how many
were in fixtures. California utilities had counted how many were in a
small sample of houses, but that was before the lamp had become popular.
Market researchers had asked how many people bought halogen lamps of all
types, but didn\'t know how many were torchieres. The library was not
much help, either, since only two articles had ever been published on
this subject before I set out to write mine.

I called the author of those two articles and got the names and numbers
of her sources (the manufacturers). Then I called the manufacturers and
asked how many they thought had been sold in total the previous year . I
also asked them how average prices for the lamps had changed over time
and about when they began selling the lamps in the United States. Taking
all these different pieces of information under consideration, I created
a table of sales of torchieres per year and the number still in use. It
was pure guesswork, informed by the information I could find, but
guesswork nonetheless, with a fancy spreadsheet and graphs to back it
up.

The U.S. Consumer Product Safety Commission, a federal agency, was about
to put out a press release on the problems with halogen torchieres at
the same time I was finishing my report, but they also had no idea how
many had been sold. They asked for my number (40 million), which my
publisher reported to them as 35 to 40 million to be conservative. The
CPSC thought that range seemed high, so it used 30 to 40 million in its
press release to be even more conservative. Over the next year, dozens
of newspaper articles and TV programs cited the CPSC estimate of 30 to
40 million and attributed it to the federal agency, ignoring the
original source and never bothering to examine the eight-line footnote
in my original report documenting how the original estimate of 40
million was determined. The number has taken on a life of its own, proof
of truth by institutional adoption and media repetition.

Your best defense against these sanctified guesses is to read the
documentation for any data, including the footnotes. Track down cited
sources and read them, too. You should be able to figure out the methods
used to create any data. If the documentation is not up to this task,
you should regard the data with extreme suspicion. Don\'t ever use data
unless you know how it was derived, you trust the cited sources, and you
agree with the stated methods.

> \"The Government are very keen on amassing statistics. They collect
> them, add them, raise them to the nth power, take the cube root and
> prepare wonderful diagrams. But you must never forget that every one
> of these figures comes in the first instance from the village
> watchman, who just puts down what he damn pleases.\"
>
> \--Sir Josiah Stamp, Inland Revenue Department of England,
> 1896-1919\--

### Don\'t believe everything you read

Never act solely on information you read in the newspaper or hear over
the internet. News stories are often wrong, either because the reporter
misunderstood her sources or because the sources themselves were
misleading or incorrect. E-mail purportedly claiming to contain valuable
information can easily be a hoax.

In March 1997, Apple computer was wrestling with severe financial
problems. Sales of its computers were declining, and the company was
losing money. I counted four separate trade press and news accounts that
stated with certainty that Apple would \"abandon\" or \"scrap\" its
Newton technology in the face of those financial problems. I followed
these accounts with some interest, because I am an avid Newton user.

In the end, the reorganization in March 1997 left the Newton division
untouched, because Apple wanted to sell or license the technology
wholesale. I knew Apple wouldn\'t scrap the technology outright because
they were planning, within a week of this reorganization, to introduce
previously announced breakthrough products in this line (the EMate 300
and the Message Pad 2000). These new technologies were demonstrably
better than any other devices on the market, and they delivered superior
customer value. Yet ill-informed reporters persisted in repeating the
incorrect statement about the Newton, in spite of their potential harm
to the sales and marketing of the new products. The reporters just got
it wrong.

Sometimes the sources themselves are at fault. The Software Publishers
Association (SPA) periodically releases estimates of the size of the
software markets for Windows and Macintosh systems. In 1995 and 1996,
the SPA proclaimed that Macintosh software sales had declined in both
years relative to the year-before estimates. The problem is that the SPA
compared *preliminary* data in 1995 and 1996 to *final* data in the
previous year, but the final data in a given year contained substantial
adjustments relative to the preliminary data (in the 1995 data, the
adjustment was 45%!). The data series used by SPA was therefore not a
consistent one. To add insult to injury, the SPA\'s Jim Sanders said
that these data clearly showed that \"the Mac platform is in decay\".

When final 1995 numbers are compared to final 1994 numbers, the real
change in Mac software sales is about a 25% increase from 1994 to 1995,
not a 14% decline as initially reported by the SPA. This result is not
surprising, since Apple shipped record numbers of Macs in 1995.

The details of why the SPA made these appalling errors are not really
that important for our purposes.[^44] The main lesson is that even
respected institutions can distribute junk data to the world, and pass
it off as legitimate. If some Mac zealots had not delved into these
numbers and compared the results year to year, unsuspecting software
developers might have been led to the wrong conclusion. In fact, these
widely reported estimates probably did damage to Apple\'s business. How
much damage no one can say for sure.

In the face of the public furor over its obvious mistakes, the SPA
finally suspended its data program in September 1997.[^45] In hindsight,
they should have subjected their methodology to external peer review and
been much more cautious in their conclusions about Macintosh software
sales.

The most famous book on ways to misuse numbers is Darrell Huff\'s *How
to Lie with Statistics*. Huff documents dozens and dozens of examples
where people, through incompetence or malicious intent, mislead readers
with numbers, graphs, and other tools of the numerate. You owe it to
yourself to read Huff\'s book. It is justly revered, both because of its
accessible tone and its message: lying with statistics is commonplace,
and it\'s wise to be aware of the most widely use tricks.

What should you do to protect yourself from such shenanigans? First, go
back to the questions (LINK). Find out who asked the survey questions,
who they asked, and when they were asked. Be a detective (LINK) and
determine the potential motives of the surveying organization. Dig into
the numbers (LINK) and compare results to other numbers you know to be
true. Are these results consistent with other sources? Can you find an
internal inconsistency in the results that casts doubt on the results?
Answers to these latter two questions made it obvious that the SPA
results were nonsense.

Finally, don\'t pass on information to dozens of people in E-mail unless
you are confident it\'s true. I\'ve received dozens of E-mail warnings
about computer viruses that will erase my hard drive or have some other
terrible effect. I know most of these are bogus because they usually
don\'t mention which kinds of computers they affect. Viruses (with a few
exceptions) afflict Macs, PCs, or UNIX machines, but one type of machine
can\'t generally \"catch\" the virus from another type.

Because of the ease with which information can be sent to dozens of
people, news (or falsehoods) can spread quickly around the world. One
recent example of this phenomenon is a collection of advice purported to
be a commencement speech given by Kurt Vonnegut at MIT. The advice was
witty, thoughtful, and altogether in keeping with Vonnegut\'s
curmudgeonly but lovable take on the world. In fact, his own wife passed
it around to her friends, thinking it was his work. It was widely
distributed on E-mail in the days following the MIT Commencement on 5
June 1997, and attributed to Vonnegut.

The essay included such gems as:

\"Wear sunscreen. If I could offer you only one tip for the future,
sunscreen would be it. The long-term benefits of sunscreen have been
proved by scientists, whereas the rest of my advice has no basis more
reliable than my own meandering experience. I will dispense this advice
now.\"

\"Do one thing every day that scares you.\"

\"Sing.\"

\"Don\'t be reckless with other people\'s hearts. Don\'t put up with
people who are reckless with yours.\"

\"Be kind to your knees. You\'ll miss them when they\'re gone.\"

\"Live in New York City once, but leave before it makes you hard. Live
in Northern California once, but leave before it makes you soft.\"

\"Floss.\"

The only problem was that Kurt Vonnegut never spoke at any MIT
Commencement, nor did he write those delightful words of wisdom. The
essay was a newspaper column by Mary Schmich, who writes for the Chicago
Tribune. The column had somehow been stamped with Vonnegut\'s name, then
sent merrily on its way to thousands of unsuspecting E-mail readers.

This last example is a little reminder of how quickly and easily
misinformation can be passed off as fact in the digital age. Keep it in
mind as you venture forth into cyberspace.

#### Further reading

Crossen, Cynthia. 1994. *Tainted Truth: The Manipulation of Fact in
America*. New York, NY: Simon & Schuster. Crossen summarizes how the
results of research are often manipulated for self-serving ends.

Huff, Darrell. 1993. *How to Lie with Statistics*. New York, NY: W. W.
Norton & Co., Inc. If this book doesn\'t make you a skeptical observer
of numbers and statistics, nothing will. First published in the 1950s,
it\'s still fresh and useful today.

> \"The surest way to corrupt a youth is to instruct him to hold in
> higher regard those who think alike than those who think differently\"
>
> \--Nietzche\--

######## What Scientists Say, and What They Mean: A Guide for the Perplexed

Phrase: It has long been known

Translation: I haven\'t bothered to look up the reference

Phrase: It is believed

Translation: I think

Phrase: It is generally believed

Translation: A couple of other guys think so, too

Phrase: It is not unreasonable to assume

Translation: If you believe this, you\'ll believe anything

Phrase: Of great theoretical importance

Translation: I find it kind of interesting

Phrase: Of great practical importance

Translation: I can get some mileage out of it

Phrase: Typical results are shown

Translation: The best results are shown

Phrase: 3 samples were chosen for further study

Translation: The others didn\'t make sense, so we ignored them

Phrase: The 4 hour sample was not studied

Translation: I dropped it on the floor

Phrase: The 4 hour determination may not be significant

Translation: I dropped it on the floor, but scooped most of it up

Phrase: The significance of these results is unclear

Translation: Look at the pretty artifact

Phrase: It has not been possible to provide definitive answers

Translation: The experiment was negative, but at least I can publish the
data somewhere

Phrase: Correct within an order of magnitude

Translation: Wrong

Phrase: It might be argued that

Translation: I have such a good answer for that objection that I shall
now raise it

Phrase: Much additional work will be required

Translation: This paper is not very good, but neither are all the others
in this miserable field

Phrase: These investigations proved highly rewarding

Translation: My grant is going to be renewed!

Phrase: I thank X for assistance with the experiments and Y for useful
discussions on the interpretation of the data

Translation: X did the experiment and Y explained it to me

SOURCE: Net lore (i.e., source unknown. I searched for the title of this
piece on Alta Vista and Infoseek and found 3 to 5 instances of it, but
no source is ever listed.).

### Go back to the questions

It is common practice for companies to make decisions about marketing
new products based on surveys of consumer behavior and preferences. In
1996, I spent about four hours filling out such a survey, and it got me
to think about the limitations of such data.

The Survey of American Consumers is conducted by the American Institute
of Consumer Studies.[^46] An engaging young woman came to my door,
explained what her job was, and started asking me questions. After about
15 minutes of questions she said that she had a booklet with more
detailed questions, and that if I filled it out and returned it in the
postage-paid envelope, I would be sent a check for \$20.

The survey itself was 96 pages long, and it contained questions on every
aspect of my purchasing behavior, from what kind of toothpaste I buy to
how many miles I drive my car per year. The sheer endurance needed to
fill out the survey was substantial, and I suspect that I was among the
more thorough respondents. I dug through my cabinets to identify the
brands of various foods and other household supplies. Sometimes I just
didn\'t have the brand name but I knew I had used some form of the
product during the specified time period, so I guessed (the instructions
to the survey said to \"estimate as best you can\").

One thing that struck me was the number of times that the question did
not allow me to explain my situation. For example, the survey asked
about my newspaper reading habits (\"Which sections of the daily
newspaper do you read?\") but did not allow me to explain that I read
the San Jose Mercury News on-line and do not receive a hard copy
newspaper.

Another question asked about whether I used talcum powder. I do use it,
but not to freshen up after a shower. Instead, I put it on ants that
have invaded my kitchen, and it drives them away, without using
poison.[^47] The makers of talcum powder might try to infer something
from the results of this survey, but it tells them nothing about my
non-conventional use of their product.

Still another question asked about where I do my food shopping, but did
not give me an option for the two local food stores at which I shop.
Thus, it appears from my responses to the survey that I have gone food
shopping once in the past 30 days, even though I had eaten quite well
during that period, thank you very much.

Given a sufficiently large number of respondents, a survey like this one
will give results that reflect the average reality of consumer
purchases. Nevertheless, the anomalies I encountered should at least
make you cautious about drawing firm conclusions from one survey without
independent confirmation of the results.

These cautions are especially important when using \"survey\" data that
can be skewed by so-called \"self selection bias\". This problem
commonly crops up when the institution conducting the survey has no
control over who responds, which is typical for surveys conducted over
the World-Wide Web. Respondents with a particular point of view can
easily \"stuff the voting box\", as Byte Magazine found when it tried to
assess the prevalence of different computer operating systems.[^48]

In spite of all these limitations, millions of business investment
dollars are spent based on the results of such surveys. Before making
such decisions, it is important always to see the actual questionnaire
used to collect the data, not simply how the data are reported. The way
the question is asked can affect the results in substantial ways. If the
data are broken down to show their location within Census tracts or zip
codes, then the small number of respondents in those boundaries may lead
to unrepresentative results.

The way questions are asked and to whom they are asked can determine the
outcome of a poll. Crossen (1994) recounts a classic example from the
early 1990s:

\"Consider this question posed by Ross Perot. In a mail-in questionnaire
published in TV Guide, the question was \'Should the President have the
Line Item Veto to eliminate waste?\'; 97 percent said yes. The same
question was later asked of a sample that was scientifically selected
rather than self-selected, and 71 percent said yes. The question was
rewritten in a more neutral way\--\'Should the President have the Line
Item Veto, or not?\'\--and asked of a scientifically selected sample.
This time only 57 percent said yes.\"

To truly understand the meaning of a survey, it is essential to go back
to the original survey questions. ***Never*** base critical decisions on
someone else\'s summary of survey results unless you have implicit trust
in that person\'s judgment and understanding of the situation. It is
also crucial to understand how the sample of respondents was selected
(self-selected samples are the bane of a good analyst\'s existence).

Real data always reflect the inherent inability of humans to track the
changes happening all around us. That\'s why it\'s important to check
new data with our intuition, experience, and independent sources of
confirmation before taking action. LINK TO Cycle OF ACTION.

#### Further reading

Crossen, Cynthia. 1994. *Tainted Truth: The Manipulation of Fact in
America*. New York, NY: Simon & Schuster. In this eye-opening book,
Crossen demonstrates how research has become a tool for hucksters to
make friends and influence people.

Weiss, Michael J. 1994. *Latitudes and Attitudes: An Atlas of American
Tastes, Trends, Politics, and Passions*. New York, NY: Little Brown and
Company. This book is a beautiful summary of consumer and market data.
It shows the power of comprehensive consumer surveys to describe
consumer preferences across the US.

> \"Everybody believes in public opinion polls, from the man in the
> street to President Thomas E. Dewey\"
>
> \--Goodman Ace, 1948\--[^49]

### Reading tables and graphs

I confess! When reading a new report, I go straight for the tables and
graphs. I do this for two reasons:

1\) I want to determine if I find the author credible, and

2\) I like to understand the bottom-line results.

These two goals are quickly served by rummaging through tables and
graphs for half an hour. I then read the paper to follow the author\'s
reasoning more closely.

If the tables and graphs are poorly designed or confusing, I lose
respect for the author. It is essential that tables and graphs
summarizing analysis results be clear, accurate, and well documented. If
they aren\'t, including them is worse than useless, because they hurt
your argument and your credibility.

First check for internal consistency. I always begin at the bottom-line
table and work backwards. I examine the column and row headings to be
sure I understand what each one represents (I read the footnotes if I
have questions). I then see that the components of the total add up to
the total. This procedure convinces me that the calculations are
accurate and it helps me become familiar with the various parts of the
analysis.

It\'s a good idea to look over the numbers in the table and identify
those that are abnormally small or large (LINK TO DIG INTO THE NUMBERS).
Typographical errors are quite common in tables, and a quick scan can
help you find them. For example, if you\'re reading a table summarizing
hours worked per week by different team members on a project, an entry
from one person that\'s ten times larger than for others should catch
your attention and prompt you to investigate further. The number itself
may not be wrong, but checking it will increase your confidence in the
numbers and help you understand how they were calculated.

Sometimes numbers don\'t add up exactly because of *rounding errors*,
not because there is a mistake in the calculations. Say you\'ve
formatted a spreadsheet table so that there are no decimal places for
the entries in the table. These entries might be 9.4 and 90.4, but in
the table they are shown as 9 and 90, because the convention is to round
numbers down to the next whole number when showing fewer decimal places.
The sum is 99.8, which rounds to 100 and is shown in the spreadsheet as
the total. The sum of 9 and 90 is 99, which makes the total of 100 look
wrong, even though there\'s a perfectly sensible explanation. If you\'re
not aware of this potential pitfall, you could be misled by these
apparent errors.

Read the footnotes assiduously. They should convey the logic of the
calculations and the sources of key input data. If you can\'t determine
the methods used from the footnotes, you should be especially suspicious
of the results and investigate further.

Take ratios of results in the tables and graphs and determine if the
relationships they embody make sense. If one component of the total is
growing much faster than the others over time, or if it is especially
large compared to others, then investigate further. Look for large
discrepancies and investigate when you uncover them.

The next step is to check consistency with independent external sources.
Does the information in the tables or graphs contradict other
information I know to be true? That table of hours worked may list Joe
as someone who worked little on the project at hand, but if you know for
a fact that Joe slaved over this project for many weeks because it was
his idea, then you\'ll need to check the calculation. Similarly, if
there\'s an entry of 175 hours for one week, it must be a typo or a
miscalculation, because there are only 168 hours in a week.

Compare numbers to other sources to make sure those in the tables and
graphs are roughly right. Do simple calculations to convert data you
have to a form that is comparable to that in the table. Take ratios of
numbers in the table and compare to ratios of external data. Make sure
the \"order of magnitude\" is right for the key input data. Compare
growth over time to growth in other key data like population and GDP, to
give you perspective on how fast the data of interest are growing
relative to these commonly-used indicators. (LINK TO DIG INTO THE
NUMBERS)

Follow up when you encounter cognitive dissonance\--any contradiction
between your knowledge and the information in the table will lead to
greater understanding, one way or the other. If there is a logical
explanation for the contradiction, you\'ve learned about the
relationships between the new knowledge in the table and what you knew
before. If the contradiction indicates a real problem, you\'ve
identified a flaw in the analysis. Seek out cognitive dissonance and
root out its causes, and you are guaranteed to become wiser and learn
more.

> \"Although we often hear that data speak for themselves, their voices
> can be soft and sly\".
>
> \--Frederick Mosteller, Stephen E. Fienberg, and Robert E. K.
> Rourke\--[^50]

### Distinguish facts from values

Every human choice embodies certain values. When technical people
advocate certain technology choices (e.g., whether or not to build more
nuclear power plants) they often portray their advice as totally
rational, completely objective, and value-free. This portrayal
***cannot*** be correct\--if someone makes a choice, they have also made
a value judgment. For example, Cohen[^51] portrays his support for a
revival of nuclear power as the result of scientific analysis, when in
fact it is a *public policy judgment* that is based both on his values
and his scientific understanding.

Robert Reich\'s public policy students often forgot this important
truth:

\"Some of my students used to regard public policy-making as a matter of
finding the \'right\' answer to a public problem. Politics was a set of
obstacles which had to be circumvented so the \'right\' answer could be
implemented. Policy was clean\--it could be done on a computer. Politics
was dirty\--unpredictable, passionate, sometimes mean spirited or
corrupt. Policy was good; politics, a necessary evil.\" (Reich 1997,
p.106)

The purpose of analysis is to support public or private choices, so by
its very nature it cannot be \'clean\' or value-free. It can, however,
illuminate the *consequences* of these choices so that the people and
institutions making them can judge between alternative outcomes.

When Fuchs et al. (1997) analyzed economist\'s positions on policy
issues, they found that those positions were strongly correlated with
each economist\'s values. Economists with a conservative political
ideology viewed policy issues one way, while those with a more liberal
bent generally viewed them in a different way. Marshall (1997)
summarized the results nicely:

What the authors found was that economists\' views on policy questions
correspond more closely to their values than to the \[facts\].

To some extent, in other words, mere facts don\'t change their mind on
issues. Keep that in mind the next time you see an ad, signed by 200
economists, supporting some candidate or piece of legislation. The ad
may tell you less about their expertise than their ideology.

I define facts as those assertions about the physical world that can be
verified through experiment. Events can also be facts, though the
interpretation of the meaning of those events is usually strongly
colored by our ideology (LINK) and attitudes. Values, on the other hand,
are explicitly subjective, and are an expression of those ideas and
feelings that are most important to us.

The relationship between facts and values can be mapped nicely on a two
by two matrix (see below).[^52] If people agree about both the facts and
the values related to a particular decision, it becomes merely an issue
of computation to reach an acceptable solution. Concurrence about values
but disagreement about facts is another easy choice: in this case,
experimentation is the order of the day. If the facts are not in
question but people\'s values are in conflict, then they can negotiate
to solve the problem. The most complicated and difficult choices are
those where both facts and values are in dispute. In such situations,
paralysis or chaos result. Most difficult public policy choices are
problems of this type, and the best approach here is to experiment and
bring the facts to a point where people no longer dispute them; people
can then start to negotiate about values.

In any of these discussions, it is a common rhetorical trick to link an
ostensibly demonstrable fact with a value judgment. Such statements are
of the form \"Fact A is true, therefore we should take Action B\". By
linking facts and values in this way, the speaker is explicitly or
implicitly hiding the value judgment she has made.

Let the reader beware: Fact A may or may not be true, and Action B may
or may not be a good idea whether Fact A is true or not. Too often these
glib statements are allowed to pass without proper scrutiny. By being
aware of this commonplace rhetorical technique, you can prepare yourself
to respond.

The purpose of good analysis is to help you illuminate the facts, so you
can use your values to make good choices. Value choices are inescapable,
which is why separating demonstrable facts from value judgments is so
essential. Do so and you are sure to be on a solid footing.

#### Further reading

Booth, Wayne C., Gregory G. Colomb and Joseph M. Williams. 1995. *The
Craft of Research*. Chicago, IL: The University of Chicago Press. This
superb book presents a framework for dissecting arguments that can be
applied in most situations. The authors were trained in rhetoric and
literature, but their way of thinking about claims and evidence applies
equally well to technical topics.

Fuchs, Victor R., Alan B. Krueger and James M. Poterba. 1997. *Why Do
Economists Disagree About Policy? The Roles of Beliefs About Parameters
and Values*. Princeton University, Industrial Relations Section. Working
Paper #389. August.

Hughes, William. 1997. *Critical Thinking: An Introduction to the Basic
Skills*. 2d Peterborough, Ontario: Broadview Press. On pages 108-110,
Hughes gives seven rules for assessing arguments that can help you
dissect and understand people\'s assertions when they don\'t separate
facts and values.

Jonas, Hans. 1973. \"Technology and Responsibility: Reflections on the
New Task of Ethics.\" *Social Research*. vol. 40, no. 1. p. 31. This
article, more than any other I read in graduate school, got me to think
carefully about how technology can challenge some of our most deeply
held values.

Marshall, Jonathan. 1997. \"Economists Can\'t Seem to Agree on
Anything\". *SF Chronicle*, August 25, 1997, p. B2.

######### 

######### How Society Chooses

![](media/image13.wmf){width="5.138888888888889in"
height="3.0416666666666665in"}

> \"The significant problems we face cannot be solved by the same level
> of thinking that created them.\"
>
> \--Albert Einstein\--[^53]

## Create your analysis

Now that you\'ve gleaned what you can from the analysis of others, it\'s
time to roll up your sleeves and get to work. The chapters in this
section each illustrate an important lesson I\'ve learned over the
years.

*Reflect*: Free yourself from interruptions and give yourself time to
reflect. Without such time, you\'ll never achieve your full
problem-solving potential.

*Getting unstuck*: Everyone gets stuck sometimes, but it need not hobble
your efforts if you use the tricks in this chapter.

*Inquire*: When faced with a problem outside your expertise, don\'t
surrender! It\'s an advantage to be unconstrained by the mental shackles
most disciplines place on their practitioners. Some of the most
important insights in modern thought come from people who could think
\"outside the box\".

*Be a detective*: Detectives are real-world practitioners of the
scientific method. The time-honored techniques of these seasoned problem
solvers can help you in your efforts.

*Create consistent comparisons*: People often relate best to anecdotes.
A consistent comparison is a well chosen set of two anecdotes that
illustrates your point in a compelling way. It is a powerful technique,
and one well worth learning.

*Tell a good story*: Scenario analysis is the art of structured
storytelling, and it\'s an essential tool for any good analyst. Most
people don\'t realize that this art is both well developed and pertinent
to many everyday situations.

*Dig into the numbers*: Don\'t be shy about delving into the actual
numbers, even if you\'re a highly paid executive. You\'ll learn things
you would never see if someone else crunches the numbers.

*Recycling old envelopes*: You can calculate almost anything using only
common knowledge\--you just need to learn how to put this knowledge to
use, and this chapter (which focuses on back-of-the-envelope
calculations) is just the thing to help you do it.

*Hear both sides*: In any intellectual dispute, it pays to hear two
well-prepared debaters argue their points before drawing any
conclusions. Always make such debates fodder for your deliberations and
your decisions will benefit.

*The value of models*: Models are \"laboratories for the imagination\",
and in this chapter we explore the subtleties of using them to explain
the world around you.

*Understanding forecasts*: The future is uncertain, but people keep
trying to forecast it anyway. Numerous pitfalls await, and without a
keen eye for the tricks of this trade, you\'ll be hard pressed to avoid
them.

*The uncertainty principle*: Just as the observer of a subatomic
particle can disturb that particle by the act of observation, the
observer of an institution can disturb that institution by observing and
reporting on it.

*Get real!*: People always think that *they\'ll* be the lucky ones, and
that they\'ll win in spite of impossible odds. While this feeling is
reinforced by the popular culture, it\'s a common misconception and one
well worth avoiding.

#### How to use this section

Each of these short chapters offers key insights to help you get
started. The order in which you read them is not so important, so first
choose the ones the interest you most, but do read them all before
you\'re through.

> \"A great many people think they are thinking when they are merely
> rearranging their prejudices\"
>
> \--William James\--

### Reflect

Any successful problem solver develops an array of techniques to spark
her creative process. Bryan Mattimore describes dozens of such
techniques in his book *99% Inspiration*, every single one of which
involves reflection on the nature of the problem and the possible
solutions. This reflection can be conscious, unconscious, intellectual,
intuitive, planned, unplanned, or all of the above. The key is to give
yourself time to reflect on the problem and let a solution bubble to the
top of your consciousness.

Reading books like Mattimore\'s can be helpful, but each person must
develop techniques that mesh best with their problem-solving style. When
I am faced with a particularly thorny issue, I go for a walk or a run,
which stirs my creative juices and removes me from other distractions.
You should do what works for you: many people listen to music, create
pottery, or clean house to attain the focus they need for true
reflection.

I begin my reflections by listing what I know about the problem and what
a satisfactory solution would be (without knowing the exact solution you
can define the kind of information that would solve the problem at
hand). I then try alternative approaches in my mind, analyzing each one
in terms of likely effort expended and probability of success,
considering the available data. I let my mind wander as I walk, which
often leads to important insights. By the end of my walk (assuming I
haven\'t been run over during my reveries) I have at least one possible
way to attack the problem.

It is a rare person who can think creatively in the face of constant
interruptions. I can\'t do it, and I suspect most others can\'t either.
Put aside some time every day to reflect a bit. Turn off the phone,
close the door, and put up a \"Do not disturb\" sign. Schedule this
reflection period during your particularly productive times of day (LINK
TO \"VALUE YOUR TIME\" section).

Brenda Ueland (1987) believes that creativity only bursts forth when we
are idle and thinking on our own, uninterrupted. The constant busyness
we often experience in modern life just gets in the way of our own
creativity. That\'s why she advocates at least half an hour per day of
this sort of idleness.

One martial artist (Bronislaw Kaper) takes this idea to its logical
extreme and consciously schedules entire days to do nothing (Hyams
1982). By taking time to reflect, you too can use such pauses to your
advantage.

#### Further reading

Hyams, Joe. 1982. *Zen in the Martial Arts*. New York, NY: Bantam Books.

Mattimore, Bryan W. 1994. *99% Inspiration: Tips, Tales & Techniques for
Liberating Your Business Creativity*. New York, NY: AMACOM (A Division
of American Management Association).

Ueland, Brenda. 1987. *If You Want to Write: A Book About Art,
Independence, and Spirit*. St. Paul, Minnesota: Graywolf Press.

> \"Music is the space between the notes.\"
>
> \--Claude Debussy\--

###   {#section-4 .unnumbered}

### Getting unstuck

Sometimes, try as I might, I just can\'t push past a seemingly
insurmountable obstacle. The problem itself may be particularly thorny,
or it may pose personal challenges. Any successful problem solver
develops tricks for \"getting unstuck\" in the face of such obstacles.

One of my most successful tricks is to break my normal routine. Some of
my best insights have come on airplanes, where I have time to reflect
(LINK) and few of my normal distractions. Go for a walk or a run, ride
your bike, or go on a trip. Call a friend you haven\'t seen in awhile.
Do something out of the ordinary and forget about the problem for a few
hours. Brilliant insights often occur when you\'re relaxed and not
thinking at all.

You can also try working on a related problem or one that\'s entirely
different. Either way, you\'ll be able to start fresh with the more
vexing problem after you\'ve attacked something a bit more tractable.

An artist I met once described how, once he had an idea for a painting,
he would start on the canvas right away. Once he got \"inside the
picture\", the difficulties would often just melt away. The same
technique works for other kinds of problem solving. If a problem remains
daunting, ask if there\'s some small question that you can answer (jump
\"into the picture\"). Answering that question often results in a bunch
of obvious follow-on questions that can also be answered, and answering
these usually puts you on the road to a solution.

A time-tested technique for breaking logjams is to describe the problem
to someone else. One friend recalled for me how her mother helped her
solve math problems in junior high school. \"Mom couldn\'t remember the
details about specific math techniques, but suggested that I explain the
problem to her. By the time I had given her the details, I had virtually
always had a new insight for a solution. Explaining the problem forced
me to think clearly\".[^54]

There is a series of questions that flow through my mind whenever I\'m
faced with a problem unlike any I\'ve seen before. These questions,
largely based on Polya\'s work, are as follows:

\(1\) What is the unknown (i.e., the number I am trying to calculate)?

\(2\) What data are given, and which can be estimated using knowledge I
already have? Are the given data correct based on other knowledge I
have? Are there any additional data I can bring to bear on the problem?

\(3\) Can I restate the problem?

\(4\) Can I draw a picture or graph framing the problem?

\(5\) Is there another similar problem I have already solved that could
give me the key to a solution? Can I solve a related problem?

\(6\) Can I make some simple assumptions that might help me solve the
problem more easily? Can I solve just one part of the problem? Can I
solve a simplified version of the problem, using round numbers that are
roughly right?

\(7\) Are there any answers that I know are wrong? (sometimes you can
get closer to an answer by eliminating those that cannot be correct).

The study of problem solving, which Polya calls \"heuristic\", is still
a nascent art. Each obstacle is an opportunity to create and refine your
personal heuristic. Examine your own problem solving style, and create
techniques for getting unstuck that work for you.

#### Further reading

Elffers, Joost. 1997. *Play with Your Food*. Stewart, Tabori, Chang: New
York, NY. This book encourages you to think about food in a way you
never have before. If you really get stuck, get this book and play until
you\'re ready to get back to the problem.

Mattimore, Bryan W. 1994. *99% Inspiration: Tips, Tales & Techniques for
Liberating Your Business Creativity*. New York, NY: AMACOM (A Division
of American Management Association). This book is a treasure trove of
ideas for becoming more productive.

Polya, G. *How to Solve It: A New Aspect of Mathematical Method*. 2d
ed., New York, NY: Doubleday & Company, Inc., 1957. A serious but
approachable book about problem solving. It\'s a real classic!

FIGURE: Brer Rabbit and the Tar Baby!

> \"Experience is that marvelous thing that enables you to recognize a
> mistake when you make it again.\"
>
> \--F. P. Jones\--

### Inquire

Quick\--your boss just assigned you a task that is outside your area of
expertise. What do you do?

I am often faced with this problem, as are most analysts in business,
academia, and government. My first move is always to pick up the phone.
I have a group of friends with a wide variety of interests, and usually
there is at least one with direct knowledge of a particular topic. By
asking these people focused questions, I can quickly determine what is
possible within the time allotted. I often develop these questions as
part of my effort to \"Get Unstuck\" \[LINK\]).

Don\'t be afraid to approach experts with questions. They are often
delighted that someone is taking an intelligent interest in their work,
and will usually bend over backwards to explain why their results are
useful and important. The reason many people become experts is because
of a passionate interest or concern. They *care* about their topic, and
want other people to care, too. So go ahead and call them up. Your
interest will make them happy, and you might just learn something along
the way!

If you don\'t know anyone who knows about the topic, try the internet.
Use one of the many search engines available for this task (try several,
because they are all virtually guaranteed to give different results). I
like to use the summary search engine at the LBNL web site
(http://www.lbl.gov/Glimpse/search-all.html), but there are others. Some
search engines (e.g., http://www.askjeeves.com) allow you to ask
questions in plain English and receive focused results that number ten
or twenty links (instead of the thousands you normally get from such
searches). To hone your internet research skills, consult some of the
recent work that teaches these skills (e.g., Barrett 1997).

One important tip in internet research is to consult the Frequently
Asked Questions (FAQs) that exist on the internet for a truly amazing
number of topics. FAQs are compiled by experts and are often a superb
place to begin your research. Use one of the many excellent internet
search engines to find FAQs that are relevant to your inquiry (see
Barrett 1997 for details).

The web site http://www.amazon.com is a surprisingly good place to do
research. The main purpose of this site is to sell books, but its search
routines are easy to use, and the site presents for each book a list of
other books commonly purchased by people who bought the featured book.
It also allows you to see other books by the same author with one click
of the mouse. There are about two million books in the Amazon.com
database, which is more than enough to cover most topics thoroughly. I
spent an instructive couple of hours exploring the site early one
morning, and I came away impressed.

Another place to begin is with a major reference work or a good
encyclopedia. You can find something related to virtually any query in
The Statistical Abstract of the U.S. or in the Encyclopedia Brittanica.
The general information contained in these works can often be enough to
get you started in untangling even the thorniest research problem.

If you are still stymied, talk to the reference librarian at the local
library. Librarians have an uncanny knack for tackling just this sort of
inquiry, and they are highly skilled professionals. At a minimum, the
librarian should be able to point you to the standard reference work in
any area of knowledge, as well as the relevant trade journals (which can
be treasure troves of information about real world markets and
technologies).

The stereotype of librarians is that they are pathologically shy and
retiring. This generalization is probably less accurate than most, but
even if true, it denigrates the crucial role that librarians play in
research. The importance of these professionals was brought to the fore
when I read a reporter\'s tale of searching for information in the
spanking new main library in the city of San Francisco.[^55] The
reporter tried searching for books on a probably mythical Christian king
who was the impetus for several important missions of exploration during
medieval times.

While his topic may seem obscure, the reporter\'s experience will ring
familiar to most. He tried first to use the library\'s electronic card
catalog and came up with two listings on his subject, one which seemed
irrelevant. He then logged onto the library\'s on-line database using
his modem and found seven new listings that didn\'t show up in the first
search. Then he found another new listing by changing the order of his
search terms from \"Prester John\" to \"John, Prester\". Finally, he
consulted the old card catalog that the computer terminals had largely
displaced, and found an additional reference (a book that was so old
that it hadn\'t been entered into the computerized database and was not
kept in the main library but in a storage annex).

Without the help of a librarian, this reporter might have missed some
key sources. Only a librarian could tell him the tricks that would
reveal the sources he sought in an expeditious manner. Trial and error
can yield results, but your search will be far quicker if you consult a
librarian first.

Work outside your area of expertise is difficult, but can be rewarding.
As we know from the history of science, many of the greatest discoveries
were made by scientists who were formally trained in another discipline.
Developing the skills to explore new fields will serve you well in any
endeavor you undertake.

#### Further reading

Daniel J. Barrett. 1997. *NetResearch: Finding Information Online*.
O\'Reilly and Associates. (for a summary go to
http://www.ora.com/catalog/netresearch/ and for the latest updates to
the book as well as related links, see
http://www.cs.umass.edu/\~barrett/netresearch/)

Berkman, Robert I. 1990. *Find It Fast: How to Uncover Expert
Information In Any Subject*.. New York, NY: Harper & Rowe, Publishers.
One researcher\'s tricks to digging out information in the real world.

U.S. Bureau of the Census. 1997. *The Statistical Abstract of the United
States 1997* 117th edition Washington D.C.: U.S. Government Printing
Office. One of the most important data sources for any U.S. researcher.

Edwards, Paul, ed. 1972. *The Encyclopedia of Philosophy*. New York, NY:
Macmillan Publishing Co., Inc. & the Free Press. Volumes 1 through 8.
This set is an essential desk reference that describes the historical
development of ideas. It can lend context to your research available
nowhere else.

*The Encyclopedia Brittanica*. The hard copy of this encyclopedia costs
more than \$1000, but the electronic versions are a much better deal. A
multimedia CD-ROM version costs about \$125 (for both Windows and
Macintosh), and a subscription-based web version (http://www.eb.com)
costs \$8.50/month or \$85/year. Other encyclopedias offer similar
deals.

######### Sometimes you can find experts when you least expect it\...

![](media/image14.wmf){width="5.666666666666667in"
height="2.9583333333333335in"}

Need to get permission to reproduce this cartoon and pay any royalty
due.[^56]

> Minds think with ideas, not information. No amount of data, bandwith,
> or processing power can substitute for inspired thought.
>
> \--Clifford Stoll\--[^57]

### Be a detective

One of the defining features of the character Sherlock Holmes is his
scientific approach to solving mysteries. He had an immense store of
knowledge of past criminal activity upon which he could rely. He
systematically collected information from the scene of the crime, and
noticed the most ostensibly trivial details that might bear upon his
case. He did not prejudge his case in advance of the facts. Finally, he
was skilled in weaving alternative theories that might fit the facts.

Holmes\' approach is not unlike that required to solve problems in the
modern world. Any problem solver worth her salt knows as much as
possible about data and previous work on the problem at hand (if you\'re
new to it, then go the library and read). Attention to detail is also
essential as is not allowing your initial snap judgments to color your
interpretation of parts of the puzzle. And weaving alternative theories
to fit the facts is perhaps the most important skill of a seasoned
problem solver.

Spreadsheets[^58] are my favorite problem solving tool because they
allow me to easily and quickly create many different scenario
calculations. Business people use them to create financial projects for
their business plans. Scientists use them to analyze data from their
experiments. Policy analysts use them to project the effects of proposed
government policies on people\'s lives and the economy as a whole. In
each case, spreadsheets enable both quick scoping calculations and
extremely detailed analysis.

Sometimes in creating a spreadsheet I discover an intermediate or final
result that is either internally inconsistent with another such result
from the same calculations, or cannot be right based on first
principles. An example of the first kind of inconsistency would be to
have a business plan that showed operating losses for the first five
years of the forecast but that showed no expenditures of startup capital
after five years. An example of the latter inconsistency would be to
project that a small startup company would gain 99% market share within
a few years and attract no competitors.

When I discover such inconsistencies in my spreadsheets, Holmes\'
technique of systematically investigating alternative theories comes
into play. I begin by making a list of possible reasons why the
calculation is in error. I then trace the flow of data through the
calculations, starting from the point that I noticed the inconsistency.
I look at each and every input to a particular calculation to see if
I\'ve made any obvious errors in entering formulas or data, tracing all
the way back to the beginning. If, as is commonly the case with
complicated calculations, I still can\'t figure out what\'s wrong, I
will recreate my calculation in a new spreadsheet, usually using a
simplified version of the calculation and plugging in data that are
roughly right (and that are usually nice round numbers ). This approach
allows me to begin with a clean slate. Often when I am immersed in
calculations I forget one small detail. Starting fresh in this way helps
me catch those little errors.

One quick check you can do on your spreadsheet calculations is to vary a
key input number, and see if the results change as you would expect them
to change. If you increase that number by 50%, do you expect the result
to increase or decrease by a comparable amount, or by half that amount?
If your results vary in an unexpected way or in unusual proportions,
either your calculation is wrong or your conceptualization of the
problem is flawed in some way. Either way, you\'ve learned something.

If two complex calculations should yield the same result but do not,
break down each calculation into its component parts, and compare each
part of the first calculation to the corresponding one in the second.
This technique will make it obvious in which part of the calculations
any differences lay. For example, assume that a formula in a spreadsheet
has three terms that add to a total (Total = A + B + C) and that an
independent calculation has three terms that should also add to that
total (Total = X + Y + Z). Assume also that the value of each term in
the first calculation should correspond to the value of each term in the
second calculation (A should equal X, for example). If the result of the
two calculations is different, then compare A to X, B to Y, C to Z, and
see if the differences in the overall total can be explained by
differences between any of the individual components. You can usually
narrow down the discrepancy to one or more of the terms in the equation,
and eliminate some possibilities. While tedious, this technique can help
you identify the potential explanations for such discrepancies. It is
especially useful when each of the terms is in itself a complex
calculation.

When working with spreadsheets, many people copy and paste data from one
sheet into another. This practice is a pernicious one, because it makes
it difficult to update your calculations later, and nearly impossible to
figure out how the calculations flow once they are no longer fresh in
your mind. Obey this golden rule: *never paste values from one
spreadsheet into another*. Link spreadsheets together directly, so that
when you change your calculations in the first spreadsheet, the other
one is automatically updated. Following this rule helps you track down
errors in the calculation much more quickly. It also saves you time if
you have to change input assumptions later. LINK TO DOCUMENT SECTION.

If your results look too good to be true, they probably are. Be
suspicious of your own calculations when you first create them. Dig into
the numbers (LINK) to determine whether you can trust them.

Another common technique used by detectives is to consider the motives
of the person or institution putting forth a particular viewpoint. I
suspect, for example, that a lobbyist for the tobacco industry might be
prone to overlook evidence that smoking is harmful. Seek out sources
whose interests coincide with the truth, and discount somewhat the
opinions of those with a vested interest. However, remember that having
an interest or an opinion is not evidence of being biased any more than
*not* having an opinion is evidence of being unbiased: some people \[a
rare few\] are honest even to their own detriment.

#### Further reading

Sir Arthur Conan Doyle. *A Study in Scarlet*. This novel, in which Conan
Doyle created the modern detective, first brought Sherlock Holmes into
English literature. It\'s a gripping, well told story, and one in which
the scientific method features prominently.

> \"When you have eliminated the impossible, whatever remains, however
> improbable, must be the truth\"
>
> \-- Sherlock Holmes \--

### Create consistent comparisons

In the real world, so many things change at once that it is often
difficult to decipher cause and effect. One of the most compelling
techniques for making a point in the face of this complexity is to find
or create a consistent comparison that eliminates the complicating
factors and illustrates your point in an unambiguous way. A pure
consistent comparison is one where the only feature differing between
two examples is the one that illustrates your point. Even when there are
some minor differences, you can often correct for these and still create
a powerful comparison.

Isolating the root cause by creating an experiment is one of the most
fundamental techniques of modern science. When medical researchers
create \"control groups\" to compare to people being given an
experimental drug, they are creating a consistent comparison, or as near
to one as is humanly possible. While you can\'t always create such an
experiment, you can often find one lying around if you look carefully. I
discovered such a case in the early 1990s, which led me to write one of
my favorite papers.

Since the 1970s, U.S. automakers had insisted that they were unable to
increase the fuel efficiency of passenger cars in a cost-effective way.
In 1992, Honda introduced a new Civic hatchback that was the same size
and had the same engine power as the model it replaced, but it was 56%
more fuel efficient than the previous year\'s model. I read news
articles about this new car and badgered various friends who work on
transportation issues to investigate this case further. After several
months of fruitless nagging, I decided to explore it myself.

I found that the two cars differed slightly in features, but were
essentially identical except for their fuel economy. The case was
therefore a good one to use to estimate the actual retail costs of
improving fuel economy in the real world. After correcting for the costs
of the different features (based on dealer cost and list prices), I
found that the more efficient Civic (the 1992 VX) cost about \$700 more
than the 1991 Civic DX, and saved about 110 gallons of gasoline per year
for typical drivers. The added cost would pay for itself in about five
years at current fuel prices, making it a cost-effective investment.

The example was an exceptionally powerful one because the size and
engine power of the two vehicles were identical, and the feature list
nearly so. As a check on our calculation of the retail cost, we also
investigated the projected retail cost of the technologies used to
improve the VX\'s fuel economy, based on *independent engineering
estimates*, and found that these estimates were quite close to our
calculation of the actual retail cost difference between the 1991 DX and
the 1992 VX. This independent check gave us confidence that we were on
the right track.

I eventually purchased a 1993 Civic VX, and it has performed flawlessly.
It gets 50 miles per gallon on the freeway, yet holds four passengers
comfortably. It\'s not a luxury vehicle by any means, but it\'s a
testament to what automakers can do if they put their minds to it.

The paper documenting this case subsequently won a prize from the
National Research Council\'s Transportation Research Board \"for
excellence in transportation research\". The prize is an indication of
the compelling nature of a well-chosen consistent comparison.

#### Further reading

Koomey, Jonathan G., Deborah E. Schechter and Deborah Gordon. 1993.
*Cost-Effectiveness of Fuel Economy Improvements in 1992 Honda Civic
Hatchbacks*. Energy and Resources Group, University of California,
Berkeley. Presented at the 72nd Annual meeting of the National Research
Council\'s Transportation Research Board, January 10-14, 1993,
Washington, DC. May 1993.

> \"The cure for boredom is curiosity. There is no cure for curiosity.\"
>
> \--Dorothy Parker\--[^59]

### Tell a good story

Life is about making choices in the face of uncertainty. One
oft-neglected approach to such decisionmaking is the art that Peter
Schwartz calls scenario building. The purpose of scenario analysis, as
explained by Schwartz in his now classic book, *The Art of the Long
View,* is to explore several possible futures in a systematic way. No
matter how things turn out, the analyst will have \"on the shelf\" a
scenario (or story) that resembles that future and have thought through
in advance the likely consequences of that scenario on the decisions at
hand. Such a story \"resonates in some ways with what \[people\] already
know, and leads them from that resonance to reperceive the world\".

Schwartz describes eight steps for successful scenario development:

1\) *Identify focal issue or decision*: Scenarios should be developed in
advance of some major strategic decision, so that when the time to
decide arrives, the options are laid out in detail and the implications
of each decision path are made manifest. Such a decision might be
\"should our company expand into new markets next year, or should we
focus on expanding our market share in our current markets?\"

2\) *Key forces in the local environment*: The next step is to identify
the key factors in the immediate locality that will determine whether
the focal decision succeeds or fails. These factors might include such
things as customer acceptance of a new technology or competitor response
to the new strategy.

3\) *Driving forces*: This step involves identifying the forces in the
larger society that affect the key local forces in step 2. These forces
could include demographics, inflation, commodity prices, or political
instability. Some of them are beyond the control of any individual or
institution (predetermined elements), and others are subject to human
influence.

4\) *Rank by importance and uncertainty*: Of the trends and driving
forces identified in steps 2 and 3, a small number of them will be both
highly important and highly uncertain (see also Morgan and Henrion
1992). These are the issues upon which differences in the scenarios will
be based. Predetermined elements don\'t differ between scenarios.

5\) *Selecting scenario logics*: The most important and uncertain
factors should then be used to define the set of scenarios to be
considered. In essence, these factors are the dimensions of uncertainty
that the scenarios will explore. Schwartz gives the example here of an
automobile company that decides that fuel prices and protectionism are
the two main drivers of their company\'s future, the scenarios they
consider would be: \"(1) high fuel prices in a protectionist
environment--where domestic suppliers of small cars will have an
advantage; (2) high fuel prices in a global economy--where
fuel-efficient imports may capture the low end of the market; (3) low
fuel prices in a protectionist environment--where American gas guzzlers
will have a good market at home but not abroad; (4) low fuel prices in a
global economy--where there will be intense global competition for
fuel-efficient models, but larger cars may enjoy strong foreign
markets.\"

6\) *Fleshing out the scenarios*: Each of the key driving forces and
factors from steps 2 and 3 should be assessed for each of the main
scenarios. Usually, these factors can be assigned relative rankings
(i.e., \"Inflation should be higher than average in scenario 1, and
lower in scenario 2\"), which can then be translated into actual
numbers. The exact numbers are not so important, as long as they reflect
in a credible way people\'s expectations about how the future might
unfold in a particular scenario. The scenario must then be woven into a
plausible story describing a course of events that would result in the
scenario\'s final outcome. Each scenario must be named in a way that
vividly reminds people of its essence.

7\) *Implications*: Revisit the decision identified in Step 1, and
assess how that decision would fare in each of the different scenarios.
If the decision (e.g., expanding into new markets) would lead to success
(as defined in Step 2) for all or most of the scenarios considered, that
decision is probably a wise one. If it is likely to be successful in
only one scenario, the decision is \"a high-risk gamble\...especially if
the company has little control over the likelihood of the required
scenario coming to pass\".

8\) *Selection of Leading Indicators and Signposts*: Once the scenarios
have been developed, it is crucial to identify a few key indicators by
which decisionmakers can monitor the evolution of events. These
indicators are used to determine which of the different scenarios best
matches up with actual events, and can be used to trigger an
institutional response (\"if inflation gets up to 7% and the stock
market goes down 15%, then do X\"). If you can\'t measure it, you can\'t
manage it, so defining appropriate indicators is crucial to success.

Schwartz believes that scenarios are successful when \"they are both
plausible and surprising; when they have the power to break old
stereotypes; and when the makers assume ownership of them and put them
to work\". The institutional process of creating scenarios (what
Schwartz calls \"holding a strategic conversation\") can also reveal new
opportunities for an organization and promote team building among its
employees .

Another name for scenario analysis is the commonly used scientific
technique of creating a \"thought experiment\". All scientists use this
technique, in which they imagine certain real world complexities to be
absent, and then think through the consequences for the situation they
are investigating. For example, Galileo imagined a world in which the
complicating effects of friction did not exist, and was able to make
great strides in what would come to be called the field of kinetics (the
study of moving objects). Such thought experiments are as valuable in
business as in science. Instead of limiting your imagination by current
circumstances and constraints, imagine a world in which some key
constraints don\'t exist, and ask yourself how you or your institution
might respond.

Most of Schwartz\'s examples of scenario analysis have only a small
quantitative component, but many other futurists are obsessed with
numbers and computer models. Most explorations of the future with which
I am directly familiar err by focusing too much on the mechanics of
quantitative analysis (e.g. on particular modeling tools) and far too
little on careful scenario development based on the procedures described
above. Quantitative analysis can lend coherence and credence to scenario
exercises by elaborating on consequences of future events, but modeling
tools should support that process and not drive it, as is so often the
case.

In my years of involvement with development of national energy policy, I
have been most struck by how few resources are devoted to sensible
scenario development and associated data, and how much to the
development of different modeling tools to assess such policies.
Computer tools are sexy and appealing (at least to the funding
agencies). Data and scenario analysis, upon which the results generally
hinge, are virtually always given short shrift.

Tens of millions of dollars are spent every year on models whose
capabilities are redundant with others, usually because a particular
agency with money wants its own \"in-house\" modeling capability and is
unwilling because of institutional rivalry or personal biases to adopt
one of the pre-existing frameworks. The policy makers fail to realize
that models are ALL unable to predict the future in any systematic way,
and that small improvements in modeling methodology are made irrelevant
by inadequate scenario development.

In our analysis of the costs of reducing carbon emissions in the
European electricity sector, we avoid these failings of previous
analyses.[^60] We calculate carbon emissions and total costs of
delivering electricity services, using a consistent methodology for all
scenarios. Our analysis explores several dimensions of potential
uncertainty, including fuel prices, technology costs, effectiveness of
policy implementation, and whether nuclear power plants retire or not
(each of these uncertain elements are assigned a low-high range).

We combine these different elements of uncertainty in ways that tell
coherent stories. For example, we show the results for a case where
nuclear power costs are low while costs for all other resources are
high. This case is the most optimistic one for nuclear power, and it
shows that even in that case, nuclear power will not achieve the levels
of output projected by its advocates. This kind of \"bounding scenario
analysis\" is a powerful way to explore the future quantitatively. Other
scenarios explore how the electricity supply system might evolve in the
face of deregulation, how renewable supply side resources might fare in
such a transition, and how much carbon savings can be achieved from an
\"energy efficiency only\" strategy. In all cases, we pair our
quantitative analysis with a story, and we identify who might believe
that story.

Our analysis differs slightly in character from the scenario analyses
described by Schwartz, because it was developed to influence *public*
decision making on energy policy, not to serve a particular *private*
institutional need. Our comprehensive analysis and presentation of
results allows different people and institutions to read their own world
view into our analysis. This kind of presentation increases the
usefulness and transferability of the conclusions, and if you can create
such an analysis, you will have vastly more influence.

#### Further reading

Morgan, M. Granger, and Max Henrion. 1992. *Uncertainty: A Guide to
Dealing with Uncertainty in Quantitative Risk and Policy Analysis*. New
York, NY: Cambridge University Press.

Schwartz, Peter. 1996. *The Art of the Long View: Planning for the
Future in an Uncertain World*. New York, NY: Doubleday.

FIGURE: PICTURE OF AN OLDER PERSON TELLING STORIES TO YOUNGER FOLKS
GATHERED AROUND.

> \"Expect nothing, be ready for anything.\"
>
> \--Anonymous\--

### Dig into the numbers

Anyone who has delved into data from the real world knows that they\'re
messy. Survey takers write down the wrong response. People entering data
into the computer from the survey forms type in the wrong data.
Computers sometimes garble data because of software bugs.[^61] Data
formats become obsolete as software changes. Data loggers become
uncalibrated or break. Analysts mislabel units and make calculational
errors.

\"Cleaning the data\" to correct for such problems is virtually always
necessary, but it is an often ignored step. Always ask about the process
used to clean the data: if the response is just a blank stare, or worse,
you know you\'re in trouble. Some companies now assign people to check
for data quality in the face of all the problems associated with real
data.[^62]

One of the more famous examples of how small mistakes in data processing
can have disastrous results is that of the Mariner I spacecraft. The
rocket went off course soon after launch and had to be destroyed. The
cause of the malfunction was a missing hyphen in a single line of
Fortran code. Arthur C. Clarke quipped later that this \$80 million
launch was ruined by \"the most expensive hyphen in history\".[^63]

Bad data can have a real cost for a business. If a 500,000 piece mailing
use an address list with an error rate of 20%, the company immediately
wastes \$300,000 in sending mailings to incorrect addresses. It then
loses more money because among those 100,000 missed prospects are about
1500 people who would have become customers and purchased thousands of
dollars of the company\'s products over their lifetime. The losses from
these missed customers can be many times greater than the immediate
direct losses.[^64]

It is crucial to pore over raw survey data to check for anomalies before
doing extensive analyses. For example, typographical errors can lead
numbers to be ten, or hundred, or a million times bigger than they
should be. Looking over the raw data can help you identify such problems
before you waste time doing statistical analysis using incorrect
numbers.

#### Some specific advice

If you have data to be analyzed, type them into the spreadsheet
yourself, assuming there aren\'t many hundreds of them. There\'s no
substitute for this effort, even if you are a highly paid executive,
because it will help you identify inconsistencies and problems with the
data and give you ideas for how to interpret them.

Check that the main totals are the sum of the subtotals. Most documents
are rife with typos and incorrect calculations. You should therefore not
rely blindly on any data source\'s summations, but calculate them from
the base data. You can check your typing accuracy by comparing the sums
to those in the source of data. If they match exactly, it is unlikely
that your typing is in error.

Check relationships between numbers that should be related in a
predictable way. Unexpected results of such comparisons can teach
valuable lessons. For example, if examining data on carbon emissions of
different countries, a newcomer to the field of greenhouse gas emissions
analysis might expect that the amount of carbon emitted per person would
not differ much between industrialized countries. In examining such
data, however, we find large differences in carbon emitted per person,
from less than 1 metric ton/person/yr in Portugal to more than 6
tons/person/yr in Luxembourg. Determining why such differences exist is
the logical next step, which will inexorably lead to further analysis
and understanding.

You should be able to trace someone else\'s calculation through in a
logical way. If you can\'t, you can at least begin to list the questions
you need to answer to start tracing the calculation. Ultimately, if you
can\'t reproduce their calculation, the author has failed a fundamental
rule of good data presentation and their analysis is suspect.

Compare the numbers to something else with which you are familiar, as a
\"first-order\" sanity check. Normalize numbers to make these
comparisons easier. For example, total U.S. emissions of carbon dioxide
in millions of tons per year are difficult to grasp for most people, but
if normalized to number of kilograms per person per year will be a bit
more understandable. Another common base for such normalizations is
economic activity (per dollar of Gross National Product \[GNP\]) or
physical units related to the emissions (such as per kilowatt-hour
\[kWh\] of electricity generated or per kilometer driven).

If you have data that extend over time (\"time series data\"), normalize
them to a base year to enhance comparisons. By expressing such data as
an index (e.g., 1940 = 1.0) you can compare trends to those of other
data that might be related. For example, if you plot U.S. raw steel
production (Figure a), population (Figure b), and GNP (Figure c) in
separate graphs, it is difficult to gain perspective on how fast steel
production is changing over time relative to these other two important
determinants of economic and social activity. However, if you plot steel
production over time as an index with 1940 = 1.0 (see Figure d), you can
plot population and GNP on the same graph. Such a graph will show
instantly whether growth rates in these data differ.

In this example, real GNP grows by a factor of more than five from 1940
to 1990. Steel production roughly doubles by 1970, and then declines by
1990 to roughly 50% above 1940 levels, while population just about
doubles over 1940 levels by 1990.

The trends for steel production in Figure d illustrate dramatically the
changing fortunes of steel in a post-industrial economy. Just after
World War II, steel use per capita increased as automobile ownership
expanded and use of steel for bridges and other forms of construction
also increased. After 1970, the steel industry began to face serious
competition from foreign steel makers as well as from alternative
materials like aluminum alloys and composites, so production declined
even as population increased and real GNP went through the roof.

######### Figure a: U.S. production of raw steel 1940-1990 (million short tons)[^65]

######### ![](media/image15.pct){width="443.0in" height="137.0in"}

######### Figure b: U.S. population 1940-1990 (million people)[^66]

![](media/image16.pct){width="440.0in" height="101.0in"}

######### Figure c: U.S. gross national product 1940-1990 (Billion 1982 dollars)[^67]

![](media/image17.pct){width="443.0in" height="117.0in"}

######### Figure d: U.S. production of raw steel, population, and gross national product 1940-1990, expressed relative to 1940 levels (1940 = 1.0)![](media/image18.pct){width="441.0in" height="276.0in"}

#### Break problems into component parts

In 1986, a team of analysts with whom I was working had just finished
two reports assessing the potential future effects of proposed
regulations on the minimum efficiency of residential appliances. My boss
needed to summarize the analysis results for our sponsors, and he asked
me to help him compare the results of the two studies.

The method I developed to illustrate the main causes of differences
between the results of the two studies has more general application, so
I summarize the calculation below. The main results were the cumulative
dollar benefits expected from the regulations. They are illustrated in
Table 1:

######### Table 1: Comparison of key results from the two studies

+-----------------+--------------+-----------+-----------+-----------+
|                 |              | *Utility  | *Utility  | *Ratio*   |
|                 |              | \# 1*     | \# 2*     |           |
+-----------------+--------------+-----------+-----------+-----------+
|                 |              | *Texas    | *Nevada   | *         |
|                 |              | Utilities | Power     | TUEC/NPC* |
|                 |              | Electric  | Company*  |           |
|                 |              | Company   |           |           |
|                 |              | (TUEC)*   | *(NPC)*   |           |
+-----------------+--------------+-----------+-----------+-----------+
|                 |              |           |           |           |
+-----------------+--------------+-----------+-----------+-----------+
| Cumulative net  | million 1985 | 65        | 37        | *1.8*     |
| benefits        | \$           |           |           |           |
+-----------------+--------------+-----------+-----------+-----------+
|                 |              |           |           |           |
+-----------------+--------------+-----------+-----------+-----------+
| \# customers    | millions     | 1.07      | 0.31      | *3.4*     |
| 1996            |              |           |           |           |
+-----------------+--------------+-----------+-----------+-----------+
| kWh             | kWh/c        | 67.5      | 68.3      | *1.0*     |
| s               | ustomer/year |           |           |           |
| avings/customer |              |           |           |           |
+-----------------+--------------+-----------+-----------+-----------+
| Net benefits    | \$/kWh       | 0.08      | 0.16      | *0.5*     |
+-----------------+--------------+-----------+-----------+-----------+
|                 |              |           |           |           |
+-----------------+--------------+-----------+-----------+-----------+

\(1\) Savings are from minimum appliance efficiency standards
implemented in 1987, projected using computer models of residential
energy use and utility power plant economics.

\(2\) Net benefits = Gross benefits minus costs to society, summed over
time.[^68]

The puzzle Mark wanted to solve was that the cumulative net benefits for
the Texas Utilities Electric Company (TUEC) were only 1.8 times larger
than those for the Nevada Power Company (NPC), even though TUEC was 3.4
times larger as measured by the number of customers. There were two
possible sources of this difference: one is that the average kWh savings
per customer were different between the two utilities (because appliance
ownership and climate differed), the other is that the net benefits per
kilowatt-hour (kWh) were different. This inference follows from the
basic equation we use to estimate the total cumulative savings for each
utility:

Total net benefits (\$) =# customers x x

I investigated both these possibilities, and discovered that kWh savings
per customer were virtually identical for the two utilities (68 kWh per
customer per year), while net benefits per kWh saved were only about
half as much for TUEC as for NPC.

The key factor was therefore this difference in net benefits per kWh,
and these calculations led me to investigate this parameter further. The
net benefits per kWh equal gross benefits minus costs to society. The
costs to society per kWh did not vary much between the two utilities,
but the gross benefits per kWh did vary substantially. This was the
cause of our puzzlement, and I was able to pinpoint it using these
relatively simple calculations after our detailed analysis was
completed.

Dissecting analysis results by comparing ratios of key parameters is a
powerful approach, and one that I recommend to your attention. Any time
you have two comparable studies that give results that vary
unexpectedly, this kind of \"ratio analysis\" can lead to important
insights into the causes of differences between the two studies.

Creating a presentation or an executive summary for a complex report
always involves this kind of analysis. First create an equation like the
one above that calculates a key analysis result as the product of
several inputs multiplied together, then determine which of these inputs
affects the results most significantly. Creating such an equation can
help you think systematically about your results.

#### Tricks of the trade

When John Holdren was a professor at UC Berkeley, he taught a delightful
class titled \"Tricks of the Trade\". In this class he described many of
the unwritten rules about being effective in the energy/environment
field. In my notes for that class (still useful after all these years),
Holdren lists key pitfalls in data acquisition and handling. I aggregate
them below into three golden rules :

• Avoid data that are mislabeled, ambiguous, badly documented, or
otherwise of unclear pedigree.

• Avoid unreliable data that are invented, cooked, or incompetently
created. Beware of illusory precision.

• Avoid spurious comparability. Beware of numbers that are ostensibly
comparable but are fundamentally inconsistent. Create and use only
consistent comparisons (LINK).

Holdren\'s advice when dealing with data is to \"be suspicious,
skeptical, and cynical. Assume nothing\". Though it may sound paranoid
to the uninitiated, such caution is an absolute necessity for the
seasoned analyst.

#### Further reading

Cook, Terry. 1995. \"It\'s 10 O\'Clock: Do You Know Where Your Data
Are?\". *Technology Review*. January. p. 48. An insightful article that
discusses some unprecedented problems with data that we face in the
information age.

> \"For every foolproof system, there is a bigger fool.\"
>
> \-- Edward Teller \--

*FIG: cartoon of someone digging into the ground, which is composed of
numbers*.

### Recycling old envelopes

Ross and Ross (1986), Mattimore (1994), and von Baeyer (1993) describe
how the physicist Enrico Fermi used to dole out exam problems but not
supply all the information necessary for a solution. His students (my
former professor Art Rosenfeld was among them) were expected to make
educated guesses about the missing parameters to solve these so-called
\"Fermi problems\". This technique is not often taught in school, but it
is one well worth learning. For virtually any problem, it is possible to
create an approximate solution using information you know or can
estimate from daily life experience. Most people don\'t believe this
until they try it a few times, but it\'s true.

I read an article recently that described how Global Water Corp. of
Vancouver had signed a contract with an Alaskan city (Sitka) which would
allow it to harvest water from glacial runoff.[^69] The company was
going to take converted oil tankers and ship the water to China, where
it would then be bottled and sold. I initially thought that this venture
was absurd. A few back-of-the-envelope calculations later, and I
convinced myself quickly that there was method to this apparent madness.

The company signed a fifteen year contract covering 4.8 billion gallons
of water a year. The article claimed that half-liter bottles of water
from Europe currently cost Chinese consumers about \$5.25, while this
new source of water would cost about \$0.70 per half liter bottle
(\$1.40 per liter).

I began by thinking about the commodity the oil tanker was originally
meant to carry: crude oil. I knew from reading the newspaper that such
oil was selling for about \$20 per barrel. I also knew that there are 42
gallons per barrel of oil, yielding a cost per gallon of about
\$0.50.[^70] There are about 4 liters per gallon (a liter is a bit more
than a quart), so the price of oil per liter is about \$0.12. The
bottled water would (assuming the article was correct) sell for \$1.40
per liter.

The price of the bottled water is about ten times greater than that of
the original cargo for the tanker, which convinced me that this company
could make some serious money shipping water to China. Of course, there
are costs of cleaning and converting the tanker to hold pure water, and
costs of packaging the bottled water that a gasoline vendor would not
incur, but the costs of transport would be comparable. I reasoned that
the one-time cost of converting the tanker would be relatively small
when amortized over many trips. The costs of packaging and marketing the
water would at most amount to ten to fifty cents per liter (this was a
guess based on what bottled water costs in the U.S.). The venture still
seemed likely to be profitable after these adjustments.[^71]

I once sat outside an air-conditioned store in Cambridge, Massachusetts
that had its door propped open on a very hot day. Using some simple
assumptions and conversions I had memorized, I calculated how many
dollars per hour the store was wasting by refrigerating the outside
world. That calculation was enough to convince the store manager to
close the door (he probably never had anyone badger him about *that*
before, even with all the crazy scientists wandering around Cambridge!).

Another calculation I did about ten years ago was to figure out the
wattage of a candle (it\'s about 100 watts). All I needed to assume is
that wax is similar to petroleum (I had the energy content of petroleum
memorized) and note the time it took for the candle to burn away. I also
needed to know some conversions to turn English units into metric units.
LINK TO FACTS AT YOUR FINGERTIPS SECTION.

Such \"back-of-the-envelope\" calculations, supplemented by what
Mattimore calls \"smart guessing\", are not just for scientists. A
business plan for a new venture, for example, usually involves dozens of
such smart guesses about sales of a product, costs of sales, overhead,
and other parameters. The first draft of such a plan is essentially a
back-of-the-envelope calculation that will later be refined and improved
in discussions with experienced business people.

People involved in fast-moving new ventures don\'t have time to research
issues thoroughly, and don\'t have money to hire someone to do it for
them. Often, back-of-the-envelope calculations are the best that can be
done in the time allotted. The law of averages keeps the accuracy of
such calculations from deteriorating too much, because you are just as
likely to overestimate as to underestimate (assuming that you\'re not
biased one way or another). Across all the assumptions you make, these
deviations average out.

One modern tool that is invaluable for such calculations is the
spreadsheet (proving that you don\'t need an envelope to create
back-of-the-envelope calculations). I often gin up quick calculations
using a spreadsheet and plug in guesses for those parameters I don\'t
know at the time. This approach allows me to think through the
calculations without hindrance, then go back later to check the input
assumptions. Later modifications to these assumptions trickle through
the calculations nicely without requiring much further work on my part.

Another simple but powerful tool in back-of-the-envelope calculations is
called \"unit analysis\". Think of the calculation we did above to
calculate the cost of oil in \$/liter. I could just list the numbers and
multiply them together, like this:

Cost (\$/liter) = = 0.12

Alternatively, I could list each of the terms with units attached, so
that we can see if they cancel out and yield the end result in units we
desire:

Cost (\$/liter) = x x =

I initially scoffed when my high school chemistry teacher taught us this
methodical way of carrying through and checking our calculations. At the
time, the calculations were so simple that I didn\'t need to be
methodical, and could just do them in my head. I\'ve learned through
experience, however, that calculations are rarely that simple in
practice, and even if they are simple, I still make mistakes because
I\'m tired or rushed. It pays to write out the units for your
calculation to make sure you are calculating correctly.

To become truly proficient at these techniques, I recommend memorizing
key conversion factors, at least in a rough way.[^72] Which ones you
memorize depend on your chosen field. I find energy and environmental
conversion factors to be most useful in my work, but I have also
memorized many other conversions because they often come in handy. If
you simply can\'t memorize such items, then create a small page with
common conversion factors and keep it in your wallet.[^73] While this
may brand you as a nerd in the eyes of your compadres, it will surely
impress them when you can answer questions in a rough way while only
consulting your little data sheet. LINK TO facts at your fingertips
section.

von Baeyer points out that doing such calculations instead of relying on
authority engenders self confidence and independence. This self
confidence makes it more likely that you will tackle such problems in
the future, thereby ensuring you will gain further experience and self
confidence. To know in your heart that you can roughly estimate just
about anything is a marvelous feeling of mastery.

#### Further reading

Harte, John. 1985. *Consider a Spherical Cow: A Course in Environmental
Problem Solving*. Los Altos, CA: William Kaufmann, Inc. The first short
chapter in this classic book asks the question \"How many cobblers are
there in the U.S.?\", and then proceeds to investigate ever more
detailed puzzles in environmental science. This book, and the course in
which it is used, were inspirational to me in graduate school.

Hershey, Robert L. 1982. *How to Think with Numbers*. Los Altos, CA:
William Kaufmann, Inc.. This book is hard to find, but if you\'re
interested in learning about some relevant tricks, it\'s worth tracking
down.

Hershey, Robert L. 1997. \"Thinking Big\". *The Washington Post*,
September 10, 1997, p. H8. A short article talking about scientific
notation and its usefulness for treating really BIG numbers.

Mattimore, Bryan W. 1994. *99% Inspiration: Tips, Tales & Techniques for
Liberating Your Business Creativity*. New York, NY: AMACOM (A Division
of American Management Association). \"Chapter 10\--Fermi Problems:
Guesstimating in Today\'s Decimal-Point World\".

Ross, Joan and Marc Ross. 1986. \"Fermi problems, or how to make the
most of what you already know.\" In *Estimation and Mental Computation:
1986 Yearbook*. Edited by H. L. Schoen and M. J. Zweng. National Council
of Teachers of Mathematics.

von Baeyer, Hans Christian. 1993. *The Fermi Solution: Essays on
Science*. New York, NY: Random House. von Baeyer\'s ruminations on
science are interesting and illuminating. His chapter on Fermi problems
discusses their history.

######### The desire for quantification can go too far.

![](media/image19.wmf){width="6.346527777777778in"
height="2.3055555555555554in"}Need to get permission to reproduce this
cartoon and pay any royalty due.

> \"Whether the problem concerns cooking, automobile repair, or personal
> relationships, there are two basic types of responses: the
> fainthearted turn to authority--to reference books, bosses, expert
> consultants, physicians, ministers--while the independent of mind
> delve into that private store of common sense and limited factual
> knowledge that everyone carries, make reasonable assumptions, and
> derive their own, admittedly approximate, solutions.\"
>
> \--Hans Christian von Baeyer\--

### Hear both sides

One of the first rules of management in dealing with a conflict between
two coworkers is to listen carefully to both sides of the story before
saying or doing *anything*. The same lesson applies in evaluating any
two courses of action or points of view. It almost always pays to hear
two eloquent supporters of divergent viewpoints address each other\'s
criticisms in a discussion or debate format. If you do not include such
discussions as regular fodder for your deliberations, you are missing
out on one of the most potent ways to clarify any set of issues. In rare
cases, of course, such debates can muddy the waters still further, but
they are almost always helpful as long as the participants are well
matched and prepared.

President John F. Kennedy knew this when he created a cadre of advisors
who held widely divergent viewpoints during the Cuban Missile crisis.
After the Bay of Pigs debacle, Kennedy vowed never to rely solely on his
military advisors again. He created an ad hoc group of advisors, led by
his brother Robert, to bring to his attention other alternatives. One of
these options was the blockade of Cuba, which was ultimately
successful.[^74]

In the academic world, there is a tradition of \"brown bag lunches\",
where researchers present work that is at an early stage to friends and
close colleagues. These lunches serve to identify problems with analysis
before a paper is written or presented to a wider group. They work best
when all concerned are on friendly terms and the objective of clarifying
the key argument is accepted by all participants.

The business world does not generally encourage the kind of interaction
found in the \"brown bags\", to its detriment. One acquaintance of mine
works for a large mutual fund company. She lamented to me that the
people under her supervision did not possess the ability or inclination
to check their own work, which often led to analysis being produced for
her that did not answer the management questions at hand. These analysts
were responsible for creating reports that the firm used to assess
financial fitness, so the reports were important. In spite of this
importance, there was no tradition of intellectual criticism among the
analysts at this firm, and there was consequently no effective way to
teach them what constituted good analysis.

If you have reached a conclusion, it still pays to have a critic review
your work. In fact, such review is virtually always a good idea. Confirm
your results using as many independent sources as you can find. These
can be only approximate comparisons, but you should always do this, just
like you would test computer equipment before you put it in the field.

> \"It is a capital mistake to theorize before you have all the
> evidence. It biases the judgment.\"
>
> \--Sherlock Holmes\--

### The value of models

In this book, when I use the term \"model\" I\'m not referring to Cindy
Crawford and her ilk. Instead, I\'m talking about conceptual (usually
mathematical) models that can be used to understand the world, which can
range from the back of an envelope to a computer model with millions of
lines of code. Starfield et al. (1990) summarize it nicely when they
state \"An explicit model is a laboratory for the imagination\". Models
allow us to experiment, test, and learn about the world in a systematic
way, without leaving our desk.

People are often intimidated by the use of computer models to conduct an
analysis, but never forget that computer models depend on data and
assumptions that are based in large part on analysts\' judgment. It is a
rare model that does not contain dozens or hundreds of such assumptions,
and in fact, these data and assumptions determine the outcome in many
cases.

In real estate, Robert J. Ringer found himself confronted with owners of
buildings for sale with fantastic expectations for how good an
investment their building was:

\"As a general but very consistent rule, owners are so unrealistic when
it comes to vacancy factors, replacement costs, and other expense items
that are not readily ascertainable that I normally considered their
projections and operating statements to be virtually meaningless.\"

Using his electronic calculator and his knowledge of real estate, Ringer
was able to determine quickly whether the building was salable at or
near the owner\'s asking price. His simple conceptual model allowed him
to create a quick estimate of net cash flow on the proverbial
back-of-an-envelope. This simple screening technique allowed him to
avoid wasting time on deals that would never close because of the huge
gap between the reality of the project\'s net cash flow and the fantasy
of the owner\'s expectations.

The technique of using a simple and transparent model to evaluate the
results of a more complex calculation is immensely powerful. Analysts at
the International Institute for Applied Systems Analysis (IIASA) found
this out to their chagrin when Bill Keepin, a visitor to IIASA, was able
to almost exactly reproduce the results of a multiyear, multimillion
dollar study using some of the study\'s key input assumptions and a hand
calculator. Keepin (1983) showed that the study\'s results followed
directly from the input assumptions. He concluded that the study\'s
projections of future energy supply \"are opinion, rather than credible
scientific analysis, and they therefore cannot be relied upon by policy
makers seeking a genuine understanding of the energy choices for
tomorrow\".

Beware of big complicated models and the results they produce. Generally
they involve so much work to keep them current that not enough time is
spent on data compilation and scenario analysis (LINK TO TELL A GOOD
STORY). Morgan and Henrion (1992) devoted an entire chapter to such
models, and begin by summarizing this fundamental truth:

\"There are some models, especially some science and engineering models,
that are large or complex because they need to be. But many more are
large or complex because their authors gave too little thought to why
and how they were being built and how they would be used\"

Such large models are essential only for the most complex and esoteric
analyses, and a simpler model will usually serve as well (and be more
understandable to your intended audience).

Don\'t be too impressed by a model\'s complexity. Instead, ask about the
data and assumptions used to create scenarios. Focus on the coherence of
the scenarios and their relevance to your decisions, and ignore the
marketing doublespeak of those whose obsession with tools outweighs
their concern with useful results. Sadly, many modelers fall into this
latter camp, and you would do well to avoid them.

#### Further reading

Keepin, Bill. 1983. *A Critical Appraisal of the IIASA Energy
Scenarios*. International Institute for Applied Systems Analysis.
WP-83-104. October. (Need to get Keepin\'s permission before citing).
The complete documentation for Keepin\'s criticisms of the IIASA
scenarios.

Keepin, William and B. Wynne. 1984. \"Technical analysis of IIASA energy
scenarios.\" *Nature*. vol. 312, p. 691. This is the journal article
summary of Keepin\'s work at IIASA.

Morgan, M. Granger and Max Henrion. 1992. *Uncertainty: A Guide to
Dealing with Uncertainty in Quantitative Risk and Policy Analysis*. New
York, NY: Cambridge University Press. This book gives sage advice about
appropriate uses for models in analysis, and is one of the few places
where the large complex models are taken to task for their failings.

Ringer, Robert J. 1974. *Winning Through Intimidation*. New York: Funk &
Wagnalls. A funny and entertaining look at one man\'s transformation
from loser to winner, in part through his clever use of models.

Starfield, Anthony M., Karl A. Smith and Andrew L. Bleloch. 1990. *How
to Model It: Problem Solving for the Computer Age*. New York, NY:
McGraw-Hill, Inc. This book is an excellent, albeit technical, guide to
modeling practice in the real world, particularly computer modeling. It
teaches problem-solving skills essential for good modeling, and focuses
mainly on scientific applications, though its approach is useful for
others as well. It is unfortunately out of print, but many libraries
will have it.

> \"Programming today is a race between software engineers trying to
> build bigger and better idiot-proof programs, and the universe trying
> to produce bigger and better idiots. So far, the universe is
> winning.\"
>
> \--Rich Cook\--[^75]

### Understanding forecasts

The future is uncertain, but people keep trying to forecast it anyway.
In an article with an ironically humorous and utterly extraneous
subtitle \"Few Wall Street Analysts said that \'96 would be this good:
Forecasts for \'97 may or may not be just as inaccurate\", Chet Currier
of the Associated Press assessed the sorry record of most financial
analysts in predicting the huge increase in stock prices in 1996.[^76]

After describing how virtually all analysts predicted modest gains or
little change from end-of-1995 levels, Currier states that the major
market indices racked up gains of 20 percent or more in 1996. His
conclusions about the value of such forecasts are particularly
insightful:

None of the errant prognostications cited above were wild guesses, or
interpretations of the leaf patterns in some hallucinogenic tea. They
were the carefully considered projections of respected commentators who
enjoy wide followings as financial analysts. All of them remain in their
same jobs, or comparable positions of prominence, a year later, and are
presumably busy right now preparing to make their projections for
another year.

As usual, these commentaries may well provide a useful point of
departure for your own investment planning in 1997. *The reasoning
behind a prediction is often the most interesting and useful thing about
it*. (italics added). But the experience of this year dramatizes how
skeptically you may take all forecasts of something as unpredictable as
the stock market -- even from people who know what they are talking
about.

Newspapers always trumpet the results of such forecasts, but as the
stock market example shows, you should be wary.

Schnaars (1989) gives many examples of forecasts for popularity of new
products that were totally off the mark. In fact, about 80% of the
forecasts he examines were grossly inaccurate, which made him question
the entire basis for predicting adoption of new technologies. He
concluded, after reviewing dozens of examples, that explicit scenario
analysis (LINK TO \"TELL A GOOD STORY\") is far superior to most
conventional ways of exploring the future.

Forecasts are used

• for planning (either institutional or personal);

• for advocacy (to lend credence or context to concern over a particular
issue);

• for research (to assess the potential effects of current or expected
future trends).

Most institutions create forecasts for their own budget planning. Such a
forecast attempts to capture recent trends and allow the institution to
plan for expected growth or declines in revenues. A typical forecast in
this context would look one or two years ahead, sometimes three or four
years in exceptional cases. Longer term forecasts are used for
government institutions (like Social Security) that have obligations
spread over many years. Forecasts can also serve as part of a scenario
planning exercise (LINK TO TELL A GOOD STORY).

An institution concerned with a particular issue will often create a
forecast that validates concern for that issue, as part of their effort
to create free media. In a recent edition of the Sunday San Francisco
Chronicle/Examiner (31 August 1997) I found two separate examples where
a forecast was cited for this purpose. In one case, a California real
estate investor and analyst predicted that housing prices in California
would double over the next eight years (p. E-1). In another, the
Cremation Association of America predicted that the rate of cremation,
measured as the % of deaths that lead to cremation, would increase from
21% in 1997 to 38% in 2010 (p. D-1). In both cases, a person or
institution developed forecasts that showed why a particular issue of
concern to them should also be of interest to the public.

The investor who predicted a doubling of California home prices was
roundly criticized by his colleagues, but he succeeded in getting the
attention of the press, in part because his prognostication conflicts so
glaringly with the conventional wisdom. That strategy is a successful
one that is widely used: predict something just a tad outrageous, and
use your credibility from previous work to lend credence to your
prediction. Then talk to as many reporters as will listen, in hopes they
will mention you and your company in their columns.

More complex issues can require detailed forecasts to truly understand
them, which usually involves computer modeling (LINK TO THE VALUE OF
MODELS). In these applications, researchers will create a baseline
forecast that attempts to predict how the world will evolve assuming
current trends continue. This forecast is often called a
\"business-as-usual\" case. Then, one or more alternative forecasts are
created, to assess the potential effects of changes in key factors on
the results. For example, an economic forecaster might use such an
analysis to assess the likely effects of a change in property taxes on
economic growth in her state.

#### The wonders of exponential growth

Forecasts of population or economic activity are often expressed as
annual exponential growth rates (Ehrlich et al 1977). For example, the
annual growth in gross domestic product was roughly 3.5% in 1996. If
this growth continued at 3.5% per year for five years, total growth
would be 1.035 x 1.035 x 1.035 x 1.035 x 1.035 - 1, or 19%.

Exponential growth rates generally apply to population because the rate
of growth is proportional to the number of people who are alive at any
time (the more people there are, the more babies are born). Similarly,
money in a bank account grows exponentially because the interest earned
is related to how much money is in the account at any time.

Exponential growth can lead to surprising results in the longer term.
Let\'s compare two countries of 100 million people, one of which grows 2
million people per year every year, and the other of which grows 2%
every year (2 million in the first year). After 100 years, the first
society will have grown to 300 million people, while the second society
will have grown to more than 700 million! The differences become even
more pronounced for higher growth rates. If the society grew at 4%/year,
it would grow to more than 5 billion 100 years hence. A doubling of the
exponential growth rate leads to a seven fold increase in total
population!

The famous futurist Julian Simon once made the outrageous statement that
\"We have the technology to house, feed, and clothe a growing population
for six billion years\".[^77] When told that any reasonable population
growth rate would lead to the mass of all people exceeding the mass of
the known universe over that period, he revised his statement to six
*million* years. Even this modest revision wasn\'t enough to get Simon
out of trouble, though. In six million years at modest growth rates, the
number of people would exceed the number of electrons in the known
universe!

Exponential growth can also lead to trouble when the rate of growth
changes dramatically. In the 1950s and 1960s, growth in U.S. electricity
use increased at a roughly constant 7%/year. Power planners had an easy
job until the 1970s, when the oil shock and problems with large power
plants led to price increases and slowing demand for power. Growth rates
in electricity demand slowed to 2-3%/year, and utilities were forced to
cancel hundreds of huge power plants because of the drastically reduced
growth in power use.

#### A key trick of the trade

In evaluating forecasts of exponential growth rates, an important trick
is known as \"the rule of 70\". If you have an annual growth rate (say
2% per year for U.S. population) then the time it will take (in years)
for population to double is roughly or 35 years. Growth of 7% per year
implies a doubling time of 10 years. You don\'t need to know why this
works, you just need to remember it.

Why is a doubling time useful? It is much easier to calculate multiples
of two than to actually calculate exponential growth in the formal way.
When only an approximate number is needed, a doubling time is good
enough. Say you want an approximate estimate of how much \$10,000 now
will be worth when your retire. If the growth rate after inflation is
7%/year (a reasonable estimate for money invested in a tax-deferred
stock account) the doubling time is ten years. In 30 years, the money
will have undergone three doublings, or a factor of eight increase (2 x
2 x 2), and will be worth about \$80,000 in 1997 dollars (of course,
you\'ll have to pay taxes on that money when you withdraw it).

This approximation becomes less accurate as the growth rate increases.
For rates above 10% per year, you\'ll probably want to check your
results using more accurate methods. but for most situations, the rule
of 70 is more than good enough (Ehrlich et al. 1977).

#### The logistic curve

Many phenomena in nature and in business are described by the so-called
\"logistic curve\" (also known as the \"S\" curve). This curve (shown in
the Figure) is characterized by slow but rising exponential growth in
the early phase, rapid growth in the middle phase, and growth that
tapers off as fundamental limits are reached (Ehrlich et al. 1977). The
logistic curve describes most phenomena eventually, and applies equally
well to technology adoption and population growth. We just don\'t know
where the top limits are in many cases, so applying this concept can be
tricky.

######### A logistic curve

![](media/image20.wmf){width="4.652777777777778in"
height="1.6666666666666667in"}

#### Predicting environmental consequences

One kind of prediction entails special difficulties: estimates of the
damages from pollution. Such predictions will always be inaccurate
because pollution\'s many effects are spread over time and space, and
the causal links are extremely complicated. Our understanding of these
links will always lag behind our ability to alter them, as David Bella
points out:

\...an improved ability to foresee consequences requires a greater
increase of general knowledge than does an improved ability to produce
changes\...changes can be accomplished one at a time as if they were
essentially in isolation from each other. Moreover, only a small part of
the environment and only a few environmental properties must be
understood in order to produce a change. In contrast, to foresee the
consequences of change requires that one examine the combined effect of
many changes (Bella 1979).

This fundamental truth is one to keep in mind when making forecasts of
environmental risks. For many such hazards, we can *never* have complete
information, and yet preventative action may be prudent and appropriate.

#### Coping with an uncertain future

Forecasts, in spite of their failings, are still widely used and useful.
If they are part of your planning process, don\'t create just one
forecast. Instead, use a set of forecasts to explore the future. Vary
key factors and investigate which of them to ignore and which to dissect
further. All forecasts are wrong in some important respect, but if the
process of creating them can teach you something about the world and how
events may unfold, creating them will be worth the effort. (LINK TO
\"Tell a good story\" section).

Adopt strategies that are robust in the face of inevitably imperfect
forecasts. Apple Computer was plagued for years by poor forecasting,
which led to surpluses of some models and shortages of others.
Eventually, the company moved to \"build-to-order\" manufacturing in
November 1977, which allowed it to assemble computers as requested by
customers. This strategy reduces dependence on forecasts, but introduces
other challenges in manufacturing. These proved surmountable, which is
why Dell Computer and others had previously adopted this approach.

#### Further reading

Bella, David A. 1979. \"Technological Constraints on Technological
Optimism.\" *Technological Forecasting and Social Change*. vol. 14, no.
p. 15. Bella uses simple mathematical concepts to show why unbridled
optimism about exponential growth must eventually give way to the
sobering realities of our imperfect foresight.

Bright, James R. 1972. *A Brief Introduction to Technology Forecasting:
Concepts and Exercises*. Austin, Texas: The Pemaquid Press. This book is
a bit dated in some ways, and its optimism about accurate forecasting is
a bit quaint, but it\'s long on examples, practical advice, and careful
thinking about forecasting in the real world.

*Ecological Numeracy*. New book, due out in 1998. Search Amazon for it.

Ehrlich, Paul R., Anne H. Ehrlich, and John P. Holdren. 1977.
*Ecoscience: Population, Resources, Environment*. San Francisco: W.H.
Freeman and Company. A beautiful book with a nice write-up of the
subtleties of population forecasts, exponential growth, and logistic
curves (see Chapter 4). Even though it\'s two decades old, there\'s
still nothing like it.

*Institute for Business Forecasting*. http://www.ibf.org. ibf@ibf.org.
800/440-0499, 718/463-3914, 718/544-9086 fax. Flushing, NY. An
institution devoted to disseminating knowledge about business
forecasting and planning. If you\'re interested in forecasting in a
business context, this organization is a good place to start.

Morgan, M. Granger, and Max Henrion. 1992. *Uncertainty: A Guide to
Dealing with Uncertainty in Quantitative Risk and Policy Analysis*. New
York, NY: Cambridge University Press. One of the best books on
uncertainty ever written. A true classic.

Schnaars, Steven. 1989. *Megamistakes: Forecasting and the Myth of Rapid
Technological Change*. New York, NY: Free Press. This book gives great
examples of major forecasting failures. Both lively and insightful, this
book is a great read for those interested in pitfalls in forecasting.

> \"In the space of 176 years, the Lower Mississippi has shortened
> itself 242 miles. That is an average of trifle over one mile and a
> third per year. Therefore, any calm person who is not blind or idiotic
> can see that in the Old Oolitic Silurian Period, just a million years
> ago next November, the Lower Mississippi was upward of 1,300,000 miles
> long. By the same token, any person can see that 742 years from now,
> the Lower Mississippi will be only a mile and three-quarters long.
> There is something fascinating about science. One gets such wholesale
> returns of conjecture out of such a trifling investment of fact.\"
>
> \--Mark Twain\--

### The uncertainty principle and the mass media

One of Werner Heisenberg\'s major contributions to quantum theory was
the so-called \"uncertainty principle\". This principle governs the
relationship between the observer and the physical world on the
subatomic level. To measure the velocity and position of a particle (an
electron, for example) the observer must bounce something off that
particle, but by that action will change the electron\'s velocity and
location. The more the observer tries to localize the location of the
electron, the more the measurement disturbs the velocity. Similarly, if
the observer attempts to pinpoint the velocity of the electron, its
position becomes less and less certain. The observer is constrained to
only know the position and velocity of the electron to some level of
certainty, but never more.

This principle can be loosely summarized for use in the everyday world
as \"the observer disturbs the system\". Any time a measurement is made,
that measurement changes the system being measured, usually in some
minuscule way. Measuring the temperature of a liquid, for example,
requires that a small amount of heat be extracted from the liquid and
transferred to the thermometer. In physical terms, a good measurement is
one that disturbs the system as little as possible. Things are a bit
more complicated with social systems, especially when observations by
influential members of the mass media can disrupt such systems in a
significant way. The recent historical record is rife with both
intentional and unintentional disturbances of this kind.

The phenomenon of publicly available ranking systems is one example of
how intentionally making new information available can lead to
institutional change. Before Milton Moskowitz and Robert Levering wrote
their book \"The 100 Best Companies to Work for in America\" in 1984,
people paid little attention to family issues in their choice of jobs,
at least in part because systematically compiled information was not yet
available. When their work was published, the rankings became a force
for change that did not exist before.[^78] The same effect occurs
whenever pollution emissions or campaign contributions are made public
in an accessible way: Embarrassment or enlightened self interest causes
people and institutions to modify their behavior.

Another example of intentional disruption of observed systems is the set
of opinion polls that influence the opinions they attempt to measure
(Crossen 1994, p. 106). The pollsters ask questions of a carefully
selected sample of individuals, then create a press release summarizing
some of the key results. These \"promotional surveys\" are increasingly
common. They can be conducted by a company for their own marketing, a
non-profit institution for their own fund-raising, or by a \"neutral\"
third party like a trade magazine. For example, Kiwi Brands (a shoe
polish company), polls people on a regular basis on such important
topics as the relationship between ambitious people and shining shoes,
and what the biggest grooming mistakes are: not surprisingly, badly
shined shoes came out near the top (Crossen 1994, p.94). Another company
well-known for this tactic makes sugarless gum (\"Four out of five
dentists surveyed prefer\...\"). In any case, read the fine print, and
learn who was surveyed, how and when they were surveyed, and what
questions they were asked. It is easy to create a survey that will give
the desired result, and there\'s no way to know without going back to
the survey questions. Link to GO BACK TO THE QUESTIONS.

A prominent recent example of unintended harm caused by mass media
reporting is that of Apple computer as it wrestled with its financial
problems in 1996 and 1997. Reporters repeated over and over again the
time-worn opinions of pundits, both well and ill-informed. The whirlwind
of bad publicity, in part brought on by Apple management\'s bad
decisions and in part the creation of a media feeding frenzy, created
uncertainty and doubt among Apple\'s customers, thereby influencing the
very event on which the news people were reporting (the struggles of
Apple computer). The reporters created a kind of \"social proof\"
(Cialdini 1993, p. 114) that Apple would fail, making it more likely
that they would.

An unintended result of polls is that polling results, taken out of
context, often make headlines. People read the headlines and rethink
their opinions based on them (if thinking is the right word to describe
this process). The poll itself becomes news, and in so doing it
influences the outcome of elections and changes public opinion.[^79] How
much of an effect polls have on elections is uncertain, but they clearly
can influence the results, as shown by the 1948 U.S. Presidential
election. Crossen quoted Burns Roper (son of the founder of the Roper
polling organization) as saying that the 1948 polls \"lulled the
Republican supporters into a false sense of confidence and scared the
bejesus out of organized labor\". The pollsters also decided that people
had decided on their presidential choice weeks before the election, so
they stopped regular polling. The result was the erroneous Chicago
Tribune headline \"Dewey defeats Truman!\".

Mass media amplifies the effect of such observations, and this tendency
will only become more prevalent as the cost of transmitting information
continues to plummet. If you are a member of the media, take
responsibility for the massive influence your institution can have, and
use special care to report news in a constructive way. If you repeat the
conventional wisdom without analysis, you enlighten no one and may cause
harm to people and institutions that you do not intend.

#### Further reading

Cialdini, Robert B. 1993. *Influence: The Psychology of Persuasion*. New
York, NY: Quill\--William Morrow. Cialdini\'s book shows how
\"compliance professionals\" use their knowledge of the human psyche to
influence people\'s decisions.

Crossen, Cynthia. 1994. *Tainted Truth: The Manipulation of Fact in
America*. New York, NY: Simon & Schuster. Crossen gives dozens of
examples of how the results of research and reporting can be manipulated
for good or ill.

Fuller, Jack. 1996. *News Values: Ideas for an Information Age*.
Chicago, IL: University of Chicago Press. Fuller explores how new
technologies continue to change the role of the journalist in modern
society.

> \"Public opinion is a compound of folly, weakness, prejudice, wrong
> feeling, right feeling, obstinacy, and newspaper paragraphs.\"
>
> \--Robert Peel \--[^80]

### Get real!

People think they are special and that they will win in spite of
impossible odds, a phenomenon I refer to as \"the lottery fallacy\".
This feeling is reinforced by TV shows and movies where heroes evade
thousands of machine gun bullets and get past an entire army to
accomplish their mission, then rescue the kidnapped heiress and escape
unscathed, after destroying the evildoers\' headquarters and personally
humiliating him.

Buying lottery tickets is a losing proposition, but millions of people
do it every year. The odds of winning a huge prize are millions to one
against. The money from the ticket purchases goes into a big pool, from
which comes the expenses of running the lottery operation and any money
that gets siphoned off for schools (in California, hundreds of millions
is extracted for this purpose). What\'s left is used to pay off winners,
and this amount is by definition smaller than the amount of money that
went into the pot. So on average, you will lose.

You may feel that the fun and excitement you get from buying an
occasional lottery ticket is worth the minor expense it represents. If
so, go ahead. Just don\'t expect to make money in the deal.[^81]

What about the winners? People do win those big prizes, but very
few.[^82] Millions of people must lose for those few to win big, and
odds are you will be among the millions of losers, not the lucky few.

Investing in common stocks is another example where this fallacy comes
into play. Burton Malkiel, in *A Random Walk Down Wall Street*, argued
that \"a blindfolded chimpanzee throwing darts at the stock pages could
do as well as the experts\". If you think you can consistently beat the
market indices, then put your money into individual stocks. If you
can\'t (and 70-80% of the pros can\'t) then invest in index funds and
don\'t waste time worrying about the vagaries of individual stocks,
which can be nerve wracking (particularly for stock of smaller
companies).

In facing this all too common fallacy, I find it best to refer to
Ringer\'s theory of reality:

\"The theory emphasizes first of all, that reality isn\'t the way you
wish things to be, nor the way they appear to be, but the way they
actually are. Secondly, the theory states that you either acknowledge
reality and use it to your benefit or it will automatically work against
you.\"

I find that many brilliant people fall prey to this fallacy, and would
do well to heed Ringer\'s words.

#### Further reading

Malkiel, Burton G. 1996. *A Random Walk Down Wall Street*. 6th ed. New
York, NY: W.W. Norton. The classic book that debunks the fallacy of the
stock pickers in the financial district.

McGervey, John D. 1986. *Probabilities in Everyday Life*. New York, NY:
Ballantine Books. This book applies the lessons of probability and
statistics to insurance, auto safety, smoking, the stock market, and
other parts of our everyday experience.

Ringer, Robert J. 1974. *Winning Through Intimidation*. New York: Funk &
Wagnalls. Ringer describes his theory of reality in an easy to read book
chock full of real life experiences.

> \"Human beings, who are almost unique in having the ability to learn
> from the experience of others, are also remarkable for their apparent
> disinclination to do so.\"
>
> \--Douglas Adams, [Last Chance to See]{.underline}\--

## Show your stuff

Don\'t forget the final step in any analysis: making your results useful
to others. Your reports should be treasured by your colleagues as
classics full of pertinent data, exhaustive documentation, and clear
thinking. If they\'re not, you should follow the advice contained in
these chapters and change your ways.

*Familiarity breeds interest*: The first rule of data presentation is to
know your audience and present information they care about in a form
they can easily grasp. Most analysts forget that other people don\'t
care nearly as much about the results of the analysis as they do.

*Document, document, document*: An astounding number of analysts
routinely omit vital data and assumptions from their reports, but this
pernicious practice is one to avoid. The best analysts document
*everything*, giving credit where credit is due, leaving a trail for
them to remember, and leaving a trail for others to follow.
Documentation is also a key step in checking your work, because it
forces you to think clearly about your analysis.

*Let the tables and graphs do the work*: When writing technical reports,
create the analysis, tables, and graphs first, then write around them.
If the analysis is well thought out, the tables and graphs well
designed, and the audience clearly defined, the report should
practically write itself.

*Designing compelling graphs and figures*: Follow Edward Tufte\'s rules
for graphical excellence and avoid the all too common pitfalls of
creating charts and graphs. Your goal should be to give to the reader
\"the greatest number of ideas in the shortest time with the least ink
in the smallest space\".

*Designing good tables*: A good table is a work of art, while a bad one
is worse than useless. Make your tables a resource that your readers
keep as a reference for many years to come.

*Using numbers effectively in oral presentations*: Even veteran
presenters use too many of the wrong numbers in their presentations.
Only show numbers that support a key conclusion, and focus on the
conclusion, NOT the numbers themselves.

If you follow this advice, you\'ll soon gain a reputation for analytical
integrity and clear thinking that can\'t be beat. If you don\'t, you\'ll
be contributing to the piles of mediocre and barely used reports that
inhabit bookshelves everywhere, and you most certainly don\'t want to do
that.

> \"They say that 95% of television is garbage. But 95% of EVERYTHING is
> garbage\".
>
> \--Gene Roddenberry \--

### Familiarity breeds interest

How results are expressed are at least as important as what the results
are. Most analysts are so involved in their calculations that they
forget that other people don\'t care nearly as much about the results of
the analysis as they do. Good analysis is often handicapped in its
presentation because the authors have not taken the vital last step of
thinking about what the audience cares about and expressing the results
in those terms.

One of my favorite professors in graduate school, Art Rosenfeld, is a
master at making complex analysis results understandable to the lay
person. Art presented his estimates of energy savings from the latest
efficient gadget (a new refrigerator, for example) for many different
people. Sometimes he would testify before Congress, other times he would
address public utility commissioners, and still other times he would
tell his story for a general audience. He would always tailor his talk
for the listeners. If oil drilling in the Arctic National Wildlife
Refuge was a hot topic before Congress that day, he would express his
energy savings estimates in terms of \"Arctic Refuges saved\". If the
public utility commissioners were concerned about canceling a power
plant construction project, Art would calculate how much the savings
were worth in terms of \"avoided power plants\". And for the lay
audience, Art would show how much money they\'d save per household if
they bought the better fridge. In each case, Art\'s message gained
resonance because he thought seriously about the concerns of the
audience.

In both talks and papers, your audience will only retain a few key
points from your work. *You* must decide what those few points should
be, and hammer them home. If you don\'t you risk not reaching the people
you set out to convince. So step back and reflect (LINK) after you\'ve
completed your analysis. Think about what you\'ve learned, and explain
it in three or four simple sentences. Dig into the numbers (LINK) to
support your main points, but don\'t overwhelm your audience with
numbers. Instead, focus on the conclusions and describe why your
analysis makes such a compelling case.

This lesson is particularly important when the analysis results are
intended to affect public opinion or a public policy debate. Journalists
generally latch on to only a few key results of a complex study when
deciding what \"angle\" to take with a story. Express these results to
reach out to the desired audience, and your effectiveness will increase
manyfold.

#### Further reading

Fuller, Jack. 1996. *News Values: Ideas for an Information Age*.
Chicago, IL: University of Chicago Press. A clearly written discussion
of the changing role for journalists in the information age.

> \"In order to achieve victory, you must place yourself in your
> opponent\'s skin. If you don\'t understand yourself, you will lose one
> hundred percent of the time. If you understand yourself you will win
> fifty percent of the time. If you understand yourself and your
> opponent you will win one hundred percent of the time.\"
>
> \--Tsutomu Oshima\--[^83]

### Document, document, document

Over the years, friends have clipped dozens of articles for me on topics
in which they knew I had an interest. In almost all cases, they
neglected to write on the clipping the full reference, thus rendering it
unusable for research purposes. An article without a reference is
useless except as background info. An article with a reference is a
citable source.

Documentation lends *credibility* to any intellectual effort. It also
recognizes the previous work of others and allows readers to follow your
thought processes (it also allows YOU to recreate your thinking months
after you have achieved some conceptual breakthrough). Any competent
analyst ought to be able to recreate your analysis from the
documentation you provide, and you must be able to do the same more
quickly than others can. Finally, the process of documenting your
results can help you check those results and ensure accuracy.

#### Giving credit where credit is due

Progress in intellectual endeavors depends on prior work being fairly
acknowledged. When you have relied on a particular fact or insight that
you got from another source, you must reference it. You can put this
reference in a footnote, you can use the (Author Date) method (e.g.,
(Koomey 1995)), or you can consult the *Chicago* *Manual of Style* for
more reference formats.

If you quote from another source, cite the source. If it is a very
extensive quotation (more than a few paragraphs) it may be appropriate
to obtain the author\'s permission for using it. \"Fair use\" has a
specific legal definition. You are allowed to quote excerpts from other
people\'s copyrighted work to illustrate, clarify, or comment on their
work. These excerpts cannot be excessively long, and whether it is fair
use or not depends on the context in which the excerpt is used. For more
details on copyrights in the digital age, see Carroll (1994).

The internet has made issues of credit and authorship more complex. One
friend occasionally teaches a class on environmental science, and he
found that in many term papers, three-quarters of all the sources cited
were World Wide Web addresses (called Uniform Resource Locators, or
URLs), with little or no explanation given for how the information was
derived, or whether it was credible. For the students, the mere
existence of a URL was enough reason to cite it, when in fact more
investigation and documentation is really required to ensure that the
web site gives proper credit and accurately reports results.

#### Leaving a trail for you to remember

The fancy technical name for data documentation is \"source tagging\".
Researchers who specialize in data quality focus on how to decide which
information about the source is appropriate for electronic data, and how
to attach this key information to those data. I refer to data without
attached sources \"disembodied\". I also call it useless, because
without the source, it\'s no better than a rumor.

Don\'t underestimate the importance of good archiving practice.
Companies neglect it at their peril, because mission-critical
information must be accessible to those making strategic decisions. This
task takes on greater importance in the electronic age, because the
\"paper trail\" that existed previously is now commonly absent (Cook
1995).

Keep a notebook for each major project . Record in this book important
insights you have, notes from meetings, reports to track down, or any
other items related to the progress of your work. My Newton MessagePad
serves this function for me, but if you haven\'t adopted one of these
handheld devices, then a paper notebook will serve just fine. These
notes may come in handy later if you want to build on previous work.

Always write down the full reference to an article that you\'ve copied,
as soon as you\'ve copied it. Write it on the first page of the copied
article. No exceptions! Until I learned this lesson, I can\'t tell you
how many times I had to go back and find the source because I had not
written down the reference at the moment I copied the article.

Always write down why you think a particular article is significant for
your work, because in six months you will probably have forgotten.

Always date your work, even your roughest handwritten notes.
Reconstructing the evolution of your thoughts often requires knowing
*when* you wrote something down.

If you write computer code, put extensive explanations of your logic
into the code in \"comments\". Describe your reasoning as if someone
else will be reading that code and trying to understand it, because that
is literally the case: in six months, you will be a very different
person who has forgotten the little details that led you to structure
your code in the way you did.

If you work with spreadsheets, never EVER type data directly into
spreadsheet formulas. For example, some people enter formulas that look
like this into a spreadsheet cell: =2 x 457.3/9415 + 24. Instead, enter
these numbers into cells, then create formulas that use these cells for
calculations, like this: =F2 x F3/F4 + F5 (in this case, the various
numbers are in cells F2 through F5). You can easily change the data
later without having to fiddle with the formulas, plus you can see the
input data for the calculation without having to poke around in the cell
with the formula. Such planning ahead can save you time and make it
easier to spot mistakes in the data and calculations.

Segregate input data in one part of the spreadsheet and results in
another part. Be systematic and you will be able to recreate your
calculations even a year from now, after you\'ve forgotten the
assumptions that are now so fresh in your mind.

#### Leaving a trail for others to follow

The importance of documenting your work comes to the fore when there is
some kind of a crisis. In August 1997, my boss was called before the
President\'s Council of Economic Advisors to testify about a report we
had recently completed. I had created most of the calculations
summarizing the results, and had written a detailed description of how
the calculations were done. I crafted the documentation for a technical
analyst who wanted to reproduce my calculations and who had time to
delve into depth. While Mark is such a technical analyst, it soon became
obvious that he needed something a little different, because his time
was so limited. I talked him through the calculations, and the testimony
went well. The lesson was not lost on me, however.

Assume that the person reading your documentation will be testifying on
it and has limited time to figure it out. This is the appropriate
approach when documenting, because it will ensure that you take
appropriate care. You may think that your task won\'t require such
thoroughness, but what if you have a family emergency? Will someone else
at your company be able to pick up where you left off?

Describe the calculations in such excruciating detail that even in the
worst case, an intelligent reader can understand and reproduce your
calculations as quickly as possible. Put a clear description of your
methods in the main body of your report, then relegate the complete
documentation to appendices. Such detail may seem like overkill at the
time, but you will never regret it. Documentation is one of Stephen
Covey\'s \"important but not urgent\" tasks, and one that should be a
priority in any analysis work.

Your references to other people\'s work must be complete so that others
can follow up. References must contain enough information that a reader
can find a reference at the library even in the face of the inevitable
typos and smudged copies. Redundancy is good in this case. The table
below shows the information you should include for the most commonly
referenced sources. How you choose to present this information in your
bibliography is up to you, but I recommend following the basic formats
laid out in the Chicago Manual of Style.

Computer programs that automate the process of tracking references (such
as EndNote) are one of the few innovations since the word processor to
increase significantly the productivity of writers and researchers. Use
them if you do more than a little writing. These tools can ensure that
your references are complete, and do so with a fraction of the time and
annoyance that in the past accompanied compiling a bibliography. EndNote
allows you to create one database of your references, and once you\'ve
entered them into the database, you need never enter them again. It
formats the references in any way you choose, and comes with a
collection of formats for many of the more common academic journals.
Acquire a reference database with all due haste!

######## Information needed for complete references

<table>
<colgroup>
<col style="width: 17%" />
<col style="width: 9%" />
<col style="width: 6%" />
<col style="width: 6%" />
<col style="width: 12%" />
<col style="width: 9%" />
<col style="width: 9%" />
<col style="width: 11%" />
<col style="width: 6%" />
<col style="width: 11%" />
</colgroup>
<tbody>
<tr class="odd">
<td><em>Type</em></td>
<td><em>Author</em></td>
<td><em>Year</em></td>
<td><em>Title</em></td>
<td><em>Institution</em></td>
<td><em>Volume</em></td>
<td><em>Number</em></td>
<td><em>Page Number</em></td>
<td><em>Date</em></td>
<td><em>Other</em></td>
</tr>
<tr class="even">
<td><em>Book</em></td>
<td>Yes</td>
<td>Yes</td>
<td>Yes</td>
<td>Publisher</td>
<td>Edition</td>
<td></td>
<td>If applicable</td>
<td></td>
<td>City</td>
</tr>
<tr class="odd">
<td><em>Edited Book</em></td>
<td>Yes</td>
<td>Yes</td>
<td>Yes</td>
<td>Publisher</td>
<td>Edition</td>
<td></td>
<td>If applicable</td>
<td></td>
<td>City</td>
</tr>
<tr class="even">
<td><em>Book Section</em></td>
<td>Yes</td>
<td>Yes</td>
<td>Yes</td>
<td>Publisher</td>
<td>Edition</td>
<td></td>
<td>Yes</td>
<td></td>
<td><p>Book title,</p>
<p>City</p></td>
</tr>
<tr class="odd">
<td><em>Journal Article</em></td>
<td>Yes</td>
<td>Yes</td>
<td>Yes</td>
<td>Journal name</td>
<td>Yes</td>
<td>Yes</td>
<td>Yes</td>
<td>Yes</td>
<td></td>
</tr>
<tr class="even">
<td><em>Magazine Article</em></td>
<td>Yes</td>
<td>Yes</td>
<td>Yes</td>
<td>Magazine name</td>
<td>Yes</td>
<td>Yes</td>
<td>Yes</td>
<td>Yes</td>
<td></td>
</tr>
<tr class="odd">
<td><em>Newspaper Article</em></td>
<td>Yes</td>
<td>Yes</td>
<td>Yes</td>
<td>Newspaper name</td>
<td></td>
<td></td>
<td>Yes</td>
<td>Yes</td>
<td><p>Section,</p>
<p>City</p></td>
</tr>
<tr class="even">
<td><em>Report</em></td>
<td>Yes</td>
<td>Yes</td>
<td>Yes</td>
<td>Name of institution</td>
<td></td>
<td></td>
<td>If applicable</td>
<td>Yes</td>
<td>City</td>
</tr>
<tr class="odd">
<td><em>Conference proceedings</em></td>
<td>Yes</td>
<td>Yes</td>
<td>Yes</td>
<td>Conference organizer</td>
<td>Yes</td>
<td></td>
<td>Yes</td>
<td>Yes</td>
<td>Location</td>
</tr>
<tr class="even">
<td><em>Thesis</em></td>
<td>Yes</td>
<td>Yes</td>
<td>Yes</td>
<td>University</td>
<td></td>
<td></td>
<td>If applicable</td>
<td>Yes</td>
<td>Thesis type, University location</td>
</tr>
<tr class="odd">
<td><em>Personal Communication</em></td>
<td>Yes</td>
<td>Yes</td>
<td>Yes</td>
<td>Name of institution</td>
<td></td>
<td></td>
<td>If applicable</td>
<td>Yes</td>
<td>Address, phone #, fax #, email</td>
</tr>
</tbody>
</table>

#### Checking your work

You wouldn\'t put a computer system into a client\'s office without
first testing it, would you? Why then do so many people produce analysis
without checking the results for internal consistency, completeness, and
congruence with the stated goals of the people requesting the analysis?
An important function of the documentation process is to help conduct
this testing. I find that explaining my methods in detail helps me
determine where I may have made conceptual blunders. If I can\'t explain
the reasoning behind my methods, then I know I have more thinking to do.

Documentation should be an integral part of your analysis process. As
you are conducting your research, you should write down your methods or
try to explain them to a colleague. Your analyses will be the better for
this give and take.

#### If you are working with dollars[^84]

The value of money is affected by inflation. Over time, inflation makes
each dollar worth less, so a dollar spent in 1997 is NOT the same as a
dollar spent in 1990. You therefore must document the year in which
expenditures occur.

Any time you present tables or figures that include dollars, you *must*
specify \"current dollars\" or \"constant dollars in some year\".
Current dollars imply that if the money is spent in 1990 it is in 1990
dollars, and if spent in 1997 it is in 1997 dollars. Alternatively (and
more accurately in my view), you can correct for inflation using an
inflation index (such as the Consumer Price Index or the GDP deflator)
from the Statistical Abstract of the U.S. (see Facts at Fingertips
section) and calculate all expenditures in constant dollars. The
expenditures in current dollars in 1990 would then be converted to 1997
dollars by multiplying it by the following factor:

.

If the inflation index in 1997 is 1.5 and the inflation index in 1990 is
1.2, then expenditures in 1990 must be multiplied by or 1.25 to convert
them to 1997 dollars. All other years would be treated similarly using
the appropriate inflation index (of course, 1997 expenditures are
already in 1997 dollars!).

Correcting for inflation, while simple, is commonly overlooked. When the
postal service announced in May 1998 that they would be granted an
increase in the price of a first class stamp in 1999, the San Francisco
Chronicle presented a summary of the new rates, and printed a graph of
first class postal rates since 1885.[^85] This graph showed a huge
increase in prices, from 2¢ in 1885 to 33¢ in 1999. An uninformed
observer might easily have concluded from this graph that the price of
first class mail had skyrocketed, particularly since the 1970s. The
problem, of course, is that the graph was not corrected for inflation,
and nowhere did the Chronicle article mention this. The Oakland Tribune
story on the same topic[^86] correctly reproduced two graphs, one in
current dollars and one in inflation-adjusted dollars. The second graph
showed that the 33¢ price of first class mail in 1999 actually
represented a decline in constant dollars from 35¢ in 1885.

![](media/image21.wmf){width="2.5555555555555554in"
height="4.291666666666667in"}
![](media/image22.wmf){width="2.4305555555555554in"
height="4.333333333333333in"}

I once found a report that contained information on the costs of
computing power in the 1950s and 1960s, and at first I was delighted. My
delight turned to dismay when I realized that the report did not
document whether the costs were current or constant dollars, and I was
unable to use the information because I couldn\'t determine how to treat
the costs. What a waste! Don\'t let this happen to your work.

#### Eschew the passive voice

When documenting or writing reports, take responsibility by using the
active voice. For example, say \"I estimated the following parameters\"
instead of \" the following parameters were estimated\". Make it crystal
clear who did what, so that there is no ambiguity. The passive voice is
the tool of people who want to avoid responsibility for their actions,
while the purpose of documentation is to assign responsibility, for all
the reasons listed above. Don\'t thwart your efforts by using ambiguous
passive constructions.

#### Documentation pays off

Any time there is a new \"hot\" analysis topic, people with little prior
knowledge jump in. You can perform an immense service to any debate (and
garner attention to your point of view) simply by being systematic in
identifying and investigating such topics, and documenting them in a
careful way.

A case in point is the raucous debate over the cost of pollution to
society from power plants in the late 1980s and early 1990s.[^87] I
wrote a paper that was not a conceptual breakthrough but simply created
consistent comparisons (LINK) between different estimates of these
social costs.[^88] I have received hundreds of requests from all over
the world for this admittedly rather dry technical report, as well as
offers to speak at conferences and opportunities to contribute to other
reports on the topic. It garnered attention in large part because of its
systematic and careful documentation. You too can use this approach to
your advantage.

#### Back up your data!

Many analysts don\'t routinely back up their electronic files. This
oversight is potentially disastrous, as the otherwise technology-savvy
Stanford school of business found out in April 1998. Two large computers
were moved before the operators verified that all data had been backed
up, and almost a dozen researchers lost years of work.[^89] Hard disks,
floppy disks, and data tapes don\'t last forever, and they are
guaranteed to fail at the least opportune time. I have never lost any
vital data (because I\'m more methodical than most), but I have many
colleagues who have, and it\'s never fun. So after you\'ve gone to the
trouble to carefully document your work, make sure that there are at
least a couple of electronic copies, and put a paper version in a safe
place. One clever trick is to email electronic files to yourself, which
places them on the server of your internet service provider and gives
you an additional layer of redundancy. Also don\'t forget that data
formats change over time, so if you have some older files, convert them
to a new format before trashing the program that created them.

#### Conclusions

I have been repeatedly shocked by the lack of awareness of good data
documentation practices. I believe these skills should be taught in high
school, yet technically-trained college graduates routinely do not
identify their data sources or the methods they used for their
calculations! Don\'t fall into this trap. Make sure that any intelligent
person (including YOU!) can reproduce your analysis from the footnotes.
With this one simple step, you will vault above 95% of the world\'s
other analysts. Leaping over the other 5% is up to the merits of your
content!

#### Further reading

*The Chicago Manual of Style: The Essential Guide for Writers, Editors,
and Publishers* , 1993. 14th ed. Chicago, IL: University of Chicago
Press. The classic reference on style and formats, including discussion
of spelling, punctuation, grammar, tables, abbreviations, numbers,
foreign languages, and other complexities afflicting the modern writer.

Cook, Terry. 1995. \"It\'s 10 O\'Clock: Do You Know Where Your Data
Are?\". *Technology Review*. January. p. 48. An eye-opening article that
discusses the real world problems with keeping track of data in the
electronic age.

Carroll, Terry. 1994. *Frequently Asked Questions About Copyright (v.
1.1.3); Part 2: Copyright Basics*. Obtained over the internet
(disclaimer in work states that \"it may be freely redistributed in its
entirety provided that this copyright notice is not removed\"). Last
update: January 6, 1994.

> \"Results! Why, man, I have gotten a lot of results. I know several
> thousand things that won\'t work.\"
>
> \--Thomas A. Edison\--[^90]

### Let the tables and graphs do the work

When writing, do your tables and graphs first, then write around them.
In any writing that relies on quantitative analysis, the calculations
are the bulk of the work. I always create the tables documenting data
and presenting results first, then I write text describing the inputs
and analysis afterwards. For particularly important results, I then make
graphs that best emphasize the points I want to make.

As an example of the right way to summarize technical analysis, see the
text below:

In any economic or environmental analysis, remember that people\'s time
has value, a value that often far exceeds that of other inputs to the
production of goods and services. Labor is a resource that has a real
economic cost. Don\'t ignore it.

Consider the following case: many hotels and multifamily buildings have
incandescent lights in their hallways that operate all the time. These
lamps can be replaced by compact fluorescent lamps (CFLs) that use about
two-thirds less energy and last ten times as long. The modern CFLs start
instantly and have excellent light quality. They typically cost about
\$15 versus about \$0.50 for an incandescent bulb.

**Table 1** summarizes the cost-effectiveness analysis for the CFL in
this application. The energy savings alone are worth about \$26, and the
annual cost for purchasing the CFL (minus the cost of the incandescent
bulbs that you need not purchase because of the CFL\'s longer lifetime)
is about \$9. The investment therefore pays for itself in about 4
months.

When you add the value of labor saved, this excellent investment becomes
a stunning one. The labor savings totally offset the annual investment
costs, so the energy savings of \$25 per year are essentially free.

######### Table 1: Cost-effectiveness analysis of compact fluorescent lamps

+--------------+-------------+--------+---------------+---------------+
|              |             |        |               |               |
+--------------+-------------+--------+---------------+---------------+
|              | *Units*     | *I     | *Compact      | *Difference   |
|              |             | ncande | fluorescent*  | (CFL - Inc)*  |
|              |             | scent* |               |               |
+--------------+-------------+--------+---------------+---------------+
|              |             |        |               |               |
+--------------+-------------+--------+---------------+---------------+
| Power of     | Watts/bulb  | 60     | 18            | -42           |
| bulb         |             |        |               |               |
+--------------+-------------+--------+---------------+---------------+
|              |             |        |               |               |
+--------------+-------------+--------+---------------+---------------+
| Operating    | Hours/      | 8760   | 8760          | 0             |
| time         | socket/year |        |               |               |
+--------------+-------------+--------+---------------+---------------+
| Lifetime     | Hours/bulb  | 1000   | 10000         | 9000          |
+--------------+-------------+--------+---------------+---------------+
| Bulbs        | Bulbs/      | 8.8    | 0.9           | -7.9          |
| installed    | socket/year |        |               |               |
+--------------+-------------+--------+---------------+---------------+
|              |             |        |               |               |
+--------------+-------------+--------+---------------+---------------+
| Bulb cost    | \$/bulb     | 0.5    | 15.0          | 14.5          |
+--------------+-------------+--------+---------------+---------------+
| *Annual bulb | *\$/s       | *4.4*  | *13.1*        | *8.8*         |
| cost*        | ocket/year* |        |               |               |
+--------------+-------------+--------+---------------+---------------+
|              |             |        |               |               |
+--------------+-------------+--------+---------------+---------------+
| Annual       | kWh/        | 526    | 158           | -368          |
| energy use   | socket/year |        |               |               |
+--------------+-------------+--------+---------------+---------------+
| Price of     | \$/kWh      | 0.07   | 0.07          | 0             |
| electricity  |             |        |               |               |
+--------------+-------------+--------+---------------+---------------+
| *Annual      | *\$/s       | *37*   | *11*          | *-26*         |
| energy cost* | ocket/year* |        |               |               |
+--------------+-------------+--------+---------------+---------------+
|              |             |        |               |               |
+--------------+-------------+--------+---------------+---------------+
| **Total      | **\$/so     | **41** | **24**        | **-17**       |
| cost**       | cket/year** |        |               |               |
|              |             |        |               |               |
| **w/o        |             |        |               |               |
| labor**      |             |        |               |               |
+--------------+-------------+--------+---------------+---------------+
|              |             |        |               |               |
+--------------+-------------+--------+---------------+---------------+
| Labor to     | m           | 3      | 3             | 0             |
| change bulbs | inutes/bulb |        |               |               |
+--------------+-------------+--------+---------------+---------------+
| Annual labor | hours/      | 0.44   | 0.04          | -0.39         |
|              | socket/year |        |               |               |
+--------------+-------------+--------+---------------+---------------+
| Labor rate   | \$/hour     | 25     | 25            | 0             |
| (1)          |             |        |               |               |
+--------------+-------------+--------+---------------+---------------+
| *Annual      | *\$/s       | *11.0* | *1.1*         | *-9.9*        |
| labor cost*  | ocket/year* |        |               |               |
+--------------+-------------+--------+---------------+---------------+
|              |             |        |               |               |
+--------------+-------------+--------+---------------+---------------+
| **Total cost | **\$/so     | **52** | **25**        | **-27**       |
| (2)**        | cket/year** |        |               |               |
+--------------+-------------+--------+---------------+---------------+
|              |             |        |               |               |
+--------------+-------------+--------+---------------+---------------+

\(1\) Labor rate includes overhead.

\(2\) Total cost is the sum of annual bulb cost, annual energy cost, and
annual labor cost.

Now compare it to the text below that doesn\'t follow the approach I
advocate:

In any economic or environmental analysis, remember that people\'s time
has value, a value that often far exceeds that of other inputs to the
production of goods and services. Consider the following
cost-effectiveness calculation: many hotels and multifamily buildings
have incandescent lights in their hallways that operate 24 hours per
day, 365 days per year. These lamps use about 60 Watts and last 1000
hours. When burned continuously, such bulbs, (which cost \$0.50 to \$1
each) must be replaced almost nine times per year. Compact fluorescent
lamps (CFLs) last about 10,000 hours and use one quarter to one third of
the electricity of incandescents (18 Watts in this example). The modern
CFLs start virtually instantly and have excellent light quality. They
typically cost about \$15.

The annual energy saved from the CFL is

(60 Watts-18 Watts) x x 8760 hours/year = 368 kWh/year.

Typical commercial electricity prices are about \$0.07/kWh, so these
annual savings are worth 368 kWh x \$0.07/kWh = \$26/year. This annual
savings is purchased for an annual incremental capital cost of
\$15\*8,670 hours /10,000 hours minus 8.8 bulbs x \$0.50/bulb, or \$9.
The CFL therefore pays for itself in about 4 months (x 12 months/year).

When you add the value of labor saved, this excellent investment becomes
a stunning one. Say it takes 3 minutes to pull out a ladder and change a
burned out bulb. To change a bulb 9 times takes 27 minutes each year.
The CFL only needs to be installed once, and it also takes 3 minutes.
The additional labor for the incandescents is therefore 24 minutes/year.
If the labor cost is \$25/hour with overhead, the \$10 value of that 24
minutes of labor (\$25 x ) entirely offsets the \$10 incremental capital
cost. The energy savings of \$25 per year are free, thanks to the labor
saving benefits.

This text is potentially intimidating to the reader who is less
comfortable with numbers, and it is also less compact. In addition, if
the calculations change, you\'ll have to manually change the text to
reflect those calculations. If the calculations are in a table printed
from a spreadsheet, you merely change the calculations in the table,
print it out again or copy it into your text, and change the few key
numbers in the text that refer to the table. This approach is far less
labor intensive when you need to make changes (and you almost always
need to make changes at some point!).

By segregating the numbers in a table, you make the text easier to read
and also make it simpler to write. The text should call out the most
important numbers in the table for ease of comprehension. The numbers
junkies can delve into the table for any details they feel compelled to
unearth.

The main purpose of writing of this type is to present, in as clear a
fashion as possible, the logic of the calculations, the main input data,
and the key results. Any competent analyst ought to be able to reproduce
your analysis using the information contained in your report. If they
can\'t, your work isn\'t done. An astounding number of analysts
routinely omit vital data and assumptions from their reports, and there
can be no excuse for this.

If the analysis is well thought out, the tables and graphs well
designed, and the audience clearly defined, the report should
practically write itself. I normally use the following main headings in
technical reports (your specific needs will probably dictate a slightly
different structure):

*Introduction*

*Methodology and Data*

*Results*

*Discussion*

*Limitations of this analysis*

*Future work*

*Conclusions*

*References*

*Appendices*

Using this method, the writing task becomes one of straightforward
description, with the exception of the \"Discussion\" section where you
describe why the results are important.

Place detailed calculations and supporting data in appendices. Only put
*essential* information in the main body of the report. I try to write
relatively short technical reports that summarize the analysis results,
with complete documentation relegated to appendices. It is then a simple
matter to take the short report, eliminate all but the three to five
most crucial graphs and tables, change some text, and send it off to a
journal for publication.

One trick that helps the reader is to bold-face the first mention of
Tables and graphs in the text, so that it is obvious where they are
discussed and explained. For example: \"**Table 1** shows the
results\...\". This trick makes it easier for those readers who jump
straight to the results and want to comprehend the analysis quickly.
There are many such readers, which also makes it essential that each
table and figure have footnotes that explain the key elements of the
supporting calculations.

#### Further reading

Booth, Wayne C., Gregory G. Colomb and Joseph M. Williams. 1995. *The
Craft of Research*. Chicago, IL: The University of Chicago Press. One of
the best books around on the process of writing research papers, from
start to finish.

Strunk Jr., William, and E. B. White. 1979. *The Elements of Style*, 3d
ed. New York, NY: MacMillan Publishing Company, Inc. The basic *Elements
of Style* hasn\'t changed in decades (it\'s one of those gems that
simply can\'t be improved upon). Plan on rereading this short but
essential book every few years, just to refresh your memory and enliven
your writing.

*The Chicago Manual of Style: The Essential Guide for Writers, Editors,
and Publishers*. 1993. 14th ed. Chicago, IL: University of Chicago
Press. Many writers consider this book to be the last word in rules for
writing. It\'s a comprehensive source, and one that\'s well worth having
on your shelf.

> \"Don\'t just write to be understood; write so that you cannot be
> misunderstood.\"
>
> \--Robert Louis Stevenson\--

### Designing compelling graphs and figures

Modern graphs are largely an Eighteenth century creation. Edward Tufte
(1990, 1995, 1997)), the modern master of graphic representation, states
that Playfair, Lambert, and others developed the first graphs that broke
away from a geographical representation. In the two centuries since,
such charts have become commonplace. With the advent of modern computer
tools, it has become trivial to create graphs from data. In fact, it has
probably become too easy! Graphs are often produced without thought for
their main purpose: to enlighten and inform the reader.

Tufte (1995) defines graphical excellence as \"that which gives to the
viewer the greatest number of ideas in the shortest time with the least
ink in the smallest space\". He promotes several rules for achieving
such excellence that will help you avoid the most common pitfalls:

• *Above all else show the data*: Make sure the graphs highlight your
results instead of hiding them as many graphs do.

• *Maximize the data-ink ratio*: Tufte defines the data-ink ratio as the
amount of ink that conveys information divided by the total amount of
ink used for the graph. Many graphs have low data-ink ratios, and you
should strive to avoid this pitfall. Erase non-data-ink and redundant
data-ink.[^91]

• *Eliminate chart junk*: Common mistakes include

-moiré patterns, which are hard on the eyes;

-too many gridlines, which obscure the data; and

-graphical elements that decorate but convey no information.

• *Maximize data density (numbers per square inch)*: Typical graphs in
newspapers and journals, based on Tufte\'s review of hundreds of charts
published 1970-1980, have data densities of 5 to 20 numbers per square
inch (0.8 to 3 numbers per square centimeter). The densest graph Tufte
examines (a map of galaxies contributed by some astronomers) has a data
density of 110,000 numbers per square inch (17,000 numbers per square
centimeter), while the least dense graph has a data density of 0.15
numbers per square inch (0.02 numbers per square centimeter). Maps tend
to have high data density, while badly designed charts tend to have the
lowest.[^92]

• *Make them stand alone*: Figures are often copied separately from the
report from which they came, so make sure the reader can trace your
figure back to the source.

• *Revise and edit*: All work improves with revisions. Leave enough time
to get feedback and make it better.

Tufte (1995) reminds us that rules should not be followed slavishly, and
that our focus should be on the key goal: \"\...to give visual access to
the subtle and the difficult\". Graphics should serve your arguments. If
they don\'t support those arguments in a direct and understandable way,
they should be ruthlessly deleted or drastically modified. Graphs and
Figures that don\'t contribute to the \"revelation of the complex\" have
no place in your work.

#### Further reading

Craver, John S. 1980. *Graph Paper From Your Copier*. Tucson, AZ:
HPBooks. If you need a special kind of graph paper in a pinch, this book
is just the ticket, with almost 200 different graph and map types, ready
for photocopying and use. It also has a nice 20+ page write-up of hints
for making good graphs. It was especially useful before computer
graphing became so widespread, but it still can come in handy.

Jones, Gerald E. 1995. *How to Lie with Charts*. San Francisco, CA:
Sybex. Mainly oriented towards users of computer graphing tools, this
book contains dozens of specific recommendations for making effective
use of charts and graphs.

Monmonier, Mark. 1996. *How to Lie with Maps*. Chicago, IL: University
of Chicago Press. This book contains examples of maps that mislead,
confuse, advocate, and advertise. If you work with maps a lot, it\'s a
\"must read\".

Munter, Mary. 1987. *Guide to Managerial Communication*. 2d Englewood
Cliffs, NJ: Prentice Hall, Inc. A business-oriented guide to getting
your point across.

Tufte, Edward R. 1990. *Envisioning Information*. Cheshire, CT: Graphics
Press. Tufte says this book is about \"*pictures of nouns*\", namely
graphical representations of places, things, people, and animals. It
also deals with \"visual strategies for design: color, layering, and
interaction effects.\"

Tufte, Edward R. 1995. *The Visual Display of Quantitative Information*.
Cheshire, CT: Graphics Press. Fourteenth printing. Tufte says this book
is about \"*pictures of numbers*, how to depict data and enforce
statistical accuracy\". It is by far the best book from which to learn
the subtleties of effective graph design.

Tufte, Edward R. 1997. *Visual Explanation: Images and Quantities,
Evidence and Narrative*. Cheshire, CT: Graphics Press. Tufte says this
book is about \"*pictures of verbs*, the representation of mechanism and
motion, of process and dynamics, of causes and effects, of explanation
and narrative\". It focuses on the use of graphics to explain events and
make explanatory assertions about those events.

Zelazny, Gene. 1996. *Say It With Charts: The Executive\'s Guide to
Visual Communication*. 3d edition. Chicago, IL: Irwin Professional
Publishers. A fine practical guide to using graphs in the business
world.

> \"Visual representations of evidence should be governed by principles
> of reasoning about quantitative evidence. For information displays,
> design reasoning must correspond to scientific reasoning. Clear and
> precise seeing becomes as one with clear and precise thinking\"
>
> \--Edward Tufte[^93]\--

### Designing good tables

A good table is a work of art, while a bad one is worse than useless. I
have worked to refine my table-making skills over the years, and am
constantly amazed at the perfectly dreadful tables created by authors
who should know better. A little care can make your tables a resource
that your readers keep as a reference for many years.[^94]

I follow several key principles in creating a table:

• *Make them stand alone*: tables are often photocopied separately from
the report from which they came. Make sure the reader can trace your
table back to the source. Footnotes to the table should explain the
methodology, list the key sources, and describe how each source was
used. The reader should be able to recreate the table from your
footnotes.

• *Maximize the data-ink ratio*: Tufte\'s guidance is just as relevant
for tables as for graphs. Make sure that every part of the table conveys
information. Avoid extraneous gridlines (only include them when they
enhance reader comprehension.

• *Some redundancy is good*: Unlike for graphs, where redundancy should
be eschewed, showing additional columns or rows for totals and subtotals
is essential. These ostensibly redundant sums help the reader follow the
calculation and assure her that the table is internally consistent. The
sums can also help in troubleshooting your own calculations.

Another important redundancy relates to percentages. If you have a
column of percentages, label the column heading \"% of total\" AND put a
percent symbol in every cell (e.g., 1.37%, not 1.37). Sometimes people
make mistakes in their calculations and forget to divide by 100 to make
a real percentage. By enforcing this redundancy, you guarantee that
readers will interpret your results correctly and you\'ll save them
time. Having the % symbol in every cell makes the meaning utterly
unambiguous.

Finally, all numbers less than zero should have a leading zero (e.g.,
0.35, NOT .35). Copies become smudged and decimal points can be
obscured. With a leading zero there will never be ambiguity about the
size of the number in question.

• *Enhance readability with font choice*: Choose a font that is easy on
the eyes and doesn\'t deteriorate as a document is copied. Tufte
recommends using the font used in telephone books (Bell Centennial)
whenever you have tables that have high information density, because the
phone companies have researched this topic extensively, and there\'s no
point in reinventing the wheel.

• *Enhance reader comprehension*: The single most important tool to
increase usefulness of tabular data is to add rows or columns containing
indices relative to a total or subtotal. Most readers will calculate
such indices themselves in their effort to understand the table. Do it
for them, and they will thank you for it.

• *Create extensive footnotes*: This aspect of documentation (LINK) is
utterly crucial. All tables should have footnotes sufficient to allow
any intelligent person to understand the method used to create the
analysis. These footnotes can refer to other sources, but they should be
detailed enough that the basic methodology is clear. Tufte says that
footnotes indicate \"care and craft\", and readers will thank you for
taking the time to make such care manifest in footnotes.

• *Create the table first, then write your text from the tables (LINK)*:
Creating the tables helps me think through the analysis and ensures that
my logic is both correct and reproducible. Ask yourself questions about
the analysis and check your work. If I can\'t explain it in writing,
then I don\'t understand it. Clear tables and clear writing come from
clear thinking.

• *Revise and edit*: Show your tables to colleagues before publication
and you will surely learn something. What is clear and obvious to you
may not be so obvious to them. Give yourself time to revise and improve
the tables.

Your ultimate goal should be to make your tables so good that people
treasure them and make your reports enduringly useful.

#### Further reading

Jones, Gerald E. 1995. *How to Lie with Charts*. San Francisco, CA:
Sybex. Has a nice section on table design.

Munter, Mary. 1987. *Guide to Managerial Communication*. 2d Englewood
Cliffs, NJ: Prentice Hall, Inc. A business-oriented guide to getting
your point across.

Tufte, Edward R. 1990. *Envisioning Information*. Cheshire, CT: Graphics
Press.

Tufte, Edward R. 1995. *The Visual Display of Quantitative Information*.
Cheshire, CT: Graphics Press. Fourteenth printing.

Tufte, Edward R. 1997. *Visual Explanation: Images and Quantities,
Evidence and Narrative*. Cheshire, CT: Graphics Press.

> \"There are two ways of constructing a software design \[or a table,
> for that matter\]: one way is to make it so simple that there are
> obviously no deficiencies; the other is to make it so complicated that
> there are no obvious deficiencies.\"
>
> \--C. A. R. Hoare\--[^95]

### Using numbers effectively in oral presentations

Charles Osgood has a simple rule about using numbers in oral
presentations: DON\'T! That you are reading this book indicates an
interest in numbers that is more than a casual one, and your talks are
perhaps more likely to involve numbers than most. However, Osgood\'s
rule is a good one to consider, even if your subject is a technical one.
Most people can\'t absorb numbers and their implications without
considerable hand-holding.

If you feel that numbers will enhance your argument, then summarize the
few key numbers in graphic form. Only rarely use a table, and only then
when it\'s a small one that contains the key information in large,
readable type. If you have detailed data and calculations supporting
your analysis, then give hard-copy handouts to your listeners. Summarize
the conclusions from the results and focus on the conclusions, **not**
on the numbers themselves.

Common doctrine is that data are dull. But as Edward Tufte points out,
\"if the statistics are boring, then you\'ve got the wrong numbers\".
You should use numbers to make an important point. If your audience is
not convinced of its importance and value, then ***you*** are
responsible for presenting the relevant information in an engaging way.
Data are dull only when chosen poorly and presented badly.

**Never** present unreadable data on vugrafs or slides. Let me repeat:
**Never** present unreadable data on vugrafs or slides. I\'ve seen
numerous veteran professors do it, but it\'s insulting to the audience.
You\'re the expert, and you should pre-digest the information for them
and make it accessible. It\'s not just common courtesy, it\'s also the
best way to ensure that your main points are most effectively conveyed.

Jones (1995) gives guidelines on the size and kind of text to use in
presentations, and describes the conventions for measuring font size and
spacing. He quotes the Society of Motion Picture and Television
engineers, who advocate that the minimum height of small letters used in
on-screen text should be 2.5 percent of the height of the screen.
Typical presentation screens are 1.5 meters (5 feet) high, so the
minimum font size in such a screen should be about 4 centimeters (1.5
inches).

Another way to judge type size of a vugraf is to place it at your feet.
If you can\'t read it while standing, the type is too small.

A successful speaker always thinks carefully about the following
questions:

Who is in the audience?

What is important to them?

What are the three key messages you want people to learn from your talk?
(with rare exceptions, that\'s all they will ever take away anyway).

While Ross Perot\'s use of charts and tables did not earn him electoral
victory in 1992, he demonstrated that a speaker who addresses issues the
audience cares about can be effective and entertaining even if he uses
more graphs per minute than in any television show in history.

#### Further reading

Jones, Gerald E. 1995. *How to Lie with Charts*. San Francisco, CA:
Sybex. Gives specific advice on using tables and graphs in oral
presentations.

Osgood, Charles. 1988. *Osgood on Speaking: How to Think on Your Feet
Without Falling on Your Face*. New York, NY: William Morrow and Company,
Inc. One of the best books around for teaching the art of public
speaking.

Tufte, Edward R. 1997. *Visual Explanation: Images and Quantities,
Evidence and Narrative*. Cheshire, CT: Graphics Press. In this book,
Tufte gives derives lessons from magicians and illusionists to teach
good presentation skills (see pp.68-70).

> \"When speakers use a lot of numbers, the audience most always
> slumbers.\"
>
> \--Charles Osgood\--

# Some Parting Advice: Creating the future

Like the butterfly\'s wing beat that ultimately leads to a hurricane,
seemingly trivial actions can have consequences that reverberate through
generations. With every action, with every day we live, we create the
future. Of course, forces beyond human control also have influence, but
it is how our choices relate to these external events that determine the
outcome.

Mastering the art of analysis is one important way to ensure that the
future you create is a hopeful one. Analysis helps you make choices that
are consonant with your personal values, consistent with your ideology,
and reflective of the realities imposed by external forces. Whether you
are a businessperson, scientist, athlete, or accountant, analyzing your
choices in a systematic way will lead to a better life.

I leave you with the environmentalist Edward Abbey\'s excellent advice
for those who are trying to change the world: \"Do not burn yourself
out. Be as I am - a reluctant enthusiast, a part time crusader, a half
hearted fanatic. Save the other half of yourselves and your lives for
pleasure.\" You can\'t heal the earth or help others unless you yourself
are whole. Care for yourself, and those around you. Take the time to do
those things that replenish your energy and fill your heart with joy.
Your children and our posterity will thank you for it.

> \"The future is where we will spend the rest of our lives.\"
>
> \--Anonymous\--

![](media/image23.wmf){width="6.930555555555555in"
height="8.527777777777779in"}

## Appendix A: Data Sheets

Compile the most useful numbers in pre-designed data sheets for
different types of users. Make these data sheets available on the web.
Solicit comments from users (business, energy/environment, teachers).

## Appendix B: annotated bibliography

Add an annotated bibliography at the end, just copy each section\'s
further readings with the title of the section, so they are all
collected in one place.

[^1]: ^ ^ Millman, Dan. 1984. *Way of the Peaceful Warrior*. Tiburon,
    CA: H. J. Kramer, Inc.

[^2]: I am indebted to Amory Lovins for this phrase.

[^3]: Stoll, Clifford. 1996. *Silicon Snake Oil*. New York, NY: Anchor
    Books, p. 194

[^4]: By the early 1990s, the LoC collection totalled more than 100
    million items (URL). For this calculation I assume that eighty
    percent of these items are duplicate copies of existing documents,
    so that our hapless reader would only have to peruse 20 million
    volumes.

[^5]: An \"order of magnitude\" is a factor of ten, two orders of
    magnitude is a factor of one hundred, and so on. This shorthand is
    commonly used by scientists to summarize large differences in a
    convenient way.

[^6]: 280 pages times 300 words per page times 6 characters per word

[^7]: The T3 line transmits 45 million bits per second. Each character
    uses 10 bits if it uses the straight ASCII format (8 bits per
    character (byte) plus error checking bits). A bit is the simplest
    unit of information. If we use instead the unicode standard (which
    would make each character use 18 bits) then the transmission rate of
    the T3 line would be reduced by almost a factor of two. Even with
    this correction, however, it would still be more than eight hundred
    thousand times faster than the transmittal of information by book
    via boat discussed in the text.

[^8]: Metcalfe, Bob. 1996. \"Computer laws galore, but one is holding
    back the Information Age\". *Infoworld*. vol. 18, no. 19. May 6,
    1996.

[^9]: This way of expressing large numbers is called scientific
    notation. For a nice explanation of how to use scientific notation
    to deal with large numbers, see Hershey, Robert L. 1982. *How to
    Think with Numbers*. Los Altos, CA: W. Kaufmann and Hershey, Robert
    L. 1997. \"Thinking Big\". *The Washington Post*, September 10,
    1997, p. H8.

[^10]: Robert W. Keyes. \"The Future of the Transistor\". *Scientific
    American*. June 1993. vol. 268, no. 6. pp. 70-78.

[^11]: Markoff, John. 1998. \"Crowding more data into even smaller
    spaces\". *New York Times* (Cybertimes). February 23, 1998. Read
    on-line at
    http://www.nytimes.com/library/cyber/week/022398diskdrive.html.

[^12]: Get Tod\'s reference (Emile Durkheim, *The Division of Labor in
    Society*).

[^13]: In one survey, 80 percent of respondents \"felt driven to gather
    as much information as possible, but over half of them were unable
    to handle all the information they accumulated\". \"Survey\" New
    \'dataholics\' generation on rise\". San Jose Mercury News. 8
    December 1997. Read on-line at
    http://www.sjmercury.com/breaking/docs/009611.htm.

[^14]: Of course, where you copy something exactly, you should obey the
    copyright laws and give appropriate credit where credit is due.

[^15]: as quoted in Hyams, Joe. 1982. *Zen in the Martial Arts*. New
    York, NY: Bantam Books., p.11

[^16]: Quoted in Arnold, Ken and James Gosling. 1998. *The Java
    Programming Language, Second Edition*. Reading, MA: Addison-Wesley,
    p. 137.

[^17]: While I developed the concept of the cycle of action
    independently from Norman (1990), p.47, I immediately recognized his
    formulation as superior to my own. I subsequently adopted his
    framework and modified it to be more applicable to the topic at
    hand.

[^18]: Coffee, Peter. 1997. \"Don\'t get sidetracked by trifling
    differences\". *PC Week*. 24 November 1997. p. 32.

[^19]: Of course, changing someone\'s opinions or attitudes can also
    affect their behavior, but is not guaranteed to do so.

[^20]: Quoted in Stobough, Robert, Daniel Yergin, I. C. Bupp, Mel
    Horwitch, Sergio Koreisha, Modesto A. Maidique and Frank Schuller.
    1979. *Energy Future*. New York: Ballantine Books.

[^21]: Krause, Florentin, Eric Haites, Richard Howarth and Jonathan
    Koomey. 1993. **Energy Policy in the Greenhouse**. Volume II,
    Part 1. *Cutting Carbon Emissions--Burden or Benefit?: The Economics
    of Energy-Tax and Non-Price Policies*. El Cerrito, CA: International
    Project for Sustainable Energy Paths.

[^22]: E.F. Schumacher. 1964. Review of EEC report on fuel and energy by
    the European Coal and Steel Community. *The Economic Journal*,
    March: 192, as cited by Keepin in his IIASA working paper, p. 56.
    Keepin, Bill. 1983. *A Critical Appraisal of the IIASA Energy
    Scenarios*. International Institute for Applied Systems Analysis.
    WP-83-104. October. Need to get Keepin\'s permission before citing

[^23]: GET EXAMPLES.

[^24]: The general principals can be derived using either deductive or
    inductive logic.

[^25]: Edwards, Paul, ed. 1972. *The Encyclopedia of Philosophy*. New
    York, NY: Macmillan Publishing Co., Inc. & the Free Press. Volume 4,
    p. 169.

[^26]: Quoted in Robert Pirsig. 1981. *Zen and the Art of Motorcycle
    Maintenance*. NY: Bantam Books, p.99.

[^27]: I am indebted to John Holdren for this phrase.

[^28]: Vol. I, p. 159, A Study in Scarlet.

[^29]: Business Week approx 1 Aug 97\--I read it on the way back from
    colorado.

[^30]: Whether it is cheaper or not depends on whether you consider only
    initial costs or total costs of ownership. If you consider the
    latter, Macs are significantly less expensive.

[^31]: I am indebted to Bill Ahern for prompting me to ask these
    questions of myself.

[^32]: Public Document Format, which is a standard format created by
    Adobe using their Acrobat Program. They distribute the reader for
    free to anyone who wants to download it.

[^33]: Quoted in Arnold, Ken and James Gosling. 1998. *The Java
    Programming Language, Second Edition*. Reading, MA: Addison-Wesley,
    p. 103.

[^34]: In fairness, I, as a twenty year old male, made my fair share of
    wild assertions in those days.

[^35]: NTIS, 800/553-NTIS, or http://www.ntis.gov.

[^36]: The New York Times also released an almanac for the first time in
    early 1998.

[^37]: PEMD Education Group, PO Box 39, Cloverdale, CA, 95425.
    707/894-3668 (894-5200 fax), pemd@aol.com.

[^38]: All URLs will be found at the web site devoted to the book.

[^39]: Also, set up my own web site that organizes data-related AND ALL
    OTHER web sites MENTIONED IN THE TEXT for easy access.

[^40]: Quoted in Arnold, Ken and James Gosling. 1998. *The Java
    Programming Language, Second Edition*. Reading, MA: Addison-Wesley,
    p. 150.

[^41]: Robbins, John. 1987. *Diet for a New America*. Walpole, NH:
    Stillpoint Publishing.

[^42]: ![](media/image24.wmf){width="3.0in" height="3.0in"}Meier, Alan.
    1987. \"Saving the \'Other\' Energy in Homes\". *Energy Auditor and
    Retrofitter*. November/December. p. 13.

[^43]: US DOE. 1989. *Residential Energy Consumption Survey (RECS):
    Housing Characteristics 1987*. EIA, Energy Information
    Administration, U.S. Department of Energy. DOE/EIA-0314(87). May.

[^44]: For those interested in the details, there are at least three
    reasons for the SPA\'s flawed estimates that I was able to identify:
    (1) *Self-selection bias*: The SPA survey is voluntary, so only
    those manufacturers who are members of the SPA and who choose to
    report are included; (2) *Non-representative sampling*: The SPA
    survey is dominated by larger developers, like Microsoft. On the
    other hand, a larger percentage of Macintosh software is developed
    by small developers not included in the SPA survey; (3) *Hybrid
    CD-ROMS*: until recently, the revenues from CD-ROM sales that
    contained both Mac and Windows versions were generally counted under
    Windows sales (except where the developer split the revenues for the
    SPA). Other flaws in the SPA\'s methodology may exist, but the SPA
    does not make its detailed methods public. See the WEB SITE for
    details.

[^45]: Their website explaining this suspension stated feebly that the
    SPA Data is \"not designed to be a time series\" (but in their press
    releases they explicitly used their data to draw conclusions about
    year-to-year sales, which could only be done using time series
    data).

[^46]: 2 Campus Boulevard, Newtown Square, PA 19073, 800/442 0925. The
    interviewer was Carrie Harper 916/637-4672.

[^47]: I am told that ants use a scented trail to show the way back to
    the nest. Any scented product (like talcum powder) will confuse them
    and make them go away. Talc may also clog their breathing apparatus.

[^48]: Udell, Jon. 1996. \"Web Surveys: Helpful Techniques for Web-based
    Data Collection and Analysis\". *Byte Magazine*. October. p. 133.

[^49]: As quoted in Crossen (1994) p. 116.

[^50]: Frederick Mosteller, Stephen E. Fienberg, and Robert E. K.
    Rourke. *Beginning Statistics with Data Analysis*, as quoted in
    Tufte, Edward R. 1997. *Visual Explanations: Images and Quantities,
    Evidence and Narrative*. Cheshire, CT: Graphics Press. p. 26.

[^51]: Cohen, Bernard L. 1990. The Nuclear Energy Option: An Alternative
    for the 90s. NY, NY: Plenum Press.

[^52]: This matrix was first developed by James D. Thompson in Thompson,
    James D. and A. Tuden. 1959. \"Strategies, Structures, and Processes
    of Organizational Decisions.\" In *Comparative Studies in
    Administration*. Edited by J. D. Thompson et al. Pittsburgh, PA:
    Pittsburgh University Press. pp. 195-216.

[^53]: Quoted in Arnold, Ken and James Gosling. 1998. *The Java
    Programming Language, Second Edition*. Reading, MA: Addison-Wesley,
    p. 57.

[^54]: The converse of this statement is also true, which is that clear
    and precise explanations of results indicate clear and precise
    thinking on the part of an analyst. Those who can\'t explain their
    own analysis in an understandable way are not clear thinkers, and
    you should steer clear of their work.

[^55]: Dougan, Michael. 1996. \"Ancient quest tests new library\".
    *Sunday SF Examiner*, December 15, 1996, p. C1.

[^56]: Taken from Jacobs, Mark. 1980. *Jumping Up and Down on the Roof,
    Throwing Bags of Water on People*. Garden City, NY: Dolphin Books,
    Doubleday & Company, Inc. p.111. The cartoon orginally appeared in
    *The New Yorker*.

[^57]: Stoll, Clifford. 1996. *Silicon Snake Oil*. New York, NY: Anchor
    Books, pp. 193-4.

[^58]: For a nice introduction to the concept of a spreadsheet, see
    Jones, Gerald E. 1995. *How to Lie with Charts*. San Francisco, CA:
    Sybex, in the section titled \"Spreadsheets: The Crash Course\", on
    pages 154-157.

[^59]: Quoted in Arnold, Ken and James Gosling. 1998. *The Java
    Programming Language, Second Edition*. Reading, MA: Addison-Wesley,
    p. 384.

[^60]: Krause, Florentin, Jonathan Koomey and David Olivier. 1998.
    **Energy Policy in the Greenhouse**. Volume II, Part 3A. *The Cost
    and Potential of Carbon Emission Reductions in Western Europe\'s
    Power Sector*. El Cerrito, CA: International Project for Sustainable
    Energy Paths.

[^61]: Bugs exist even in widely used software like Microsoft Excel or
    Lotus 1-2-3.

[^62]: Cafasso, Rosemary. 1996. \"Quality Control\". *PC Week Online*.
    June 18.

[^63]: CNET online retells the story of this launch as well as other
    famous computer glitches at
    http://www.cnet.com/content/features/Dlife/bugs/index.html.

[^64]: Aragon, Lawrence. 1997. \"Down with Dirt\". *PC Week*. October
    27, 1997. p. 83.

[^65]: US raw steel production from 1997 World Almanac and Book of
    Facts, p. 153.

[^66]: US population from 1997 World Almanac and Book of Facts, p. 382.

[^67]: 1940-1980 GNP from stat Ab 1990, p. 425. 1990 GNP in current \$
    from 1997 World Almanac and Book of Facts, p.133, adjusted to 1982
    dollars using the consumer price index from p.132.

[^68]: The technical term for this summation is called \"present
    valuing\", which uses an interest rate to account for the time value
    of money, but \"cumulative benefits\" is an accurate enough term for
    our purposes.

[^69]: "Trickle-Down Plan." [SF Chronicle]{.underline}, December 5 1996,
    E10.

[^70]: Gasoline typically sells for twice that amount because of the
    additional refining and transportation costs to bring the fuel to
    gas stations.

[^71]: This calculation indicates a startling fact: gasoline is far
    cheaper than bottled water or any other purchased fluid you can
    name, in spite of its extraordinary energy density. For a nice graph
    demonstrating this result, see Nadis, Steve and James J. MacKenzie.
    1993. *Car Trouble*. World Resources Institute. Washington, DC

[^72]: John Holdren always says, in a half joking way, \"You\'d be
    amazed at the number of physical constants which start with 1 or
    5\". As this comment implies, knowing conversion factors to just 1
    significant figure can still be useful.

[^73]: Graduate students at the Energy and Resources Group at UC
    Berkeley are routinely supplied with several of these pages through
    their course of study. One of these is reprinted in an appendix in
    the back of this book.

[^74]: I am indebted to Chris Calwell for bringing JFK\'s prediliction
    for debate to my attention, and to Mark Morland for supplying
    details on the Cuban missile crisis.

[^75]: Quoted in Arnold, Ken and James Gosling. 1998. *The Java
    Programming Language, Second Edition*. Reading, MA: Addison-Wesley,
    p. 372.

[^76]: Currier, Chet. *SF Chronicle*, November 25 1996, p. E3.

[^77]: I am indebted to Mark Nowak for this marvelous anecdote
    410/254-4022.

[^78]: Kunde, Diana. 1998. \"Don\'t Be Bugged\". *SF Chronicle*, Sunday,
    March 8, 1998, p. J1.

[^79]: Crossen 1994, p.101.

[^80]: As quoted in Crossen 1994, p.100.

[^81]: BUY a lottery ticket to observe the process. I probably will
    learn something that I can write about.

[^82]: As of April 1998, there were 19,000 million-dollar lottery
    winners in the U.S. (Beckett, Jamie. 1998. \"Lotto Prize Brokers Hit
    Jackpot\". *SF Chronicle*, April 11, 1998, p. D1). This total is
    after decades of lotteries held across the nation.

[^83]: As quoted in Hyams 1982 p.91.

[^84]: This specific item is one of my big pet peeves, so I describe it
    here in hopes that it may prevent one of the more egregious
    documentation mistakes from happening in reports where it might have
    been included otherwise. ***Calwell suggests I include his stamp
    price example and cost of gasoline example.***

[^85]: \"Stamp Price to Rise a Penny to 33 Cents\". *SF Chronicle*, May
    12, 1998, p. A1+.

[^86]: Knutson, Lawrence L. 1998. \"U.S. to Rain Pennies on Post Office:
    33-cent First-class Stamp Takes Effect Next Year\". *The Oakland
    Tribune*, May 12, 1998, p. NEWS-1+.

[^87]: The same sort of intellectual chaos that pervaded that debate now
    characterizes the emerging demand for advertising and commerce on
    the web. Businesses want hard numbers about the effectiveness of web
    advertising in selling their products, but solid statistics are
    scarce (NYT REF\--DATA JUNGLE). Systematic thinking in this area
    could be of immense value to the companies deciding on advertising
    budgets. Right now, the decisions are being driven more by hype than
    reality.

[^88]: ![](media/image25.wmf){width="3.0in" height="3.0in"}Koomey,
    Jonathan. 1990. Comparative Analysis of Monetary Estimates of
    External Environmental Costs Associated with Combustion of Fossil
    Fuels. Lawrence Berkeley Laboratory. LBL-28313. July. Download a
    copy in PDF format from http://enduse.lbl.gov/eufpubs.html.

[^89]: Herhold, Scott. 1998. \"Computer Files Erased at Stanford\". *San
    Jose Mercury News*, April 9, 1998, p. 1A.

[^90]: Quoted in Arnold, Ken and James Gosling. 1998. *The Java
    Programming Language, Second Edition*. Reading, MA: Addison-Wesley,
    p. XIX.

[^91]: Show a USA today type graph vs. a tufte graph as an example.

[^92]: Calwell has a favorite example from his EV report.

[^93]: Tufte 1997, p.53

[^94]: should I put in an example here?

[^95]: Quoted in Arnold, Ken and James Gosling. 1998. *The Java
    Programming Language, Second Edition*. Reading, MA: Addison-Wesley,
    p. 102.