popular

May 2001(This article was written as a kind of business plan for a
new language.
So it is missing (because it takes for granted) the most important
feature of a good programming language: very powerful abstractions.)A friend of mine once told an eminent operating systems
expert that he wanted to design a really good
programming language.  The expert told him that it would be a
waste of time, that programming languages don't become popular
or unpopular based on their merits, and so no matter how
good his language was, no one would use it.  At least, that
was what had happened to the language he had designed.What does make a language popular?  Do popular
languages deserve their popularity?  Is it worth trying to
define a good programming language?  How would you do it?I think the answers to these questions can be found by looking
at hackers, and learning what they want.  Programming
languages are for hackers, and a programming language
is good as a programming language (rather than, say, an
exercise in denotational semantics or compiler design)
if and only if hackers like it.1 The Mechanics of PopularityIt's true, certainly, that most people don't choose programming
languages simply based on their merits.  Most programmers are told
what language to use by someone else.  And yet I think the effect
of such external factors on the popularity of programming languages
is not as great as it's sometimes thought to be. I think a bigger
problem is that a hacker's idea of a good programming language is
not the same as most language designers'.Between the two, the hacker's opinion is the one that matters.
Programming languages are not theorems. They're tools, designed
for people, and they have to be designed to suit human strengths
and weaknesses as much as shoes have to be designed for human feet.
If a shoe pinches when you put it on, it's a bad shoe, however
elegant it may be as a piece of sculpture.It may be that the majority of programmers can't tell a good language
from a bad one. But that's no different with any other tool. It
doesn't mean that it's a waste of time to try designing a good
language. Expert hackers
can tell a good language when they see
one, and they'll use it. Expert hackers are a tiny minority,
admittedly, but that tiny minority write all the good software,
and their influence is such that the rest of the programmers will
tend to use whatever language they use. Often, indeed, it is not
merely influence but command: often the expert hackers are the very
people who, as their bosses or faculty advisors, tell the other
programmers what language to use.The opinion of expert hackers is not the only force that determines
the relative popularity of programming languages — legacy software
(Cobol) and hype (Ada, Java) also play a role — but I think it is
the most powerful force over the long term. Given an initial critical
mass and enough time, a programming language probably becomes about
as popular as it deserves to be. And popularity further separates
good languages from bad ones, because feedback from real live users
always leads to improvements. Look at how much any popular language
has changed during its life. Perl and Fortran are extreme cases,
but even Lisp has changed a lot. Lisp 1.5 didn't have macros, for
example; these evolved later, after hackers at MIT had spent a
couple years using Lisp to write real programs. [1]So whether or not a language has to be good to be popular, I think
a language has to be popular to be good. And it has to stay popular
to stay good. The state of the art in programming languages doesn't
stand still. And yet the Lisps we have today are still pretty much
what they had at MIT in the mid-1980s, because that's the last time
Lisp had a sufficiently large and demanding user base.Of course, hackers have to know about a language before they can
use it. How are they to hear? From other hackers. But there has to
be some initial group of hackers using the language for others even
to hear about it. I wonder how large this group has to be; how many
users make a critical mass? Off the top of my head, I'd say twenty.
If a language had twenty separate users, meaning twenty users who
decided on their own to use it, I'd consider it to be real.Getting there can't be easy. I would not be surprised if it is
harder to get from zero to twenty than from twenty to a thousand.
The best way to get those initial twenty users is probably to use
a trojan horse: to give people an application they want, which
happens to be written in the new language.2 External FactorsLet's start by acknowledging one external factor that does affect
the popularity of a programming language. To become popular, a
programming language has to be the scripting language of a popular
system. Fortran and Cobol were the scripting languages of early
IBM mainframes. C was the scripting language of Unix, and so, later,
was Perl. Tcl is the scripting language of Tk. Java and Javascript
are intended to be the scripting languages of web browsers.Lisp is not a massively popular language because it is not the
scripting language of a massively popular system. What popularity
it retains dates back to the 1960s and 1970s, when it was the
scripting language of MIT. A lot of the great programmers of the
day were associated with MIT at some point. And in the early 1970s,
before C, MIT's dialect of Lisp, called MacLisp, was one of the
only programming languages a serious hacker would want to use.Today Lisp is the scripting language of two moderately popular
systems, Emacs and Autocad, and for that reason I suspect that most
of the Lisp programming done today is done in Emacs Lisp or AutoLisp.Programming languages don't exist in isolation. To hack is a
transitive verb — hackers are usually hacking something — and in
practice languages are judged relative to whatever they're used to
hack. So if you want to design a popular language, you either have
to supply more than a language, or you have to design your language
to replace the scripting language of some existing system.Common Lisp is unpopular partly because it's an orphan. It did
originally come with a system to hack: the Lisp Machine. But Lisp
Machines (along with parallel computers) were steamrollered by the
increasing power of general purpose processors in the 1980s. Common
Lisp might have remained popular if it had been a good scripting
language for Unix. It is, alas, an atrociously bad one.One way to describe this situation is to say that a language isn't
judged on its own merits. Another view is that a programming language
really isn't a programming language unless it's also the scripting
language of something. This only seems unfair if it comes as a
surprise. I think it's no more unfair than expecting a programming
language to have, say, an implementation. It's just part of what
a programming language is.A programming language does need a good implementation, of course,
and this must be free. Companies will pay for software, but individual
hackers won't, and it's the hackers you need to attract.A language also needs to have a book about it. The book should be
thin, well-written, and full of good examples. K&R is the ideal
here. At the moment I'd almost say that a language has to have a
book published by O'Reilly. That's becoming the test of mattering
to hackers.There should be online documentation as well. In fact, the book
can start as online documentation. But I don't think that physical
books are outmoded yet. Their format is convenient, and the de
facto censorship imposed by publishers is a useful if imperfect
filter. Bookstores are one of the most important places for learning
about new languages.3 BrevityGiven that you can supply the three things any language needs — a
free implementation, a book, and something to hack — how do you
make a language that hackers will like?One thing hackers like is brevity. Hackers are lazy, in the same
way that mathematicians and modernist architects are lazy: they
hate anything extraneous. It would not be far from the truth to
say that a hacker about to write a program decides what language
to use, at least subconsciously, based on the total number of
characters he'll have to type. If this isn't precisely how hackers
think, a language designer would do well to act as if it were.It is a mistake to try to baby the user with long-winded expressions
that are meant to resemble English. Cobol is notorious for this
flaw. A hacker would consider being asked to writeadd x to y giving zinstead ofz = x+yas something between an insult to his intelligence and a sin against
God.It has sometimes been said that Lisp should use first and rest
instead of car and cdr, because it would make programs easier to
read. Maybe for the first couple hours. But a hacker can learn
quickly enough that car means the first element of a list and cdr
means the rest. Using first and rest means 50% more typing. And
they are also different lengths, meaning that the arguments won't
line up when they're called, as car and cdr often are, in successive
lines. I've found that it matters a lot how code lines up on the
page. I can barely read Lisp code when it is set in a variable-width
font, and friends say this is true for other languages too.Brevity is one place where strongly typed languages lose. All other
things being equal, no one wants to begin a program with a bunch
of declarations. Anything that can be implicit, should be.The individual tokens should be short as well. Perl and Common Lisp
occupy opposite poles on this question. Perl programs can be almost
cryptically dense, while the names of built-in Common Lisp operators
are comically long. The designers of Common Lisp probably expected
users to have text editors that would type these long names for
them. But the cost of a long name is not just the cost of typing
it. There is also the cost of reading it, and the cost of the space
it takes up on your screen.4 HackabilityThere is one thing more important than brevity to a hacker: being
able to do what you want. In the history of programming languages
a surprising amount of effort has gone into preventing programmers
from doing things considered to be improper. This is a dangerously
presumptuous plan. How can the language designer know what the
programmer is going to need to do? I think language designers would
do better to consider their target user to be a genius who will
need to do things they never anticipated, rather than a bumbler
who needs to be protected from himself. The bumbler will shoot
himself in the foot anyway. You may save him from referring to
variables in another package, but you can't save him from writing
a badly designed program to solve the wrong problem, and taking
forever to do it.Good programmers often want to do dangerous and unsavory things.
By unsavory I mean things that go behind whatever semantic facade
the language is trying to present: getting hold of the internal
representation of some high-level abstraction, for example. Hackers
like to hack, and hacking means getting inside things and second
guessing the original designer.Let yourself be second guessed. When you make any tool, people use
it in ways you didn't intend, and this is especially true of a
highly articulated tool like a programming language. Many a hacker
will want to tweak your semantic model in a way that you never
imagined. I say, let them; give the programmer access to as much
internal stuff as you can without endangering runtime systems like
the garbage collector.In Common Lisp I have often wanted to iterate through the fields
of a struct — to comb out references to a deleted object, for example,
or find fields that are uninitialized. I know the structs are just
vectors underneath. And yet I can't write a general purpose function
that I can call on any struct. I can only access the fields by
name, because that's what a struct is supposed to mean.A hacker may only want to subvert the intended model of things once
or twice in a big program. But what a difference it makes to be
able to. And it may be more than a question of just solving a
problem. There is a kind of pleasure here too. Hackers share the
surgeon's secret pleasure in poking about in gross innards, the
teenager's secret pleasure in popping zits. [2] For boys, at least,
certain kinds of horrors are fascinating. Maxim magazine publishes
an annual volume of photographs, containing a mix of pin-ups and
grisly accidents. They know their audience.Historically, Lisp has been good at letting hackers have their way.
The political correctness of Common Lisp is an aberration. Early
Lisps let you get your hands on everything. A good deal of that
spirit is, fortunately, preserved in macros. What a wonderful thing,
to be able to make arbitrary transformations on the source code.Classic macros are a real hacker's tool — simple, powerful, and
dangerous. It's so easy to understand what they do: you call a
function on the macro's arguments, and whatever it returns gets
inserted in place of the macro call. Hygienic macros embody the
opposite principle. They try to protect you from understanding what
they're doing. I have never heard hygienic macros explained in one
sentence. And they are a classic example of the dangers of deciding
what programmers are allowed to want. Hygienic macros are intended
to protect me from variable capture, among other things, but variable
capture is exactly what I want in some macros.A really good language should be both clean and dirty: cleanly
designed, with a small core of well understood and highly orthogonal
operators, but dirty in the sense that it lets hackers have their
way with it. C is like this. So were the early Lisps. A real hacker's
language will always have a slightly raffish character.A good programming language should have features that make the kind
of people who use the phrase "software engineering" shake their
heads disapprovingly. At the other end of the continuum are languages
like Ada and Pascal, models of propriety that are good for teaching
and not much else.5 Throwaway ProgramsTo be attractive to hackers, a language must be good for writing
the kinds of programs they want to write. And that means, perhaps
surprisingly, that it has to be good for writing throwaway programs.A throwaway program is a program you write quickly for some limited
task: a program to automate some system administration task, or
generate test data for a simulation, or convert data from one format
to another. The surprising thing about throwaway programs is that,
like the "temporary" buildings built at so many American universities
during World War II, they often don't get thrown away. Many evolve
into real programs, with real features and real users.I have a hunch that the best big programs begin life this way,
rather than being designed big from the start, like the Hoover Dam.
It's terrifying to build something big from scratch. When people
take on a project that's too big, they become overwhelmed. The
project either gets bogged down, or the result is sterile and
wooden: a shopping mall rather than a real downtown, Brasilia rather
than Rome, Ada rather than C.Another way to get a big program is to start with a throwaway
program and keep improving it. This approach is less daunting, and
the design of the program benefits from evolution. I think, if one
looked, that this would turn out to be the way most big programs
were developed. And those that did evolve this way are probably
still written in whatever language they were first written in,
because it's rare for a program to be ported, except for political
reasons. And so, paradoxically, if you want to make a language that
is used for big systems, you have to make it good for writing
throwaway programs, because that's where big systems come from.Perl is a striking example of this idea. It was not only designed
for writing throwaway programs, but was pretty much a throwaway
program itself. Perl began life as a collection of utilities for
generating reports, and only evolved into a programming language
as the throwaway programs people wrote in it grew larger. It was
not until Perl 5 (if then) that the language was suitable for
writing serious programs, and yet it was already massively popular.What makes a language good for throwaway programs? To start with,
it must be readily available. A throwaway program is something that
you expect to write in an hour. So the language probably must
already be installed on the computer you're using. It can't be
something you have to install before you use it. It has to be there.
C was there because it came with the operating system. Perl was
there because it was originally a tool for system administrators,
and yours had already installed it.Being available means more than being installed, though. An
interactive language, with a command-line interface, is more
available than one that you have to compile and run separately. A
popular programming language should be interactive, and start up
fast.Another thing you want in a throwaway program is brevity. Brevity
is always attractive to hackers, and never more so than in a program
they expect to turn out in an hour.6 LibrariesOf course the ultimate in brevity is to have the program already
written for you, and merely to call it. And this brings us to what
I think will be an increasingly important feature of programming
languages: library functions. Perl wins because it has large
libraries for manipulating strings. This class of library functions
are especially important for throwaway programs, which are often
originally written for converting or extracting data.  Many Perl
programs probably begin as just a couple library calls stuck
together.I think a lot of the advances that happen in programming languages
in the next fifty years will have to do with library functions. I
think future programming languages will have libraries that are as
carefully designed as the core language. Programming language design
will not be about whether to make your language strongly or weakly
typed, or object oriented, or functional, or whatever, but about
how to design great libraries. The kind of language designers who
like to think about how to design type systems may shudder at this.
It's almost like writing applications! Too bad. Languages are for
programmers, and libraries are what programmers need.It's hard to design good libraries. It's not simply a matter of
writing a lot of code. Once the libraries get too big, it can
sometimes take longer to find the function you need than to write
the code yourself. Libraries need to be designed using a small set
of orthogonal operators, just like the core language. It ought to
be possible for the programmer to guess what library call will do
what he needs.Libraries are one place Common Lisp falls short. There are only
rudimentary libraries for manipulating strings, and almost none
for talking to the operating system. For historical reasons, Common
Lisp tries to pretend that the OS doesn't exist. And because you
can't talk to the OS, you're unlikely to be able to write a serious
program using only the built-in operators in Common Lisp. You have
to use some implementation-specific hacks as well, and in practice
these tend not to give you everything you want. Hackers would think
a lot more highly of Lisp if Common Lisp had powerful string
libraries and good OS support.7 SyntaxCould a language with Lisp's syntax, or more precisely, lack of
syntax, ever become popular? I don't know the answer to this
question. I do think that syntax is not the main reason Lisp isn't
currently popular. Common Lisp has worse problems than unfamiliar
syntax. I know several programmers who are comfortable with prefix
syntax and yet use Perl by default, because it has powerful string
libraries and can talk to the os.There are two possible problems with prefix notation: that it is
unfamiliar to programmers, and that it is not dense enough. The
conventional wisdom in the Lisp world is that the first problem is
the real one. I'm not so sure. Yes, prefix notation makes ordinary
programmers panic. But I don't think ordinary programmers' opinions
matter. Languages become popular or unpopular based on what expert
hackers think of them, and I think expert hackers might be able to
deal with prefix notation. Perl syntax can be pretty incomprehensible,
but that has not stood in the way of Perl's popularity. If anything
it may have helped foster a Perl cult.A more serious problem is the diffuseness of prefix notation. For
expert hackers, that really is a problem. No one wants to write
(aref a x y) when they could write a[x,y].In this particular case there is a way to finesse our way out of
the problem. If we treat data structures as if they were functions
on indexes, we could write (a x y) instead, which is even shorter
than the Perl form. Similar tricks may shorten other types of
expressions.We can get rid of (or make optional) a lot of parentheses by making
indentation significant. That's how programmers read code anyway:
when indentation says one thing and delimiters say another, we go
by the indentation. Treating indentation as significant would
eliminate this common source of bugs as well as making programs
shorter.Sometimes infix syntax is easier to read. This is especially true
for math expressions. I've used Lisp my whole programming life and
I still don't find prefix math expressions natural. And yet it is
convenient, especially when you're generating code, to have operators
that take any number of arguments. So if we do have infix syntax,
it should probably be implemented as some kind of read-macro.I don't think we should be religiously opposed to introducing syntax
into Lisp, as long as it translates in a well-understood way into
underlying s-expressions. There is already a good deal of syntax
in Lisp. It's not necessarily bad to introduce more, as long as no
one is forced to use it. In Common Lisp, some delimiters are reserved
for the language, suggesting that at least some of the designers
intended to have more syntax in the future.One of the most egregiously unlispy pieces of syntax in Common Lisp
occurs in format strings; format is a language in its own right,
and that language is not Lisp. If there were a plan for introducing
more syntax into Lisp, format specifiers might be able to be included
in it. It would be a good thing if macros could generate format
specifiers the way they generate any other kind of code.An eminent Lisp hacker told me that his copy of CLTL falls open to
the section format. Mine too. This probably indicates room for
improvement. It may also mean that programs do a lot of I/O.8 EfficiencyA good language, as everyone knows, should generate fast code. But
in practice I don't think fast code comes primarily from things
you do in the design of the language. As Knuth pointed out long
ago, speed only matters in certain critical bottlenecks.  And as
many programmers have observed since, one is very often mistaken
about where these bottlenecks are.So, in practice, the way to get fast code is to have a very good
profiler, rather than by, say, making the language strongly typed.
You don't need to know the type of every argument in every call in
the program. You do need to be able to declare the types of arguments
in the bottlenecks. And even more, you need to be able to find out
where the bottlenecks are.One complaint people have had with Lisp is that it's hard to tell
what's expensive. This might be true. It might also be inevitable,
if you want to have a very abstract language. And in any case I
think good profiling would go a long way toward fixing the problem:
you'd soon learn what was expensive.Part of the problem here is social. Language designers like to
write fast compilers. That's how they measure their skill. They
think of the profiler as an add-on, at best. But in practice a good
profiler may do more to improve the speed of actual programs written
in the language than a compiler that generates fast code. Here,
again, language designers are somewhat out of touch with their
users. They do a really good job of solving slightly the wrong
problem.It might be a good idea to have an active profiler — to push
performance data to the programmer instead of waiting for him to
come asking for it. For example, the editor could display bottlenecks
in red when the programmer edits the source code. Another approach
would be to somehow represent what's happening in running programs.
This would be an especially big win in server-based applications,
where you have lots of running programs to look at. An active
profiler could show graphically what's happening in memory as a
program's running, or even make sounds that tell what's happening.Sound is a good cue to problems. In one place I worked, we had a
big board of dials showing what was happening to our web servers.
The hands were moved by little servomotors that made a slight noise
when they turned. I couldn't see the board from my desk, but I
found that I could tell immediately, by the sound, when there was
a problem with a server.It might even be possible to write a profiler that would automatically
detect inefficient algorithms. I would not be surprised if certain
patterns of memory access turned out to be sure signs of bad
algorithms. If there were a little guy running around inside the
computer executing our programs, he would probably have as long
and plaintive a tale to tell about his job as a federal government
employee. I often have a feeling that I'm sending the processor on
a lot of wild goose chases, but I've never had a good way to look
at what it's doing.A number of Lisps now compile into byte code, which is then executed
by an interpreter. This is usually done to make the implementation
easier to port, but it could be a useful language feature. It might
be a good idea to make the byte code an official part of the
language, and to allow programmers to use inline byte code in
bottlenecks. Then such optimizations would be portable too.The nature of speed, as perceived by the end-user, may be changing.
With the rise of server-based applications, more and more programs
may turn out to be i/o-bound. It will be worth making i/o fast.
The language can help with straightforward measures like simple,
fast, formatted output functions, and also with deep structural
changes like caching and persistent objects.Users are interested in response time. But another kind of efficiency
will be increasingly important: the number of simultaneous users
you can support per processor. Many of the interesting applications
written in the near future will be server-based, and the number of
users per server is the critical question for anyone hosting such
applications. In the capital cost of a business offering a server-based
application, this is the divisor.For years, efficiency hasn't mattered much in most end-user
applications. Developers have been able to assume that each user
would have an increasingly powerful processor sitting on their
desk. And by Parkinson's Law, software has expanded to use the
resources available. That will change with server-based applications.
In that world, the hardware and software will be supplied together.
For companies that offer server-based applications, it will make
a very big difference to the bottom line how many users they can
support per server.In some applications, the processor will be the limiting factor,
and execution speed will be the most important thing to optimize.
But often memory will be the limit; the number of simultaneous
users will be determined by the amount of memory you need for each
user's data. The language can help here too. Good support for
threads will enable all the users to share a single heap. It may
also help to have persistent objects and/or language level support
for lazy loading.9 TimeThe last ingredient a popular language needs is time. No one wants
to write programs in a language that might go away, as so many
programming languages do. So most hackers will tend to wait until
a language has been around for a couple years before even considering
using it.Inventors of wonderful new things are often surprised to discover
this, but you need time to get any message through to people. A
friend of mine rarely does anything the first time someone asks
him. He knows that people sometimes ask for things that they turn
out not to want. To avoid wasting his time, he waits till the third
or fourth time he's asked to do something; by then, whoever's asking
him may be fairly annoyed, but at least they probably really do
want whatever they're asking for.Most people have learned to do a similar sort of filtering on new
things they hear about. They don't even start paying attention
until they've heard about something ten times. They're perfectly
justified: the majority of hot new whatevers do turn out to be a
waste of time, and eventually go away. By delaying learning VRML,
I avoided having to learn it at all.So anyone who invents something new has to expect to keep repeating
their message for years before people will start to get it. We
wrote what was, as far as I know, the first web-server based
application, and it took us years to get it through to people that
it didn't have to be downloaded. It wasn't that they were stupid.
They just had us tuned out.The good news is, simple repetition solves the problem. All you
have to do is keep telling your story, and eventually people will
start to hear. It's not when people notice you're there that they
pay attention; it's when they notice you're still there.It's just as well that it usually takes a while to gain momentum.
Most technologies evolve a good deal even after they're first
launched — programming languages especially. Nothing could be better,
for a new techology, than a few years of being used only by a small
number of early adopters. Early adopters are sophisticated and
demanding, and quickly flush out whatever flaws remain in your
technology. When you only have a few users you can be in close
contact with all of them. And early adopters are forgiving when
you improve your system, even if this causes some breakage.There are two ways new technology gets introduced: the organic
growth method, and the big bang method. The organic growth method
is exemplified by the classic seat-of-the-pants underfunded garage
startup. A couple guys, working in obscurity, develop some new
technology. They launch it with no marketing and initially have
only a few (fanatically devoted) users. They continue to improve
the technology, and meanwhile their user base grows by word of
mouth. Before they know it, they're big.The other approach, the big bang method, is exemplified by the
VC-backed, heavily marketed startup. They rush to develop a product,
launch it with great publicity, and immediately (they hope) have
a large user base.Generally, the garage guys envy the big bang guys. The big bang
guys are smooth and confident and respected by the VCs. They can
afford the best of everything, and the PR campaign surrounding the
launch has the side effect of making them celebrities. The organic
growth guys, sitting in their garage, feel poor and unloved. And
yet I think they are often mistaken to feel sorry for themselves.
Organic growth seems to yield better technology and richer founders
than the big bang method. If you look at the dominant technologies
today, you'll find that most of them grew organically.This pattern doesn't only apply to companies. You see it in sponsored
research too. Multics and Common Lisp were big-bang projects, and
Unix and MacLisp were organic growth projects.10 Redesign"The best writing is rewriting," wrote E. B. White.  Every good
writer knows this, and it's true for software too. The most important
part of design is redesign. Programming languages, especially,
don't get redesigned enough.To write good software you must simultaneously keep two opposing
ideas in your head. You need the young hacker's naive faith in
his abilities, and at the same time the veteran's skepticism. You
have to be able to think
how hard can it be? with one half of
your brain while thinking
it will never work with the other.The trick is to realize that there's no real contradiction here.
You want to be optimistic and skeptical about two different things.
You have to be optimistic about the possibility of solving the
problem, but skeptical about the value of whatever solution you've
got so far.People who do good work often think that whatever they're working
on is no good. Others see what they've done and are full of wonder,
but the creator is full of worry. This pattern is no coincidence:
it is the worry that made the work good.If you can keep hope and worry balanced, they will drive a project
forward the same way your two legs drive a bicycle forward. In the
first phase of the two-cycle innovation engine, you work furiously
on some problem, inspired by your confidence that you'll be able
to solve it. In the second phase, you look at what you've done in
the cold light of morning, and see all its flaws very clearly. But
as long as your critical spirit doesn't outweigh your hope, you'll
be able to look at your admittedly incomplete system, and think,
how hard can it be to get the rest of the way?, thereby continuing
the cycle.It's tricky to keep the two forces balanced. In young hackers,
optimism predominates. They produce something, are convinced it's
great, and never improve it. In old hackers, skepticism predominates,
and they won't even dare to take on ambitious projects.Anything you can do to keep the redesign cycle going is good. Prose
can be rewritten over and over until you're happy with it. But
software, as a rule, doesn't get redesigned enough. Prose has
readers, but software has users. If a writer rewrites an essay,
people who read the old version are unlikely to complain that their
thoughts have been broken by some newly introduced incompatibility.Users are a double-edged sword. They can help you improve your
language, but they can also deter you from improving it. So choose
your users carefully, and be slow to grow their number. Having
users is like optimization: the wise course is to delay it. Also,
as a general rule, you can at any given time get away with changing
more than you think. Introducing change is like pulling off a
bandage: the pain is a memory almost as soon as you feel it.Everyone knows that it's not a good idea to have a language designed
by a committee. Committees yield bad design. But I think the worst
danger of committees is that they interfere with redesign. It is
so much work to introduce changes that no one wants to bother.
Whatever a committee decides tends to stay that way, even if most
of the members don't like it.Even a committee of two gets in the way of redesign. This happens
particularly in the interfaces between pieces of software written
by two different people. To change the interface both have to agree
to change it at once. And so interfaces tend not to change at all,
which is a problem because they tend to be one of the most ad hoc
parts of any system.One solution here might be to design systems so that interfaces
are horizontal instead of vertical — so that modules are always
vertically stacked strata of abstraction. Then the interface will
tend to be owned by one of them. The lower of two levels will either
be a language in which the upper is written, in which case the
lower level will own the interface, or it will be a slave, in which
case the interface can be dictated by the upper level.11 LispWhat all this implies is that there is hope for a new Lisp.  There
is hope for any language that gives hackers what they want, including
Lisp. I think we may have made a mistake in thinking that hackers
are turned off by Lisp's strangeness. This comforting illusion may
have prevented us from seeing the real problem with Lisp, or at
least Common Lisp, which is that it sucks for doing what hackers
want to do. A hacker's language needs powerful libraries and
something to hack. Common Lisp has neither. A hacker's language is
terse and hackable. Common Lisp is not.The good news is, it's not Lisp that sucks, but Common Lisp. If we
can develop a new Lisp that is a real hacker's language, I think
hackers will use it. They will use whatever language does the job.
All we have to do is make sure this new Lisp does some important
job better than other languages.History offers some encouragement. Over time, successive new
programming languages have taken more and more features from Lisp.
There is no longer much left to copy before the language you've
made is Lisp. The latest hot language, Python, is a watered-down
Lisp with infix syntax and no macros. A new Lisp would be a natural
step in this progression.I sometimes think that it would be a good marketing trick to call
it an improved version of Python. That sounds hipper than Lisp. To
many people, Lisp is a slow AI language with a lot of parentheses.
Fritz Kunze's official biography carefully avoids mentioning the
L-word.  But my guess is that we shouldn't be afraid to call the
new Lisp Lisp. Lisp still has a lot of latent respect among the
very best hackers — the ones who took 6.001 and understood it, for
example. And those are the users you need to win.In "How to Become a Hacker," Eric Raymond describes Lisp as something
like Latin or Greek — a language you should learn as an intellectual
exercise, even though you won't actually use it:

  Lisp is worth learning for the profound enlightenment experience
  you will have when you finally get it; that experience will make
  you a better programmer for the rest of your days, even if you
  never actually use Lisp itself a lot.

If I didn't know Lisp, reading this would set me asking questions.
A language that would make me a better programmer, if it means
anything at all, means a language that would be better for programming.
And that is in fact the implication of what Eric is saying.As long as that idea is still floating around, I think hackers will
be receptive enough to a new Lisp, even if it is called Lisp. But
this Lisp must be a hacker's language, like the classic Lisps of
the 1970s. It must be terse, simple, and hackable. And it must have
powerful libraries for doing what hackers want to do now.In the matter of libraries I think there is room to beat languages
like Perl and Python at their own game. A lot of the new applications
that will need to be written in the coming years will be
server-based
applications. There's no reason a new Lisp shouldn't have string
libraries as good as Perl, and if this new Lisp also had powerful
libraries for server-based applications, it could be very popular.
Real hackers won't turn up their noses at a new tool that will let
them solve hard problems with a few library calls. Remember, hackers
are lazy.It could be an even bigger win to have core language support for
server-based applications. For example, explicit support for programs
with multiple users, or data ownership at the level of type tags.Server-based applications also give us the answer to the question
of what this new Lisp will be used to hack. It would not hurt to
make Lisp better as a scripting language for Unix. (It would be
hard to make it worse.) But I think there are areas where existing
languages would be easier to beat. I think it might be better to
follow the model of Tcl, and supply the Lisp together with a complete
system for supporting server-based applications. Lisp is a natural
fit for server-based applications. Lexical closures provide a way
to get the effect of subroutines when the ui is just a series of
web pages. S-expressions map nicely onto html, and macros are good
at generating it. There need to be better tools for writing
server-based applications, and there needs to be a new Lisp, and
the two would work very well together.12 The Dream LanguageBy way of summary, let's try describing the hacker's dream language.
The dream language is
beautiful, clean, and terse. It has an
interactive toplevel that starts up fast. You can write programs
to solve common problems with very little code.  Nearly all the
code in any program you write is code that's specific to your
application. Everything else has been done for you.The syntax of the language is brief to a fault. You never have to
type an unnecessary character, or even to use the shift key much.Using big abstractions you can write the first version of a program
very quickly. Later, when you want to optimize, there's a really
good profiler that tells you where to focus your attention. You
can make inner loops blindingly fast, even writing inline byte code
if you need to.There are lots of good examples to learn from, and the language is
intuitive enough that you can learn how to use it from examples in
a couple minutes. You don't need to look in the manual much. The
manual is thin, and has few warnings and qualifications.The language has a small core, and powerful, highly orthogonal
libraries that are as carefully designed as the core language. The
libraries all work well together; everything in the language fits
together like the parts in a fine camera. Nothing is deprecated,
or retained for compatibility. The source code of all the libraries
is readily available. It's easy to talk to the operating system
and to applications written in other languages.The language is built in layers. The higher-level abstractions are
built in a very transparent way out of lower-level abstractions,
which you can get hold of if you want.Nothing is hidden from you that doesn't absolutely have to be. The
language offers abstractions only as a way of saving you work,
rather than as a way of telling you what to do. In fact, the language
encourages you to be an equal participant in its design. You can
change everything about it, including even its syntax, and anything
you write has, as much as possible, the same status as what comes
predefined.Notes[1]  Macros very close to the modern idea were proposed by Timothy
Hart in 1964, two years after Lisp 1.5 was released. What was
missing, initially, were ways to avoid variable capture and multiple
evaluation; Hart's examples are subject to both.[2]  In When the Air Hits Your Brain, neurosurgeon Frank Vertosick
recounts a conversation in which his chief resident, Gary, talks
about the difference between surgeons and internists ("fleas"):

  Gary and I ordered a large pizza and found an open booth. The
  chief lit a cigarette. "Look at those goddamn fleas, jabbering
  about some disease they'll see once in their lifetimes. That's
  the trouble with fleas, they only like the bizarre stuff. They
  hate their bread and butter cases. That's the difference between
  us and the fucking fleas. See, we love big juicy lumbar disc
  herniations, but they hate hypertension...."

It's hard to think of a lumbar disc herniation as juicy (except
literally). And yet I think I know what they mean. I've often had
a juicy bug to track down. Someone who's not a programmer would
find it hard to imagine that there could be pleasure in a bug.
Surely it's better if everything just works. In one way, it is.
And yet there is undeniably a grim satisfaction in hunting down
certain sorts of bugs.


pow

January 2017People who are powerful but uncharismatic will tend to be disliked.
Their power makes them a target for criticism that they don't have
the charisma to disarm. That was Hillary Clinton's problem. It also
tends to be a problem for any CEO who is more of a builder than a
schmoozer. And yet the builder-type CEO is (like Hillary) probably
the best person for the job.I don't think there is any solution to this problem. It's human
nature. The best we can do is to recognize that it's happening, and
to understand that being a magnet for criticism is sometimes a sign
not that someone is the wrong person for a job, but that they're
the right one.


submarine

April 2005"Suits make a corporate comeback," says the New
York Times.  Why does this sound familiar?  Maybe because
the suit was also back in February,

September
2004, June
2004, March
2004, September
2003,

November
2002,
April 2002,
and February
2002.

Why do the media keep running stories saying suits are back?  Because
PR firms tell
them to.  One of the most surprising things I discovered
during my brief business career was the existence of the PR industry,
lurking like a huge, quiet submarine beneath the news.  Of the
stories you read in traditional media that aren't about politics,
crimes, or disasters, more than half probably come from PR firms.I know because I spent years hunting such "press hits."  Our startup spent
its entire marketing budget on PR: at a time when we were assembling
our own computers to save money, we were paying a PR firm $16,000
a month.  And they were worth it.  PR is the news equivalent of
search engine optimization; instead of buying ads, which readers
ignore, you get yourself inserted directly into the stories.  [1]Our PR firm
was one of the best in the business.  In 18 months, they got press
hits in over 60 different publications.
And we weren't the only ones they did great things for.
In 1997 I got a call from another
startup founder considering hiring them to promote his company.  I
told him they were PR gods, worth every penny of their outrageous
fees.  But I remember thinking his company's name was odd.
Why call an auction site "eBay"?
SymbiosisPR is not dishonest.  Not quite.  In fact, the reason the best PR
firms are so effective is precisely that they aren't dishonest.
They give reporters genuinely valuable information.  A good PR firm
won't bug reporters just because the client tells them to; they've
worked hard to build their credibility with reporters, and they
don't want to destroy it by feeding them mere propaganda.If anyone is dishonest, it's the reporters.  The main reason PR
firms exist is that reporters are lazy.  Or, to put it more nicely,
overworked.  Really they ought to be out there digging up stories
for themselves.  But it's so tempting to sit in their offices and
let PR firms bring the stories to them.  After all, they know good
PR firms won't lie to them.A good flatterer doesn't lie, but tells his victim selective truths
(what a nice color your eyes are). Good PR firms use the same
strategy: they give reporters stories that are true, but whose truth
favors their clients.For example, our PR firm often pitched stories about how the Web
let small merchants compete with big ones.  This was perfectly true.
But the reason reporters ended up writing stories about this
particular truth, rather than some other one, was that small merchants
were our target market, and we were paying the piper.Different publications vary greatly in their reliance on PR firms.
At the bottom of the heap are the trade press, who make most of
their money from advertising and would give the magazines away for
free if advertisers would let them.  [2] The average
trade publication is a  bunch of ads, glued together by just enough
articles to make it look like a magazine.  They're so desperate for
"content" that some will print your press releases almost verbatim,
if you take the trouble to write them to read like articles.At the other extreme are publications like the New York Times
and the Wall Street Journal.  Their reporters do go out and
find their own stories, at least some of the time.  They'll listen
to PR firms, but briefly and skeptically.  We managed to get press
hits in almost every publication we wanted, but we never managed
to crack the print edition of the Times.  [3]The weak point of the top reporters is not laziness, but vanity.
You don't pitch stories to them.  You have to approach them as if
you were a specimen under their all-seeing microscope, and make it
seem as if the story you want them to run is something they thought
of themselves.Our greatest PR coup was a two-part one.  We estimated, based on
some fairly informal math, that there were about 5000 stores on the
Web.  We got one paper to print this number, which seemed neutral
enough.  But once this "fact" was out there in print, we could quote
it to other publications, and claim that with 1000 users we had 20%
of the online store market.This was roughly true.  We really did have the biggest share of the
online store market, and 5000 was our best guess at its size.  But
the way the story appeared in the press sounded a lot more definite.Reporters like definitive statements.  For example, many of the
stories about Jeremy Jaynes's conviction say that he was one of the
10 worst spammers.  This "fact" originated in Spamhaus's ROKSO list,
which I think even Spamhaus would admit is a rough guess at the top
spammers.  The first stories about Jaynes cited this source, but
now it's simply repeated as if it were part of the indictment.
[4]All you can say with certainty about Jaynes is that he was a fairly
big spammer.  But reporters don't want to print vague stuff like
"fairly big."  They want statements with punch, like "top ten." And
PR firms give them what they want.
Wearing suits, we're told, will make us
3.6
percent more productive.BuzzWhere the work of PR firms really does get deliberately misleading is in
the generation of "buzz."  They usually feed the same story to
several different publications at once.  And when readers see similar
stories in multiple places, they think there is some important trend
afoot.  Which is exactly what they're supposed to think.When Windows 95 was launched, people waited outside stores
at midnight to buy the first copies.  None of them would have been
there without PR firms, who generated such a buzz in
the news media that it became self-reinforcing, like a nuclear chain
reaction.I doubt PR firms realize it yet, but the Web makes it possible to
track them at work.  If you search for the obvious phrases, you
turn up several efforts over the years to place stories about the
return of the suit.  For example, the Reuters article

that got picked up by USA
Today in September 2004.  "The suit is back," it begins.Trend articles like this are almost always the work of
PR firms.  Once you know how to read them, it's straightforward to
figure out who the client is.  With trend stories, PR firms usually
line up one or more "experts" to talk about the industry generally.
In this case we get three: the NPD Group, the creative director of
GQ, and a research director at Smith Barney.  [5] When
you get to the end of the experts, look for the client. And bingo,
there it is: The Men's Wearhouse.Not surprising, considering The Men's Wearhouse was at that moment
running ads saying "The Suit is Back."  Talk about a successful
press hit-- a wire service article whose first sentence is your own
ad copy.The secret to finding other press hits from a given pitch
is to realize that they all started from the same document back at
the PR firm.  Search for a few key phrases and the names of the
clients and the experts, and you'll turn up other variants of this
story.Casual
fridays are out and dress codes are in writes Diane E. Lewis
in The Boston Globe.  In a remarkable coincidence, Ms. Lewis's
industry contacts also include the creative director of GQ.Ripped jeans and T-shirts are out, writes Mary Kathleen Flynn in
US News & World Report.  And she too knows the
creative director of GQ.Men's suits
are back writes Nicole Ford in Sexbuzz.Com ("the ultimate men's
entertainment magazine").Dressing
down loses appeal as men suit up at the office writes Tenisha
Mercer of The Detroit News.
Now that so many news articles are online, I suspect you could find
a similar pattern for most trend stories placed by PR firms.  I
propose we call this new sport "PR diving," and I'm sure there are
far more striking examples out there than this clump of five stories.OnlineAfter spending years chasing them, it's now second nature
to me to recognize press hits for what they are.  But before we
hired a PR firm I had no idea where articles in the mainstream media
came from.  I could tell a lot of them were crap, but I didn't
realize why.Remember the exercises in critical reading you did in school, where
you had to look at a piece of writing and step back and ask whether
the author was telling the whole truth?  If you really want to be
a critical reader, it turns out you have to step back one step
further, and ask not just whether the author is telling the truth,
but why he's writing about this subject at all.Online, the answer tends to be a lot simpler.  Most people who
publish online write what they write for the simple reason that
they want to.  You
can't see the fingerprints of PR firms all over the articles, as
you can in so many print publications-- which is one of the reasons,
though they may not consciously realize it, that readers trust
bloggers more than Business Week.I was talking recently to a friend who works for a
big newspaper.  He thought the print media were in serious trouble,
and that they were still mostly in denial about it.  "They think
the decline is cyclic," he said.  "Actually it's structural."In other words, the readers are leaving, and they're not coming
back.
Why? I think the main reason is that the writing online is more honest.
Imagine how incongruous the New York Times article about
suits would sound if you read it in a blog:
   The urge to look corporate-- sleek, commanding,
  prudent, yet with just a touch of hubris on your well-cut sleeve--
  is an unexpected development in a time of business disgrace.

The problem
with this article is not just that it originated in a PR firm.
The whole tone is bogus.  This is the tone of someone writing down
to their audience.Whatever its flaws, the writing you find online
is authentic.  It's not mystery meat cooked up
out of scraps of pitch letters and press releases, and pressed into
molds of zippy
journalese.  It's people writing what they think.I didn't realize, till there was an alternative, just how artificial
most of the writing in the mainstream media was.  I'm not saying
I used to believe what I read in Time and Newsweek.  Since high
school, at least, I've thought of magazines like that more as
guides to what ordinary people were being
told to think than as
sources of information.  But I didn't realize till the last
few years that writing for publication didn't have to mean writing
that way.  I didn't realize you could write as candidly and
informally as you would if you were writing to a friend.Readers aren't the only ones who've noticed the
change.  The PR industry has too.
A hilarious article
on the site of the PR Society of America gets to the heart of the
matter:
   Bloggers are sensitive about becoming mouthpieces
  for other organizations and companies, which is the reason they
  began blogging in the first place.
PR people fear bloggers for the same reason readers
like them.  And that means there may be a struggle ahead.  As
this new kind of writing draws readers away from traditional media, we
should be prepared for whatever PR mutates into to compensate.
When I think
how hard PR firms work to score press hits in the traditional
media, I can't imagine they'll work any less hard to feed stories
to bloggers, if they can figure out how.
Notes[1] PR has at least
one beneficial feature: it favors small companies.  If PR didn't
work, the only alternative would be to advertise, and only big
companies can afford that.[2] Advertisers pay
less for ads in free publications, because they assume readers
ignore something they get for free.  This is why so many trade
publications nominally have a cover price and yet give away free
subscriptions with such abandon.[3] Different sections
of the Times vary so much in their standards that they're
practically different papers.  Whoever fed the style section reporter
this story about suits coming back would have been sent packing by
the regular news reporters.[4] The most striking
example I know of this type is the "fact" that the Internet worm
of 1988 infected 6000 computers. I was there when it was cooked up,
and this was the recipe: someone guessed that there were about
60,000 computers attached to the Internet, and that the worm might
have infected ten percent of them.Actually no one knows how many computers the worm infected, because
the remedy was to reboot them, and this destroyed all traces.  But
people like numbers.  And so this one is now replicated
all over the Internet, like a little worm of its own.[5] Not all were
necessarily supplied by the PR firm. Reporters sometimes call a few
additional sources on their own, like someone adding a few fresh
vegetables to a can of soup.
Thanks to Ingrid Basset, Trevor Blackwell, Sarah Harlin, Jessica
Livingston, Jackie McDonough, Robert Morris, and Aaron Swartz (who
also found the PRSA article) for reading drafts of this.Correction: Earlier versions used a recent
Business Week article mentioning del.icio.us as an example
of a press hit, but Joshua Schachter tells me
it was spontaneous.


sun

September 2017The most valuable insights are both general and surprising.
F = ma for example. But general and surprising is a hard
combination to achieve. That territory tends to be picked
clean, precisely because those insights are so valuable.Ordinarily, the best that people can do is one without the
other: either surprising without being general (e.g.
gossip), or general without being surprising (e.g.
platitudes).Where things get interesting is the moderately valuable
insights.  You get those from small additions of whichever
quality was missing.  The more common case is a small
addition of generality: a piece of gossip that's more than
just gossip, because it teaches something interesting about
the world. But another less common approach is to focus on
the most general ideas and see if you can find something new
to say about them. Because these start out so general, you
only need a small delta of novelty to produce a useful
insight.A small delta of novelty is all you'll be able to get most
of the time. Which means if you take this route, your ideas
will seem a lot like ones that already exist. Sometimes
you'll find you've merely rediscovered an idea that did
already exist.  But don't be discouraged.  Remember the huge
multiplier that kicks in when you do manage to think of
something even a little new.Corollary: the more general the ideas you're talking about,
the less you should worry about repeating yourself.  If you
write enough, it's inevitable you will.  Your brain is much
the same from year to year and so are the stimuli that hit
it. I feel slightly bad when I find I've said something
close to what I've said before, as if I were plagiarizing
myself. But rationally one shouldn't.  You won't say
something exactly the same way the second time, and that
variation increases the chance you'll get that tiny but
critical delta of novelty.And of course, ideas beget ideas.  (That sounds
familiar.)
An idea with a small amount of novelty could lead to one
with more. But only if you keep going. So it's doubly
important not to let yourself be discouraged by people who
say there's not much new about something you've discovered.
"Not much new" is a real achievement when you're talking
about the most general ideas. It's not true that there's nothing new under the sun.  There
are some domains where there's almost nothing new.  But
there's a big difference between nothing and almost nothing,
when it's multiplied by the area under the sun.
Thanks to Sam Altman, Patrick Collison, and Jessica
Livingston for reading drafts of this.


weird

August 2021When people say that in their experience all programming languages
are basically equivalent, they're making a statement not about
languages but about the kind of programming they've done.99.5% of programming consists of gluing together calls to library
functions. All popular languages are equally good at this. So one
can easily spend one's whole career operating in the intersection
of popular programming languages.But the other .5% of programming is disproportionately interesting.
If you want to learn what it consists of, the weirdness of weird
languages is a good clue to follow.Weird languages aren't weird by accident. Not the good ones, at
least. The weirdness of the good ones usually implies the existence
of some form of programming that's not just the usual gluing together
of library calls.A concrete example: Lisp macros. Lisp macros seem weird even to
many Lisp programmers. They're not only not in the intersection of
popular languages, but by their nature would be hard to implement
properly in a language without turning it into a dialect of
Lisp. And macros are definitely evidence of techniques that go
beyond glue programming. For example, solving problems by first
writing a language for problems of that type, and then writing
your specific application in it. Nor is this all you can do with
macros; it's just one region in a space of program-manipulating
techniques that even now is far from fully explored.So if you want to expand your concept of what programming can be,
one way to do it is by learning weird languages. Pick a language
that most programmers consider weird but whose median user is smart,
and then focus on the differences between this language and the
intersection of popular languages. What can you say in this language
that would be impossibly inconvenient to say in others? In the
process of learning how to say things you couldn't previously say,
you'll probably be learning how to think things you couldn't
previously think.
Thanks to Trevor Blackwell, Patrick Collison, Daniel Gackle, Amjad
Masad, and Robert Morris for reading drafts of this.


diff

December 2001 (rev. May 2002)

(This article came about in response to some questions on
the LL1 mailing list.  It is now
incorporated in Revenge of the Nerds.)When McCarthy designed Lisp in the late 1950s, it was
a radical departure from existing languages,
the most important of which was Fortran.Lisp embodied nine new ideas:
1. Conditionals.  A conditional is an if-then-else
construct.  We take these for granted now.  They were
invented
by McCarthy in the course of developing Lisp.
(Fortran at that time only had a conditional
goto, closely based on the branch instruction in the
underlying hardware.)  McCarthy, who was on the Algol committee, got
conditionals into Algol, whence they spread to most other
languages.2. A function type. In Lisp, functions are first class
objects-- they're a data type just like integers, strings,
etc, and have a literal representation, can be stored in variables,
can be passed as arguments, and so on.3. Recursion.  Recursion existed as a mathematical concept
before Lisp of course, but Lisp was the first programming language to support
it.  (It's arguably implicit in making functions first class
objects.)4. A new concept of variables.  In Lisp, all variables
are effectively pointers. Values are what
have types, not variables, and assigning or binding
variables means copying pointers, not what they point to.5. Garbage-collection.6. Programs composed of expressions. Lisp programs are
trees of expressions, each of which returns a value.
(In some Lisps expressions
can return multiple values.)  This is in contrast to Fortran
and most succeeding languages, which distinguish between
expressions and statements.It was natural to have this
distinction in Fortran because (not surprisingly in a language
where the input format was punched cards) the language was
line-oriented.  You could not nest statements.  And
so while you needed expressions for math to work, there was
no point in making anything else return a value, because
there could not be anything waiting for it.This limitation
went away with the arrival of block-structured languages,
but by then it was too late. The distinction between
expressions and statements was entrenched.  It spread from
Fortran into Algol and thence to both their descendants.When a language is made entirely of expressions, you can
compose expressions however you want.  You can say either
(using Arc syntax)(if foo (= x 1) (= x 2))or(= x (if foo 1 2))7. A symbol type.  Symbols differ from strings in that
you can test equality by comparing a pointer.8. A notation for code using trees of symbols.9. The whole language always available.
There is
no real distinction between read-time, compile-time, and runtime.
You can compile or run code while reading, read or run code
while compiling, and read or compile code at runtime.Running code at read-time lets users reprogram Lisp's syntax;
running code at compile-time is the basis of macros; compiling
at runtime is the basis of Lisp's use as an extension
language in programs like Emacs; and reading at runtime
enables programs to communicate using s-expressions, an
idea recently reinvented as XML.
When Lisp was first invented, all these ideas were far
removed from ordinary programming practice, which was
dictated largely by the hardware available in the late 1950s.Over time, the default language, embodied
in a succession of popular languages, has
gradually evolved toward Lisp.  1-5 are now widespread.
6 is starting to appear in the mainstream.
Python has a form of 7, though there doesn't seem to be
any syntax for it.
8, which (with 9) is what makes Lisp macros
possible, is so far still unique to Lisp,
perhaps because (a) it requires those parens, or something
just as bad, and (b) if you add that final increment of power,
you can no
longer claim to have invented a new language, but only
to have designed a new dialect of Lisp ; -)Though useful to present-day programmers, it's
strange to describe Lisp in terms of its
variation from the random expedients other languages
adopted.  That was not, probably, how McCarthy
thought of it.  Lisp wasn't designed to fix the mistakes
in Fortran; it came about more as the byproduct of an
attempt to axiomatize computation.


hubs

Want to start a startup?  Get funded by
Y Combinator.

October 2011If you look at a list of US cities sorted by population, the number
of successful startups per capita varies by orders of magnitude.
Somehow it's as if most places were sprayed with startupicide.I wondered about this for years.  I could see the average town was
like a roach motel for startup ambitions: smart, ambitious people
went in, but no startups came out.  But I was never able to figure
out exactly what happened inside the motel—exactly what was
killing all the potential startups.
[1]A couple weeks ago I finally figured it out. I was framing the
question wrong.  The problem is not that most towns kill startups.
It's that death is the default for startups,
and most towns don't save them.  Instead of thinking of most places
as being sprayed with startupicide, it's more accurate to think of
startups as all being poisoned, and a few places being sprayed with
the antidote.Startups in other places are just doing what startups naturally do:
fail.  The real question is, what's saving startups in places
like Silicon Valley?
[2]EnvironmentI think there are two components to the antidote: being in a place
where startups are the cool thing to do, and chance meetings with
people who can help you.  And what drives them both is the number
of startup people around you.The first component is particularly helpful in the first stage of
a startup's life, when you go from merely having an interest in
starting a company to actually doing it.  It's quite a leap to start
a startup.  It's an unusual thing to do. But in Silicon Valley it
seems normal.
[3]In most places, if you start a startup, people treat you as if
you're unemployed.  People in the Valley aren't automatically
impressed with you just because you're starting a company, but they
pay attention.  Anyone who's been here any amount of time knows not
to default to skepticism, no matter how inexperienced you seem or
how unpromising your idea sounds at first, because they've all seen
inexperienced founders with unpromising sounding ideas who a few
years later were billionaires.Having people around you care about what you're doing is an
extraordinarily powerful force.  Even the
most willful people are susceptible to it.  About a year after we
started Y Combinator I said something to a partner at a well known
VC firm that gave him the (mistaken) impression I was considering
starting another startup.  He responded so eagerly that for about
half a second I found myself considering doing it.In most other cities, the prospect of starting a startup just doesn't
seem real.  In the Valley it's not only real but fashionable.  That
no doubt causes a lot of people to start startups who shouldn't.
But I think that's ok.  Few people are suited to running a startup,
and it's very hard to predict beforehand which are (as I know all
too well from being in the business of trying to predict beforehand),
so lots of people starting startups who shouldn't is probably the
optimal state of affairs.  As long as you're at a point in your
life when you can bear the risk of failure, the best way to find
out if you're suited to running a startup is to try
it.ChanceThe second component of the antidote is chance meetings with people
who can help you.  This force works in both phases: both in the
transition from the desire to start a startup to starting one, and
the transition from starting a company to succeeding.  The power
of chance meetings is more variable than people around you caring
about startups, which is like a sort of background radiation that
affects everyone equally, but at its strongest it is far stronger.Chance meetings produce miracles to compensate for the disasters
that characteristically befall startups.  In the Valley, terrible
things happen to startups all the time, just like they do to startups
everywhere.  The reason startups are more likely to make it here
is that great things happen to them too.  In the Valley, lightning
has a sign bit.For example, you start a site for college students and you decide
to move to the Valley for the summer to work on it.  And then on a
random suburban street in Palo Alto you happen to run into Sean
Parker, who understands the domain really well because he started
a similar startup himself, and also knows all the investors.  And
moreover has advanced views, for 2004, on founders retaining control of their companies.You can't say precisely what the miracle will be, or even for sure
that one will happen.  The best one can say is: if you're in a
startup hub, unexpected good things will probably happen to you,
especially if you deserve them.I bet this is true even for startups we fund.  Even with us working
to make things happen for them on purpose rather than by accident,
the frequency of helpful chance meetings in the Valley is so high
that it's still a significant increment on what we can deliver.Chance meetings play a role like the role relaxation plays in having
ideas.  Most people have had the experience of working hard on some
problem, not being able to solve it, giving up and going to bed,
and then thinking of the answer in the shower in the morning.  What
makes the answer appear is letting your thoughts drift a bit—and thus drift off the wrong
path you'd been pursuing last night and onto the right one adjacent
to it.Chance meetings let your acquaintance drift in the same way taking
a shower lets your thoughts drift. The critical thing in both cases
is that they drift just the right amount.  The meeting between Larry
Page and Sergey Brin was a good example.  They let their acquaintance
drift, but only a little; they were both meeting someone they had
a lot in common with.For Larry Page the most important component of the antidote was
Sergey Brin, and vice versa.  The antidote is
people.  It's not the
physical infrastructure of Silicon Valley that makes it work, or
the weather, or anything like that.  Those helped get it started,
but now that the reaction is self-sustaining what drives it is the
people.Many observers have noticed that one of the most distinctive things
about startup hubs is the degree to which people help one another
out, with no expectation of getting anything in return.  I'm not
sure why this is so.  Perhaps it's because startups are less of a
zero sum game than most types of business; they are rarely killed
by competitors.  Or perhaps it's because so many startup founders
have backgrounds in the sciences, where collaboration is encouraged.A large part of YC's function is to accelerate that process.  We're
a sort of Valley within the Valley, where the density of people
working on startups and their willingness to help one another are
both artificially amplified.NumbersBoth components of the antidote—an environment that encourages
startups, and chance meetings with people who help you—are
driven by the same underlying cause: the number of startup people
around you.  To make a startup hub, you need a lot of people
interested in startups.There are three reasons. The first, obviously, is that if you don't
have enough density, the chance meetings don't happen.
[4]
The second is that different startups need such different things, so
you need a lot of people to supply each startup with what they need
most.  Sean Parker was exactly what Facebook needed in 2004.  Another
startup might have needed a database guy, or someone with connections
in the movie business.This is one of the reasons we fund such a large number of companies,
incidentally.  The bigger the community, the greater the chance it
will contain the person who has that one thing you need most.The third reason you need a lot of people to make a startup hub is
that once you have enough people interested in the same problem,
they start to set the social norms.  And it is a particularly
valuable thing when the atmosphere around you encourages you to do
something that would otherwise seem too ambitious.  In most places
the atmosphere pulls you back toward the mean.I flew into the Bay Area a few days ago.  I notice this every time
I fly over the Valley: somehow you can sense something is going on.
Obviously you can sense prosperity in how well kept a
place looks.  But there are different kinds of prosperity.  Silicon
Valley doesn't look like Boston, or New York, or LA, or DC.  I tried
asking myself what word I'd use to describe the feeling the Valley
radiated, and the word that came to mind was optimism.Notes[1]
I'm not saying it's impossible to succeed in a city with few
other startups, just harder.  If you're sufficiently good at
generating your own morale, you can survive without external
encouragement.  Wufoo was based in Tampa and they succeeded.  But
the Wufoos are exceptionally disciplined.[2]
Incidentally, this phenomenon is not limited to startups.  Most
unusual ambitions fail, unless the person who has them manages to
find the right sort of community.[3]
Starting a company is common, but starting a startup is rare.
I've talked about the distinction between the two elsewhere, but
essentially a startup is a new business designed for scale.  Most
new businesses are service businesses and except in rare cases those
don't scale.[4]
As I was writing this, I had a demonstration of the density of
startup people in the Valley.  Jessica and I bicycled to University
Ave in Palo Alto to have lunch at the fabulous Oren's Hummus.  As
we walked in, we met Charlie Cheever sitting near the door.  Selina
Tobaccowala stopped to say hello on her way out.  Then Josh Wilson
came in to pick up a take out order.  After lunch we went to get
frozen yogurt.  On the way we met Rajat Suri.  When we got to the
yogurt place, we found Dave Shen there, and as we walked out we ran
into Yuri Sagalov.  We walked with him for a block or so and we ran
into Muzzammil Zaveri, and then a block later we met Aydin Senkut.
This is everyday life in Palo Alto.  I wasn't trying to meet people;
I was just having lunch.  And I'm sure for every startup founder
or investor I saw that I knew, there were 5 more I didn't.  If Ron
Conway had been with us he would have met 30 people he knew.Thanks to Sam Altman, Paul Buchheit, Jessica Livingston, and
Harj Taggar for reading drafts of this.


iflisp

May 2003If Lisp is so great, why don't more people use it?  I was
asked this question by a student in the audience at a
talk I gave recently.  Not for the first time, either.In languages, as in so many things, there's not much
correlation between popularity and quality.  Why does
John Grisham (King of Torts sales rank, 44) outsell
Jane Austen (Pride and Prejudice sales rank, 6191)?
Would even Grisham claim that it's because he's a better
writer?Here's the first sentence of Pride and Prejudice:

It is a truth universally acknowledged, that a single man
in possession of a good fortune must be in want of a
wife.

"It is a truth universally acknowledged?"  Long words for
the first sentence of a love story.Like Jane Austen, Lisp looks hard.  Its syntax, or lack
of syntax, makes it look completely unlike
the languages
most people are used to.  Before I learned Lisp, I was afraid
of it too.  I recently came across a notebook from 1983
in which I'd written:

I suppose I should learn Lisp, but it seems so foreign.

Fortunately, I was 19 at the time and not too resistant to learning
new things.  I was so ignorant that learning
almost anything meant learning new things.People frightened by Lisp make up other reasons for not
using it.  The standard
excuse, back when C was the default language, was that Lisp
was too slow.  Now that Lisp dialects are among
the faster
languages available, that excuse has gone away.
Now the standard excuse is openly circular: that other languages
are more popular.(Beware of such reasoning.  It gets you Windows.)Popularity is always self-perpetuating, but it's especially
so in programming languages. More libraries
get written for popular languages, which makes them still
more popular.  Programs often have to work with existing programs,
and this is easier if they're written in the same language,
so languages spread from program to program like a virus.
And managers prefer popular languages, because they give them
more leverage over developers, who can more easily be replaced.Indeed, if programming languages were all more or less equivalent,
there would be little justification for using any but the most
popular.  But they aren't all equivalent, not by a long
shot.  And that's why less popular languages, like Jane Austen's
novels, continue to survive at all.  When everyone else is reading
the latest John Grisham novel, there will always be a few people
reading Jane Austen instead.


know

December 2014I've read Villehardouin's chronicle of the Fourth Crusade at least
two times, maybe three.  And yet if I had to write down everything
I remember from it, I doubt it would amount to much more than a
page.  Multiply this times several hundred, and I get an uneasy
feeling when I look at my bookshelves. What use is it to read all
these books if I remember so little from them?A few months ago, as I was reading Constance Reid's excellent
biography of Hilbert, I figured out if not the answer to this
question, at least something that made me feel better about it.
She writes:

  Hilbert had no patience with mathematical lectures which filled
  the students with facts but did not teach them how to frame a
  problem and solve it. He often used to tell them that "a perfect
  formulation of a problem is already half its solution."

That has always seemed to me an important point, and I was even
more convinced of it after hearing it confirmed by Hilbert.But how had I come to believe in this idea in the first place?  A
combination of my own experience and other things I'd read.  None
of which I could at that moment remember!  And eventually I'd forget
that Hilbert had confirmed it too.  But my increased belief in the
importance of this idea would remain something I'd learned from
this book, even after I'd forgotten I'd learned it.Reading and experience train your model of the world.  And even if
you forget the experience or what you read, its effect on your model
of the world persists.  Your mind is like a compiled program you've
lost the source of.  It works, but you don't know why.The place to look for what I learned from Villehardouin's chronicle
is not what I remember from it, but my mental models of the crusades,
Venice, medieval culture, siege warfare, and so on.  Which doesn't
mean I couldn't have read more attentively, but at least the harvest
of reading is not so miserably small as it might seem.This is one of those things that seem obvious in retrospect.  But
it was a surprise to me and presumably would be to anyone else who
felt uneasy about (apparently) forgetting so much they'd read.Realizing it does more than make you feel a little better about
forgetting, though.  There are specific implications.For example, reading and experience are usually "compiled" at the
time they happen, using the state of your brain at that time.  The
same book would get compiled differently at different points in
your life.  Which means it is very much worth reading important
books multiple times.  I always used to feel some misgivings about
rereading books.  I unconsciously lumped reading together with work
like carpentry, where having to do something again is a sign you
did it wrong the first time.  Whereas now the phrase "already read"
seems almost ill-formed.Intriguingly, this implication isn't limited to books.  Technology
will increasingly make it possible to relive our experiences.  When
people do that today it's usually to enjoy them again (e.g. when
looking at pictures of a trip) or to find the origin of some bug in
their compiled code (e.g. when Stephen Fry succeeded in remembering
the childhood trauma that prevented him from singing).  But as
technologies for recording and playing back your life improve, it
may become common for people to relive experiences without any goal
in mind, simply to learn from them again as one might when rereading
a book.Eventually we may be able not just to play back experiences but
also to index and even edit them. So although not knowing how you
know things may seem part of being human, it may not be.
Thanks to Sam Altman, Jessica Livingston, and Robert Morris for reading
drafts of this.


rss

Aaron Swartz created a scraped
feed
of the essays page.


todo

April 2012A palliative care nurse called Bronnie Ware made a list of the
biggest regrets
of the dying.  Her list seems plausible.  I could see
myself — can see myself — making at least 4 of these
5 mistakes.If you had to compress them into a single piece of advice, it might
be: don't be a cog.  The 5 regrets paint a portrait of post-industrial
man, who shrinks himself into a shape that fits his circumstances,
then turns dutifully till he stops.The alarming thing is, the mistakes that produce these regrets are
all errors of omission.  You forget your dreams, ignore your family,
suppress your feelings, neglect your friends, and forget to be
happy.  Errors of omission are a particularly dangerous type of
mistake, because you make them by default.I would like to avoid making these mistakes.  But how do you avoid
mistakes you make by default?  Ideally you transform your life so
it has other defaults.  But it may not be possible to do that
completely. As long as these mistakes happen by default, you probably
have to be reminded not to make them.  So I inverted the 5 regrets,
yielding a list of 5 commands

   Don't ignore your dreams; don't work too much; say what you
   think; cultivate friendships; be happy.

which I then put at the top of the file I use as a todo list.


vb

January 2016Life is short, as everyone knows. When I was a kid I used to wonder
about this. Is life actually short, or are we really complaining
about its finiteness?  Would we be just as likely to feel life was
short if we lived 10 times as long?Since there didn't seem any way to answer this question, I stopped
wondering about it.  Then I had kids.  That gave me a way to answer
the question, and the answer is that life actually is short.Having kids showed me how to convert a continuous quantity, time,
into discrete quantities. You only get 52 weekends with your 2 year
old.  If Christmas-as-magic lasts from say ages 3 to 10, you only
get to watch your child experience it 8 times.  And while it's
impossible to say what is a lot or a little of a continuous quantity
like time, 8 is not a lot of something.  If you had a handful of 8
peanuts, or a shelf of 8 books to choose from, the quantity would
definitely seem limited, no matter what your lifespan was.Ok, so life actually is short.  Does it make any difference to know
that?It has for me.  It means arguments of the form "Life is too short
for x" have great force.  It's not just a figure of speech to say
that life is too short for something.  It's not just a synonym for
annoying.  If you find yourself thinking that life is too short for
something, you should try to eliminate it if you can.When I ask myself what I've found life is too short for, the word
that pops into my head is "bullshit." I realize that answer is
somewhat tautological.  It's almost the definition of bullshit that
it's the stuff that life is too short for.  And yet bullshit does
have a distinctive character.  There's something fake about it.
It's the junk food of experience.
[1]If you ask yourself what you spend your time on that's bullshit,
you probably already know the answer.  Unnecessary meetings, pointless
disputes, bureaucracy, posturing, dealing with other people's
mistakes, traffic jams, addictive but unrewarding pastimes.There are two ways this kind of thing gets into your life: it's
either forced on you, or it tricks you.  To some extent you have to
put up with the bullshit forced on you by circumstances.  You need
to make money, and making money consists mostly of errands.  Indeed,
the law of supply and demand insures that: the more rewarding some
kind of work is, the cheaper people will do it.  It may be that
less bullshit is forced on you than you think, though.  There has
always been a stream of people who opt out of the default grind and
go live somewhere where opportunities are fewer in the conventional
sense, but life feels more authentic.  This could become more common.You can do it on a smaller scale without moving.  The amount of
time you have to spend on bullshit varies between employers.  Most
large organizations (and many small ones) are steeped in it.  But
if you consciously prioritize bullshit avoidance over other factors
like money and prestige, you can probably find employers that will
waste less of your time.If you're a freelancer or a small company, you can do this at the
level of individual customers.  If you fire or avoid toxic customers,
you can decrease the amount of bullshit in your life by more than
you decrease your income.But while some amount of bullshit is inevitably forced on you, the
bullshit that sneaks into your life by tricking you is no one's
fault but your own.  And yet the bullshit you choose may be harder
to eliminate than the bullshit that's forced on you.  Things that
lure you into wasting your time have to be really good at
tricking you.  An example that will be familiar to a lot of people
is arguing online.  When someone
contradicts you, they're in a sense attacking you. Sometimes pretty
overtly.  Your instinct when attacked is to defend yourself.  But
like a lot of instincts, this one wasn't designed for the world we
now live in.  Counterintuitive as it feels, it's better most of
the time not to defend yourself.  Otherwise these people are literally
taking your life.
[2]Arguing online is only incidentally addictive. There are more
dangerous things than that. As I've written before, one byproduct
of technical progress is that things we like tend to become more
addictive.  Which means we will increasingly have to make a conscious
effort to avoid addictions  to stand outside ourselves and ask "is
this how I want to be spending my time?"As well as avoiding bullshit, one should actively seek out things
that matter.  But different things matter to different people, and
most have to learn what matters to them.  A few are lucky and realize
early on that they love math or taking care of animals or writing,
and then figure out a way to spend a lot of time doing it.  But
most people start out with a life that's a mix of things that
matter and things that don't, and only gradually learn to distinguish
between them.For the young especially, much of this confusion is induced by the
artificial situations they find themselves in. In middle school and
high school, what the other kids think of you seems the most important
thing in the world.  But when you ask adults what they got wrong
at that age, nearly all say they cared too much what other kids
thought of them.One heuristic for distinguishing stuff that matters is to ask
yourself whether you'll care about it in the future.  Fake stuff
that matters usually has a sharp peak of seeming to matter.  That's
how it tricks you.  The area under the curve is small, but its shape
jabs into your consciousness like a pin.The things that matter aren't necessarily the ones people would
call "important."  Having coffee with a friend matters.  You won't
feel later like that was a waste of time.One great thing about having small children is that they make you
spend time on things that matter: them. They grab your sleeve as
you're staring at your phone and say "will you play with me?" And
odds are that is in fact the bullshit-minimizing option.If life is short, we should expect its shortness to take us by
surprise. And that is just what tends to happen.  You take things
for granted, and then they're gone.  You think you can always write
that book, or climb that mountain, or whatever, and then you realize
the window has closed.  The saddest windows close when other people
die. Their lives are short too.  After my mother died, I wished I'd
spent more time with her.  I lived as if she'd always be there.
And in her typical quiet way she encouraged that illusion.  But an
illusion it was. I think a lot of people make the same mistake I
did.The usual way to avoid being taken by surprise by something is to
be consciously aware of it.  Back when life was more precarious,
people used to be aware of death to a degree that would now seem a
bit morbid.  I'm not sure why, but it doesn't seem the right answer
to be constantly reminding oneself of the grim reaper hovering at
everyone's shoulder.  Perhaps a better solution is to look at the
problem from the other end. Cultivate a habit of impatience about
the things you most want to do. Don't wait before climbing that
mountain or writing that book or visiting your mother.  You don't
need to be constantly reminding yourself why you shouldn't wait.
Just don't wait.I can think of two more things one does when one doesn't have much
of something: try to get more of it, and savor what one has.  Both
make sense here.How you live affects how long you live.  Most people could do better.
Me among them.But you can probably get even more effect by paying closer attention
to the time you have.  It's easy to let the days rush by.  The
"flow" that imaginative people love so much has a darker cousin
that prevents you from pausing to savor life amid the daily slurry
of errands and alarms.  One of the most striking things I've read
was not in a book, but the title of one: James Salter's Burning
the Days.It is possible to slow time somewhat. I've gotten better at it.
Kids help.  When you have small children, there are a lot of moments
so perfect that you can't help noticing.It does help too to feel that you've squeezed everything out of
some experience.  The reason I'm sad about my mother is not just
that I miss her but that I think of all the things we could have
done that we didn't.  My oldest son will be 7 soon.  And while I
miss the 3 year old version of him, I at least don't have any regrets
over what might have been.  We had the best time a daddy and a 3
year old ever had.Relentlessly prune bullshit, don't wait to do things that matter,
and savor the time you have.  That's what you do when life is short.Notes[1]
At first I didn't like it that the word that came to mind was
one that had other meanings.  But then I realized the other meanings
are fairly closely related.  Bullshit in the sense of things you
waste your time on is a lot like intellectual bullshit.[2]
I chose this example deliberately as a note to self.  I get
attacked a lot online.  People tell the craziest lies about me.
And I have so far done a pretty mediocre job of suppressing the
natural human inclination to say "Hey, that's not true!"Thanks to Jessica Livingston and Geoff Ralston for reading drafts
of this.