Revenge of the Nerds
Want to start a startup?
Get funded by
Y Combinator
.
May 2002
"We were after the C++ programmers. We managed to drag a
lot of them about halfway to Lisp."
- Guy Steele, co-author of the Java spec
In the software business there is an ongoing
struggle between the pointy-headed academics, and another
equally formidable force, the pointy-haired bosses.  Everyone
knows who the pointy-haired boss is, right?  I think most
people in the technology world not only recognize this
cartoon character, but know the actual person in their company
that he is modelled upon.
The pointy-haired boss miraculously combines two qualities
that are common by themselves, but rarely seen together:
(a) he knows nothing whatsoever about technology, and
(b) he has very strong opinions about it.
Suppose, for example, you need to write a piece of software.
The pointy-haired boss has no idea how this software
has to work, and can't tell one programming language from
another, and yet he knows what language you should write it in.
Exactly.  He thinks you should write it in Java.
Why does he think this?  Let's
take a look inside the brain of the pointy-haired boss.  What
he's thinking is something like this.  Java is a standard.
I know it must be, because I read about it in the press all the time.
Since it is a standard, I won't get in trouble for using it.
And that also means there will always be lots of Java programmers,
so if the programmers working for me now quit, as programmers
working for me mysteriously always do, I can easily replace
them.
Well, this doesn't sound that unreasonable.  But it's all
based on one unspoken assumption, and that assumption
turns out to be false.  The pointy-haired boss believes that all
programming languages are pretty much equivalent.
If that were true, he would be right on
target.  If languages are all equivalent, sure, use whatever
language everyone else is using.
But all languages are not equivalent, and I think I can prove
this to you without even getting into the differences between them.
If you asked the pointy-haired boss in 1992 what language
software should be written in, he would have answered with as
little hesitation as he does today.  Software should be
written in C++.  But if languages are all equivalent, why should the
pointy-haired boss's opinion ever change?  In fact, why should
the developers of Java have even bothered to create a new
language?
Presumably, if you create a new language, it's because you think
it's better in some way than what people already had.  And in fact, Gosling
makes it clear in the first Java white paper that Java
was designed to fix some problems with C++.
So there you have it: languages are not all equivalent.
If you follow the
trail through the pointy-haired boss's brain to Java and then
back through Java's history to its origins, you end up holding
an idea that contradicts the assumption you started with.
So, who's right?  James Gosling, or the pointy-haired boss?
Not surprisingly, Gosling is right.  Some languages
are
better,
for certain problems, than others.  And you know, that raises some
interesting questions.  Java was designed to be better, for certain
problems, than C++.  What problems?  When is Java better and
when is C++?  Are there situations where other languages are
better than either of them?
Once you start considering this question, you have opened a
real can of worms.  If the pointy-haired boss had to think
about the problem in its full complexity, it would make his
brain explode.  As long as he considers all languages
equivalent, all he has to do is choose the one
that seems to have the most momentum, and since that is more
a question of fashion than technology, even he
can probably get the right answer.
But if languages vary, he suddenly
has to solve two simultaneous equations, trying to find
an optimal balance between two things he knows nothing
about: the relative suitability of the twenty or so leading
languages for the problem he needs to solve, and the odds of
finding programmers, libraries, etc. for each.
If that's what's on the other side of the door, it
is no surprise that the pointy-haired boss doesn't want to open it.
The disadvantage of believing that all programming languages
are equivalent is that it's not true.  But the advantage is
that it makes your life a lot simpler.
And I think that's the main reason the idea is so widespread.
It is a
comfortable
idea.
We know that Java must be pretty good, because it is the
cool, new programming language.  Or is it?  If you look at the world of
programming languages from a distance, it looks like Java is
the latest thing.  (From far enough away, all you can see is
the large, flashing billboard paid for by Sun.)
But if you look at this world
up close, you find that there are degrees of coolness.  Within
the hacker subculture, there is another language called Perl
that is considered a lot cooler than Java.  Slashdot, for
example, is generated by Perl.  I don't think you would find
those guys using Java Server Pages.  But there is another,
newer language, called Python, whose users tend to look down on Perl,
and
more
waiting in the wings.
If you look at these languages in order, Java, Perl, Python,
you notice an interesting pattern.  At least, you notice this
pattern if you are a Lisp hacker.  Each one is progressively
more like Lisp.  Python copies even features
that many Lisp hackers consider to be mistakes.
You could translate simple Lisp programs into Python line for line.
It's 2002, and programming languages have almost caught up
with 1958.
Catching Up with Math
What I mean is that
Lisp was first discovered by John McCarthy in 1958,
and popular programming languages are only now
catching up with the ideas he developed then.
Now, how could that be true?  Isn't computer technology something
that changes very rapidly?  I mean, in 1958, computers were
refrigerator-sized behemoths with the processing power of
a wristwatch.  How could any technology that old even be
relevant, let alone superior to the latest developments?
I'll tell you how.  It's because Lisp was not really
designed to be a programming language, at least not in the sense
we mean today.  What we mean by a programming language is
something we use to tell a computer what to do.   McCarthy
did eventually intend to develop a programming language in
this sense, but the Lisp that we actually ended up with was based
on something separate that he did as a
theoretical exercise
-- an effort
to define a more convenient alternative to the Turing Machine.
As McCarthy said later,
Another way to show that Lisp was neater than Turing machines
was to write a universal Lisp function
and show that it is briefer and more comprehensible than the
description of a universal Turing machine.
This was the Lisp function
eval
...,
which computes the value of
a Lisp expression....
Writing
eval
required inventing a notation representing Lisp
functions as Lisp data, and such a notation
was devised for the purposes of the paper with no thought that
it would be used to express Lisp programs in practice.
What happened next was that, some time in late 1958, Steve Russell,
one of McCarthy's
grad students, looked at this definition of
eval
and realized
that if he translated it into machine language, the result
would be a Lisp interpreter.
This was a big surprise at the time.
Here is what McCarthy said about it later in an interview:
Steve Russell said, look, why don't I program this
eval
..., and
I said to him, ho, ho, you're confusing theory with practice,
this
eval
is intended for reading, not for
computing. But he went ahead and did it. That is, he compiled the
eval
in my paper into [IBM] 704 machine
code, fixing bugs, and then advertised this as a Lisp interpreter,
which it certainly was. So at that point Lisp
had essentially the form that it has today....
Suddenly, in a matter of weeks I think, McCarthy found his theoretical
exercise transformed into an actual programming language-- and a
more powerful one than he had intended.
So the short explanation of why this 1950s language is not
obsolete is that it was not technology but math, and
math doesn't get stale.   The right thing to compare Lisp
to is not 1950s hardware, but, say, the Quicksort
algorithm, which was discovered in 1960 and is still
the fastest general-purpose sort.
There is one other language still
surviving from the 1950s, Fortran, and it represents the
opposite approach to language design.  Lisp was a
piece of theory that unexpectedly got turned into a
programming language.  Fortran was developed intentionally as
a programming language, but what we would now consider a
very low-level one.
Fortran I
, the language that was
developed in 1956, was a very different animal from present-day
Fortran.   Fortran I was pretty much assembly
language with math.  In some ways it was less
powerful than more recent assembly languages; there were no
subroutines, for example, only branches.
Present-day Fortran is now arguably closer to Lisp than to
Fortran I.
Lisp and Fortran were the trunks of two separate evolutionary trees,
one rooted in math and one rooted in machine architecture.
These two trees have been converging ever since.
Lisp started out powerful, and over the next twenty years
got fast.  So-called mainstream languages started out
fast, and over the next forty years gradually got more powerful,
until now the most advanced
of them are fairly close to Lisp.
Close, but they are still missing a few things....
What Made Lisp Different
When it was first developed, Lisp embodied nine new
ideas.  Some of these we now take for granted, others are
only seen in more advanced languages, and two are still
unique to Lisp.  The nine ideas are, in order of their
adoption by the mainstream,
Conditionals.  A conditional is an if-then-else
construct.  We take these for granted now, but Fortran I
didn't have them. It had only a conditional goto
closely based on the underlying machine instruction.
A function type. In Lisp, functions are
a data type just like integers or strings.
They have a literal representation, can be stored in variables,
can be passed as arguments, and so on.
Recursion.  Lisp was the first programming language to
support it.
Dynamic typing.  In Lisp, all variables
are effectively pointers. Values are what
have types, not variables, and assigning or binding
variables means copying pointers, not what they point to.
Garbage-collection.
Programs composed of expressions.  Lisp programs are
trees of expressions, each of which returns a value.
This is in contrast to Fortran
and most succeeding languages, which distinguish between
expressions and statements.
It was natural to have this
distinction in Fortran I because
you could not nest statements.  And
so while you needed expressions for math to work, there was
no point in making anything else return a value, because
there could not be anything waiting for it.
This limitation
went away with the arrival of block-structured languages,
but by then it was too late. The distinction between
expressions and statements was entrenched.  It spread from
Fortran into Algol and then to both their descendants.
A symbol type.  Symbols are effectively pointers to strings
stored in a hash table.  So
you can test equality by comparing a pointer,
instead of comparing each character.
A notation for code using trees of symbols and constants.
The whole language there all the time.  There is
no real distinction between read-time, compile-time, and runtime.
You can compile or run code while reading, read or run code
while compiling, and read or compile code at runtime.
Running code at read-time lets users reprogram Lisp's syntax;
running code at compile-time is the basis of macros; compiling
at runtime is the basis of Lisp's use as an extension
language in programs like Emacs; and reading at runtime
enables programs to communicate using s-expressions, an
idea recently reinvented as XML.
When Lisp first appeared, these ideas were far
removed from ordinary programming practice, which was
dictated largely by the hardware available in the late 1950s.
Over time, the default language, embodied
in a succession of popular languages, has
gradually evolved toward Lisp.  Ideas 1-5 are now widespread.
Number 6 is starting to appear in the mainstream.
Python has a form of 7, though there doesn't seem to be
any syntax for it.
As for number 8, this may be the most interesting of the
lot.  Ideas 8 and 9 only became part of Lisp
by accident, because Steve Russell implemented
something McCarthy had never intended to be implemented.
And yet these ideas turn out to be responsible for
both Lisp's strange appearance and its most distinctive
features.  Lisp looks strange not so much because
it has a strange syntax as because it has no syntax;
you express programs directly in the parse trees that
get built behind the scenes when other languages are
parsed, and these trees are made
of lists, which are Lisp data structures.
Expressing the language in its own data structures turns
out to be a very powerful feature. Ideas 8 and 9
together mean that you
can write programs that write programs.  That may sound
like a bizarre idea, but it's an everyday thing in Lisp.
The most common way to do it is with something called a
macro.
The term "macro" does not mean in Lisp what it means in other
languages.
A Lisp macro can be anything from an abbreviation
to a compiler for a new language.
If you want to really understand Lisp,
or just expand your programming horizons, I would
learn more
about macros.
Macros (in the Lisp sense) are still, as far as
I know, unique to Lisp.
This is partly because in order to have macros you
probably have to make your language look as strange as
Lisp.  It may also be because if you do add that final
increment of power, you can no
longer claim to have invented a new language, but only
a new dialect of Lisp.
I mention this mostly
as a joke, but it is quite true. If you define
a language that has car, cdr, cons, quote, cond, atom,
eq, and
a notation for functions expressed as lists, then you
can build all the rest of Lisp out of it.  That is in
fact the defining quality of Lisp: it was in order to
make this so that McCarthy gave Lisp the shape it has.
Where Languages Matter
So suppose Lisp does represent a kind of limit
that mainstream languages are approaching asymptotically-- does
that mean you should actually use it to write software?
How much do you lose by using a less powerful language?
Isn't it wiser, sometimes, not to be
at the very edge of innovation?
And isn't popularity to some extent
its own justification?  Isn't the pointy-haired boss right,
for example, to want to use a language for which he can easily
hire programmers?
There are, of course, projects where the choice of programming
language doesn't matter much.  As a
rule, the more demanding the application, the more
leverage you get from using a powerful language.  But
plenty of projects are not demanding at all.
Most programming probably consists of writing
little glue programs, and for
little glue programs you
can use any language that you're already
familiar with and that has good libraries for whatever you
need to do.  If you just need to feed data from one
Windows app to another, sure, use Visual Basic.
You can write little glue programs in Lisp too
(I use it as a desktop calculator), but the biggest win
for languages like Lisp is at the other end of
the spectrum, where you need to write sophisticated
programs to solve hard problems in the face of fierce competition.
A good example is the
airline fare search program
that ITA Software licenses to
Orbitz.  These
guys entered a market already dominated by two big,
entrenched competitors, Travelocity and Expedia, and
seem to have just humiliated them technologically.
The core of ITA's application is a 200,000 line Common Lisp program
that searches many orders of magnitude more possibilities
than their competitors, who apparently
are still using mainframe-era programming techniques.
(Though ITA is also in a sense
using a mainframe-era programming language.)
I have never seen any of ITA's code, but according to
one of their top hackers they use a lot of macros,
and I am not surprised to hear it.
Centripetal Forces
I'm not saying there is no cost to using uncommon
technologies.  The pointy-haired boss is not completely
mistaken to worry about this.  But because he doesn't understand
the risks, he tends to magnify them.
I can think of three problems that could arise from using
less common languages.  Your programs might not work well with
programs written in other languages.  You might have fewer
libraries at your disposal.  And you might have trouble
hiring programmers.
How much of a problem is each of these?  The importance of
the first varies depending on whether you have control
over the whole system.  If you're writing software that has
to run on a remote user's machine on top of a buggy,
closed operating system (I mention no names), there may be
advantages to writing your application in the
same language as the OS.
But if you control the whole system and
have the source code of all the parts, as ITA presumably does, you
can use whatever languages you want.  If
any incompatibility arises, you can fix it yourself.
In server-based applications you can
get away with using the most advanced technologies,
and I think this is the main
cause of what Jonathan Erickson calls the "
programming language
renaissance
."  This is why we even hear about new
languages like Perl and Python.  We're not hearing about these
languages because people are using them to write Windows
apps, but because people are using them on servers.  And as
software shifts
off the desktop
and onto servers (a future even
Microsoft seems resigned to), there will be less
and less pressure to use middle-of-the-road technologies.
As for libraries, their importance also
depends on the application.  For less demanding problems,
the availability of libraries can outweigh the intrinsic power
of the language.  Where is the breakeven point?  Hard to say
exactly, but wherever it is, it is short of anything you'd
be likely to call an application.  If a company considers
itself to be in the software business, and they're writing
an application that will be one of their products,
then it will probably involve several hackers and take at
least six months to write.  In a project of that
size, powerful languages probably start to outweigh
the convenience of pre-existing libraries.
The third worry of the pointy-haired boss, the difficulty
of hiring programmers, I think is a red herring.   How many
hackers do you need to hire, after all?  Surely by now we
all know that software is best developed by teams of less
than ten people.   And you shouldn't have trouble hiring
hackers on that scale for any language anyone has ever heard
of.  If you can't find ten Lisp hackers, then your company is
probably based in the wrong city for developing software.
In fact, choosing a more powerful language probably decreases the
size of the team you need, because (a) if you use a more powerful
language you probably won't need as many hackers,
and (b) hackers who work in more advanced languages are likely
to be smarter.
I'm not saying that you won't get a lot of pressure to use
what are perceived as "standard" technologies.  At Viaweb
(now Yahoo Store),
we raised some eyebrows among VCs and potential acquirers by
using Lisp.  But we also raised eyebrows by using
generic Intel boxes as servers instead of
"industrial strength" servers like Suns, for using a
then-obscure open-source Unix variant called FreeBSD instead
of a real commercial OS like Windows NT, for ignoring
a supposed e-commerce standard called
SET
that no one now
even remembers, and so on.
You can't let the suits make technical decisions for you.
Did it
alarm some potential acquirers that we used Lisp?  Some, slightly,
but if we hadn't used Lisp, we wouldn't have been
able to write the software that made them want to buy us.
What seemed like an anomaly to them was in fact
cause and effect.
If you start a startup, don't design your product to please
VCs or potential acquirers.
Design your product to please
the users.
If you win the users, everything else will
follow.  And if you don't, no one will care
how comfortingly orthodox your technology choices were.
The Cost of Being Average
How much do you lose by using a less powerful language?
There is actually some data out there about that.
The most convenient measure of power is probably
code size
.
The point of high-level
languages is to give you bigger abstractions-- bigger bricks,
as it were, so you don't need as many to build
a wall of a given size.
So the more powerful
the language, the shorter the program (not simply in
characters, of course, but in distinct elements).
How does a more powerful language enable you to write
shorter programs?  One technique you can use, if the language will
let you, is something called
bottom-up programming
.  Instead of
simply writing your application in the base language, you
build on top of the base language a language for writing
programs like yours, then write your program
in it. The combined code can be much shorter than if you
had written your whole program in the base language-- indeed,
this is how most compression algorithms work.
A bottom-up program should be easier to modify as well,
because in many cases the language layer won't have to change
at all.
Code size is important, because the time it takes
to write a program depends mostly on its length.
If your program would be three times as long in another
language, it will take three times as long to write-- and
you can't get around this by hiring more people, because
beyond a certain size new hires are actually a net lose.
Fred Brooks described this phenomenon in his famous
book
The Mythical Man-Month,
and everything I've seen
has tended to confirm what he said.
So how much shorter are your programs if you write them in
Lisp?  Most of the numbers I've heard for Lisp
versus C, for example, have been around 7-10x.
But a recent article about ITA in
New
Architect
magazine said that
"one line of Lisp can replace 20 lines of C," and since
this article was full of quotes from ITA's president, I
assume they got this number from ITA.  If so then
we can put some faith in it; ITA's software includes a lot
of C and  C++ as well as Lisp, so they are speaking from
experience.
My guess is that these multiples aren't even constant.
I think they increase when
you face harder problems and also when you have smarter
programmers.  A really good hacker can squeeze more
out of better tools.
As one data point on the curve, at any rate,
if you were to compete with ITA and
chose to write your software in C, they would be able to develop
software twenty times faster than you.
If you spent a year on a new feature, they'd be able to
duplicate it in less than three weeks.  Whereas if they spent
just three months developing something new, it would be
five years
before you had it too.
And you know what?  That's the best-case scenario.
When you talk about code-size ratios, you're implicitly assuming
that you can actually write the program in the weaker language.
But in fact there are limits on what programmers can do.
If you're trying to solve a hard problem with a language that's
too low-level, you reach a point where there is just too
much to keep in your head at once.
So when I say it would take ITA's imaginary
competitor five years to duplicate something ITA could
write in Lisp in three months, I mean five years
if nothing goes wrong.  In fact, the way things work in
most companies, any
development project that would take five years is
likely never to get finished at all.
I admit this is an extreme case.  ITA's hackers seem to
be unusually smart, and C is a pretty low-level language.
But in a competitive market, even a differential of two or
three to one would
be enough to guarantee that you'd always be behind.
A Recipe
This is the kind of possibility that the pointy-haired boss
doesn't even want to think about.  And so most of them don't.
Because, you know, when it comes down to it, the pointy-haired
boss doesn't mind if his company gets their ass kicked, so
long as no one can prove it's his fault.
The safest plan for him personally
is to stick close to the center of the herd.
Within large organizations, the phrase used to
describe this approach is "industry best practice."
Its purpose is to shield the pointy-haired
boss from responsibility: if he chooses
something that is "industry best practice," and the company
loses, he can't be blamed.  He didn't choose, the industry did.
I believe this term was originally used to describe
accounting methods and so on.  What it means, roughly,
is
don't do anything weird.
And in accounting that's
probably a good idea.  The terms "cutting-edge" and
"accounting" do not sound good together.  But when you import
this criterion into decisions about technology, you start
to get the wrong answers.
Technology often
should
be
cutting-edge.  In programming languages, as Erann Gat
has pointed out, what "industry best practice"  actually
gets you is not the best, but merely the
average.  When a decision causes you to develop software at
a fraction of the rate of more aggressive competitors,
"best practice" is a misnomer.
So here we have two pieces of information that I think are
very valuable.  In fact, I know it from my own experience.
Number 1, languages vary in power.  Number 2, most managers
deliberately ignore this.  Between them, these two facts
are literally a recipe for making money.  ITA is an example
of this recipe in action.
If you want to win in a software
business, just take on the hardest problem you can find,
use the most powerful language you can get, and wait for
your competitors' pointy-haired bosses to revert to the mean.
Appendix: Power
As an illustration of what I mean about the relative power
of programming languages, consider the following problem.
We want to write a function that generates accumulators-- a
function that takes a number n, and
returns a function that takes another number i and
returns n incremented by i.
(That's
incremented by
, not plus.  An accumulator
has to accumulate.)
In Common Lisp this would be
(defun foo (n)
  (lambda (i) (incf n i)))
and in Perl 5,
sub foo {
  my ($n) = @_;
  sub {$n += shift}
}
which has more elements than the Lisp version because
you have to extract parameters manually in Perl.
In Smalltalk the code is slightly longer than in Lisp
foo: n
  |s|
  s := n.
  ^[:i| s := s+i. ]
because although in general lexical variables work, you can't
do an assignment to a parameter, so you have to create a
new variable s.
In Javascript the example is, again, slightly longer, because
Javascript retains
the distinction between statements and
expressions, so you need explicit
return
statements
to return values:
function foo(n) {
  return function (i) {
           return n += i } }
(To be fair, Perl also retains
this distinction, but deals with it in typical Perl fashion
by letting you omit
return
s.)
If you try to translate the Lisp/Perl/Smalltalk/Javascript code into
Python you run into some limitations.  Because Python
doesn't fully support lexical variables,
you have to create a data structure to hold the value of n.
And although
Python does have a function data type, there is no
literal representation for one (unless the body is
only a single expression) so you need to create a named
function to return.  This is what you end up with:
def foo(n):
  s = [n]
  def bar(i):
    s[0] += i
    return s[0]
  return bar
Python users might legitimately ask why they can't
just write
def foo(n):
  return lambda i: return n += i
or even
def foo(n):
  lambda i: n += i
and my guess is that they probably will, one day.
(But if they don't want to wait for Python to evolve the rest
of the way into Lisp, they could always just...)
In OO languages, you can, to a limited extent, simulate
a closure (a function that refers to variables defined in
enclosing scopes) by defining a class with one method
and a field to replace each variable from an enclosing
scope.  This makes the programmer do the kind of code
analysis that would be done by the compiler in a language
with full support for lexical scope, and it won't work
if more than one function refers to the same variable,
but it is enough in simple cases like this.
Python experts seem to agree that this is the
preferred way to solve the problem in Python, writing
either
def foo(n):
  class acc:
    def __init__(self, s):
        self.s = s
    def inc(self, i):
        self.s += i
        return self.s
  return acc(n).inc
or
class foo:
  def __init__(self, n):
      self.n = n
  def __call__(self, i):
      self.n += i
      return self.n
I include these because I wouldn't want Python
advocates to say I was misrepresenting the language,
but both seem to me more complex than the first
version.  You're doing the same thing, setting up
a separate place to hold the accumulator; it's just
a field in an object instead of the head of a list.
And the use of these special,
reserved field names, especially
__call__
, seems
a bit of a hack.
In the rivalry between Perl and Python, the claim of the
Python hackers seems to be that
that Python is a more elegant alternative to Perl, but what
this case shows is that power is the ultimate elegance:
the Perl program is simpler (has fewer elements), even if the
syntax is a bit uglier.
How about other languages? In the other languages
mentioned in this talk-- Fortran, C, C++, Java, and
Visual Basic-- it is not clear whether you can actually
solve this problem.
Ken Anderson says that the following code is about as close
as you can get in Java:
public interface Inttoint {
  public int call(int i);
}
public static Inttoint foo(final int n) {
  return new Inttoint() {
    int s = n;
    public int call(int i) {
    s = s + i;
    return s;
    }};
}
This falls short of the spec because it only works for
integers.  After many email exchanges with Java hackers,
I would say that writing a properly polymorphic version
that behaves like the preceding examples is somewhere
between damned awkward and impossible.  If anyone wants to
write one I'd be very curious to see it, but I personally
have timed out.
It's not literally true that you can't solve this
problem in other languages, of course.  The fact
that all these languages are Turing-equivalent means
that, strictly speaking, you can write any program in
any of them.  So how would you do it?  In the limit case,
by writing a Lisp
interpreter in the less powerful language.
That sounds like a joke, but it happens so often to
varying degrees in large programming projects that
there is a name for the phenomenon, Greenspun's Tenth
Rule:
Any sufficiently
     complicated C or Fortran program contains an ad hoc
     informally-specified bug-ridden slow implementation of half of
     Common Lisp.
If you try to solve a
hard problem, the question is not whether you will use
a powerful enough language, but whether you will (a)
use a powerful language, (b) write a de facto interpreter
for one, or (c) yourself become a human compiler for one.
We see this already
begining to happen in the Python example, where we are
in effect simulating the code that a compiler
would generate to implement a lexical variable.
This practice is not only common, but institutionalized.  For example,
in the OO world you hear a good deal about
"patterns".
I wonder if these patterns are not sometimes evidence of case (c),
the human compiler, at work.  When I see patterns in my programs,
I consider it a sign of trouble.  The shape of a program
should reflect only the problem it needs to solve.
Any other regularity in the code is a sign, to me at
least, that I'm using abstractions that aren't powerful
enough-- often that I'm generating by hand the
expansions of some macro that I need to write.
Notes
The IBM 704 CPU was about the size of a refrigerator,
but a lot heavier.  The CPU weighed 3150 pounds,
and the 4K of RAM was in a separate
box weighing another 4000 pounds.  The
Sub-Zero 690, one of the largest household refrigerators,
weighs 656 pounds.
Steve Russell also wrote the first (digital) computer
game, Spacewar, in 1962.
If you want to trick a pointy-haired boss into letting you
write software in Lisp, you could try telling him it's XML.
Here is the accumulator generator in other Lisp dialects:
Scheme: (define (foo n)
          (lambda (i) (set! n (+ n i)) n))
Goo:    (df foo (n) (op incf n _)))
Arc:    (def foo (n) [++ n _])
Erann Gat's sad tale about
"industry best practice" at JPL inspired me to address
this generally misapplied phrase.
Peter Norvig found that
16 of the 23 patterns in
Design Patterns
were
"
invisible
or simpler
" in Lisp.
Thanks to the many people who answered my questions about
various languages and/or read drafts of this, including
Ken Anderson, Trevor Blackwell, Erann Gat, Dan Giffin, Sarah Harlin,
Jeremy Hylton, Robert Morris, Peter Norvig, Guy Steele, and Anton
van Straaten.
They bear no blame for any opinions expressed.
Related:
Many people have responded to this talk,
so I have set up an additional page to deal with the issues they have
raised:
Re: Revenge of the Nerds
.
It also set off an extensive and often useful discussion on the
LL1
mailing list.  See particularly the mail by Anton van Straaten on semantic
compression.
Some of the mail on LL1 led me to try to go deeper into the subject
of language power in
Succinctness is Power
.
A larger set of canonical implementations of the
accumulator
generator benchmark
are collected together on their own page.
Japanese Translation
,
Spanish
Translation
,
Chinese Translation
You'll find this essay and 14 others in
Hackers & Painters
.


@tlakomy Indeed, when you phrase it that way it's impressive that it works at all.

@spctrexe @NateSilver538 I live in England. And I just used cal.

People who think that DoorDash costs a lot because of "corporate greed" could not have picked a worse example. Making money in food delivery is hard. Most startups that try it fail. The only reason DoorDash has done so well is that Tony Xu is so brilliantly efficient.

@CarterKoWang @jonchu I think they should use TCP/IP.

@NateSilver538 You were born on Friday the 13th too!

@NJGenerals85 @Sam_Dumitriu The goal of law in a civilized society is justice, not efficiency.

@Sam_Dumitriu The principle that the punishment should fit the crime.

@SRinnebach @ycombinator Good point.

@growing_daniel I got one a couple days ago attacking me for being a white male. I was briefly nostalgic, before blocking her.

Something I taught 12 yo: When something is described as "academically challenging," that's good. That means you learn a lot.


@TheBabyMets That certainly helped, but it clearly wasn't necessary to clean up for Xi.

@samdaceo Only a few bits of the Bay Area are bad: certain neighborhoods in downtown SF and Oakland. You'll be fine.

If you don't know PB, he was the guy who created GMail, among other things. He's one of the sharpest thinkers in Silicon Valley. If I were creating a committee of smart people to predict the future of technology, he'd be on it.

A friend who lives in SF says things are now getting slowly better. When I asked what was the turning point, he thought for a moment and said Xi's visit in late 2023. When SF residents saw how fast they cleaned up the city for that, they asked "Why can't it always be like this?"

@michael_nielsen It may be that random rather than sequential access is the way to read Gibbon.

@egrefen @michael_nielsen Newton.

@librarythingtim I'd say it's the opposite. Except for the cost of compute, which is a novel thing to worry about, startups have the advantage now.

@michael_nielsen The Decline and Fall, the Principia, Euclid, Knuth.

@rosslazer Things moved fast then, but not this fast.

Even though I'm a bit frightened of AI, it's a very exciting time to be involved with startups. My favorite kind of office hours are where we talk about wildly ambitious things the company could do in the future, and there are a lot more of them now.


@footballmood I consciously started Viaweb to get rich.

@tunguz Whether you get a lot per customer or a little, you still need to get paid in proportion to the number of them.

@GrosuCodrut Good, young programmers inherently do, even if they have no money. They can alway get an ordinary job if they try a more risky strategy and it fails.

@migueldeicaza And sadly, it's because our communities are mostly silent that this can happen.

Imagine how vocal they'd be if France were doing this.

@MatsoLife You guessed it: start a company doing that kind of work.

@markov_yuriy Start a startup or become an early employee of one.

@nikitabier I bet some people make a lot from it.

There are other ways to get rich, and 100 edge cases, but the high bit of getting rich is to work for equity rather than salary.

To get rich, do work where you get paid in proportion to the number of customers you satisfy instead of the number of hours you work. Not all work of this kind will make you rich, but nearly all work that can make you rich is of this kind.

@itsKinjalNandy I'm in Switzerland.


@korgaonkar Not exactly the opposite. The Thiel fellowship isn't to start a startup. It's to work on speculative projects of any kind. And that is what I'm claiming is one of the advantages of college over startups: that you can explore anything that interests you.

@Alessandro90947 You can have both, and in fact you'll do better at the startup if you do.

@_mattneary You got lucky. One year is ok. The danger is when one year turns into three.

@KarthikIO Every year some things stop being worth building and other new things become possible to build.

And even if there were waves, how do you know you're not too early? How do you know the peak of opportunity won't be a few years from now?

@diptinto You have to flip so many heads to get an outlier like Microsoft that any one of them being a tails would decrease the outcome. E.g. he probably also wouldn't have started such a big company if he hadn't grown up in Seattle, where he met Paul Allen.

@ForrestPKnight 1. You can't always go back to college. If you drop out for more than a year or two, you'll be older than the other students when you return and it won't be the same experience.

2. The iron is always hot.

@GYuvrender If they're founders, then by definition they're not. In college you can explore all kinds of different things just because they might be interesting. If you do that as a founder, you're doing it wrong.

@diptinto No. Anyone who could start a successful startup at 18 could start an even more successful one at 22. And have a better life as well. So even for the most talented founders, I'd advise waiting.

@danielcincu For most people who start startups it's not that great an experience either.

@_mattneary How long did you stay dropped out? You could leave for a year and come back, but if you stay away much longer, you're older than the other students and it's not the same experience.


"In families, love flows downhill."

@BCalStewardess That livery helps. I don't think I've ever seen a better match between plane and livery.

@shashj @OborneTweets Why do you think he did that?

Good sci-fi writers usually insure themselves against technological progress by not being too specific about how things work. But it's hard not to be specific about who's doing things. That's what a plot is.

The rate of progress in AI must be making it hard to write science fiction right now. To appeal to human readers you want to make humans (or living creatures at least) solve problems, but if you do the shelf life of your story could be short.

@4tressbalansht It's the other way around.

@rupwalker The mistake is to think that merely going to college is good. It matters what you study and how you study it.

@davidiach I wouldn't care either way.

@ali_moiz I rarely delete text files.

@dqnamo It's more true of a top university but a smart person can learn a lot even at a bad one. The math professors at a mediocre university will still be good research mathematicians.


"College educated youth still favor Biden quite a bit, however due to Gaza we've seen support levels in some surveys drop as much as 20 points."

— a Democratic party insider

@yusufbirader Physical stores are better for discovery.

@thinkplantgrow I walk past them when they're closed.

@mr_james_c @paultoo People hate him partly because he's rich (you can only be rich and loved if you're self-effacing) and partly because he says heretical things.

@HistoryEJ London this time. I think I'll go back to Hay again soon though.

@2sush My Family and Other Animals.

I never know what I'm going to buy. I was in the market for another copy of Jaeger's book on Aristotle, after I noticed mine is horribly written-in, but I didn't find one.

I'm happy to report that Miguel de Icaza is out of shadowban jail. (He was shadowbanned after people filed false reports against him to prevent him from speaking out about Gaza.) If you enjoyed his tweets, you can follow him at @migueldeicaza. Now back to our regular programming.

I get to go used book shopping today. Hooray!

To celebrate Reddit's IPO, I wrote something about what made them special (and still does): https://t.co/ee5NgoT5R1

@Austen Half a million.


If your beliefs are correct, you shouldn't have to post fake news. There should be enough real news supporting them.

So if you post fake news, it means there's insufficient evidence supporting your beliefs.

@sampullara @WallStreetSilv I'd have to see the videos of what he actually did. Apparently you don't need to.

@sampullara @WallStreetSilv It's not hard at all to find the full story.

https://t.co/NFa7xzJfSq

@WallStreetSilv If you did even a little research you'd find he was accused of making threatening gestures at the police and calling them names, not simply singing this song.

@0xngmi No. There isn't much attrition yet after only 6 months.

@Scobleizer Maybe, but no programmer would prefer software to be used in such a clumsy way.

A few days ago during office hours with a startup, we actually found ourselves asking: if in a few years no humans are doing the work our software is for, would AIs use it? It's getting that weird. What do you do, make sure you have an API as well as a web app?

@gemini_harry Wokeness peaked in 2020.

Source: https://t.co/hpaYjbHh1c

@levsencapital @elonmusk It's a nonprofit.


Politicians and activists who try to prevent the use of self-driving cars are trying to increase the number people hurt in car crashes by 12x. https://t.co/asNQIwK96B

@rauchg @GoogleAI You mean in the good sense?

@urivalev @omarsuleiman What's weird about it is that there are dead children in the ruins.

@JasonFebery Now do math.

@AgnesCallard It's quite clear from reading passing remarks they've made about the field that mathematicians would not rather be producing philosophy. Or probably anything else, to be fair.

@FletcherChristy I can think of three off the top of my head (though one is now an administrator). But you don't need to have them as friends to see this.

@omarsuleiman If there had been podcasts in Germany in 1942, you know what the equivalent people would have been saying.

Something I told 12 yo: Professors teaching literature would rather be producing it. Professors teaching math are producing it.

@jeremyscahill Need to know basis.

@jasonlk Did they raise in 2021?


@garrytan George Soros gets blamed for all these donations, but hasn't it been actually his son for years?

@GarettJones Too much trademark in the logo.

@Xhris57 I don't think that word means what you think it means.

@reidhoffman I do something similar with prewritten answers that are cached on paper.

@danpatcronin I've seen them fly overhead a couple times.

@wk1rby 13 is his new age.

@jdmccafferty Clouet was a good influence. Anything by someone described as a follower of Clouet tends to be good.

Six boys are here for 13 yo's birthday party. It's a sunny day and they're running around outside. A plane flies over. Is that what I think it is? Nothing else sounds like that. Yes, it's a Spitfire.

@elonmusk Interesting, but I'm talking mostly about what I see in replies and retweets.

@TedHulsyTweets You're kind of proving my point here, Ted.


@AntWilson That's impressive even with a normal aspect ratio. https://t.co/mrfAOCnJbj

@chribjel @DudespostingWs You will when you have children.

There's no reason we can't fix this regulation right now. We don't have to wait for the planes.

@JeremyTate41 If that was what "pro-life" actually meant, no one would disagree.

@DudespostingWs Actually this is exactly the picture the European mind has of America.

In fact the optimal point to aim at, if you're writing AI applications, is something that is just short of working. This requires great judgment; it would be so easy to lie to yourself about such a thing. But the prize for getting it right could be huge.

With AI though you can do this in multiple dimensions. You can make something that's too expensive, knowing it will get cheaper. But you can also make something that barely works, knowing it will start to work better.

This is the way we all wrote software back when Moore's Law was making CPUs twice as fast every 18 months. You only had to make it fast enough to be bearable, and then the next round of CPUs would do the rest.

@tttttttttf_ @ahistoryinart When you take the color out of paintings that weren't done from photographs, you see more inconsistencies in value. Paintings done from photographs are "too perfect" in that respect. https://t.co/93qGhxZYec

@pdavisjones The new name of Fort Bragg was official too.


@michael_nielsen Apparently they're hard to catch. I know this seems implausible.

@ahistoryinart One thing I've learned from these pictures is how much 19th century Danes liked symmetry inside their houses.

As I was sitting down with the second cup of tea:

@Rainmaker1973 (In case anyone is wondering, lligators are a kind of alligator native to Wales.)

@chrisalbon You're forgetting that schools do more than teach. They're also where people meet and learn from one another. If you merely brought together a lot of people in the top .5% and did nothing more than let them talk to one another, that would still be a valuable experience.

@rjvir Yes, I am.

People in the sciences thought they could ignore the fools over in the humanities and just focus on their research. But now the fools' ideology is colonizing the sciences.

@rjvir This is wrong. Reddit and Twitch were founded in Boston.

@nikitabier I think I might have been there. Is there a lot of wild ginger there?

"Volunteers from a neighborhood committee keeping watch on a Beijing street." They don't regard this as a chore; they enjoy it. The aggressively conventional-minded are always there, in every country. All they need is an excuse to go after the rest of us. https://t.co/QhdU1S5czh


@PeterSweden7 "Evans, of Poplar Court, Newcastle, pleaded guilty to violent disorder and assaulting an emergency worker."

Founder Mode: https://t.co/3hOnlKOJBi

Robert Reich wrote in The Guardian that "regulators around the world should threaten Musk with arrest if he doesn’t stop disseminating lies and hate on X." The US is part of the world. So are you saying the US government should threaten him with arrest, @rbreich?

@thegarrettscott Will Mike Johnson do this for anyone, or do you have some sort of special arrangement with him?

@Steve_Sailer How much of the increase on the FT graph does this cause account for?

What happened in America? https://t.co/KCk5oF0AzK

@anammostarac My Twitter account.

@mckaywrigley He already made it into the acknowledgements of "How to Do Great Work." He gave me several useful suggestions.

15 yo, who has an uncannily good memory, delights in finishing my sentences when I tell the boys things I've told them before. This used to mortify me, but now I've realized that his delight in scoring on me in this way means I can make him memorize anything I want.

12 yo: What are you doing?

Me: Writing a new essay. Want to read it? If you read it you get mentioned in the acknowledgements at the end.

12 yo: Under what name?

Me: I'd usually say "my younger son."

12 yo: I'll do it if I can be Nigel Pennypinch.


@growing_daniel It's not because they're woke. It's simply the power of a sitting president. The same thing would be happening if we had a senile Republican president who refused to drop out of the race.

@sarmadgulzar No. I was held back by the fact that no one in the suburbs of Pittsburgh, PA rode. No one that I knew anyway.

After a few years he even started to think that attendance was somehow his specialty. He told me that he felt cheated one term when he didn't miss a single day, and yet didn't win the attainment prize.

He'd been puzzled that he seemed to be distinguished for the regularity of his attendance, and a bit sad that he never won prizes for anything more meaningful, but he never questioned it. It just seemed one of those random things that schools do.

12 yo won the attainment prize this term. He didn't seem to care much. Turned out for the past five years he'd been confusing "attainment" and "attendance."

Met someone yesterday who told me he gave up riding after he fell and broke his neck. I realized I'd never heard or read this expression in the first person — it's always "he broke his neck" rather than "I broke my neck" — which tells you how dangerous it is.

Interesting. It wasn't so much that Labour won as that the Conservatives lost. https://t.co/hBGiHGuVFT

@pitdesi I don't believe there was some kind of organized plan to stop using the word "soccer" in the UK. In the UK that word was posh slang; most people always called it football.

@nejatian The reason in this case is not so much the phenomenon of the winners writing the history books as that it doesn't seem as big a deal to the British. For Americans the Revolution is the big story of the 18th century. For the British it's just an instance of failed colonial policy.

@amasad This was just as true in the past. Economic prosperity doesn't derive from the government.


Why Startup Hubs Work
Want to start a startup?
Get funded by
Y Combinator
.
October 2011
If you look at a list of US cities sorted by population, the number
of successful startups per capita varies by orders of magnitude.
Somehow it's as if most places were sprayed with startupicide.
I wondered about this for years.  I could see the average town was
like a roach motel for startup ambitions: smart, ambitious people
went in, but no startups came out.  But I was never able to figure
out exactly what happened inside the motel—exactly what was
killing all the potential startups.
[
1
]
A couple weeks ago I finally figured it out. I was framing the
question wrong.  The problem is not that most towns kill startups.
It's that death is the
default
for startups,
and most towns don't save them.  Instead of thinking of most places
as being sprayed with startupicide, it's more accurate to think of
startups as all being poisoned, and a few places being sprayed with
the antidote.
Startups in other places are just doing what startups naturally do:
fail.  The real question is, what's
saving
startups in places
like Silicon Valley?
[
2
]
Environment
I think there are two components to the antidote: being in a place
where startups are the cool thing to do, and chance meetings with
people who can help you.  And what drives them both is the number
of startup people around you.
The first component is particularly helpful in the first stage of
a startup's life, when you go from merely having an interest in
starting a company to actually doing it.  It's quite a leap to start
a startup.  It's an unusual thing to do. But in Silicon Valley it
seems normal.
[
3
]
In most places, if you start a startup, people treat you as if
you're unemployed.  People in the Valley aren't automatically
impressed with you just because you're starting a company, but they
pay attention.  Anyone who's been here any amount of time knows not
to default to skepticism, no matter how inexperienced you seem or
how unpromising your idea sounds at first, because they've all seen
inexperienced founders with unpromising sounding ideas who a few
years later were billionaires.
Having people around you care about what you're doing is an
extraordinarily
powerful
force.  Even the
most willful people are susceptible to it.  About a year after we
started Y Combinator I said something to a partner at a well known
VC firm that gave him the (mistaken) impression I was considering
starting another startup.  He responded so eagerly that for about
half a second I found myself considering doing it.
In most other cities, the prospect of starting a startup just doesn't
seem real.  In the Valley it's not only real but fashionable.  That
no doubt causes a lot of people to start startups who shouldn't.
But I think that's ok.  Few people are suited to running a startup,
and it's very hard to predict beforehand which are (as I know all
too well from being in the business of trying to predict beforehand),
so lots of people starting startups who shouldn't is probably the
optimal state of affairs.  As long as you're at a point in your
life when you can bear the risk of failure, the best way to find
out if you're suited to running a startup is to
try
it
.
Chance
The second component of the antidote is chance meetings with people
who can help you.  This force works in both phases: both in the
transition from the desire to start a startup to starting one, and
the transition from starting a company to succeeding.  The power
of chance meetings is more variable than people around you caring
about startups, which is like a sort of background radiation that
affects everyone equally, but at its strongest it is far stronger.
Chance meetings produce miracles to compensate for the disasters
that characteristically befall startups.  In the Valley, terrible
things happen to startups all the time, just like they do to startups
everywhere.  The reason startups are more likely to make it here
is that great things happen to them too.  In the Valley, lightning
has a sign bit.
For example, you start a site for college students and you decide
to move to the Valley for the summer to work on it.  And then on a
random suburban street in Palo Alto you happen to run into Sean
Parker, who understands the domain really well because he started
a similar startup himself, and also knows all the investors.  And
moreover has advanced views, for 2004, on founders retaining
control
of their companies.
You can't say precisely what the miracle will be, or even for sure
that one will happen.  The best one can say is: if you're in a
startup hub, unexpected good things will probably happen to you,
especially if you deserve them.
I bet this is true even for startups we fund.  Even with us working
to make things happen for them on purpose rather than by accident,
the frequency of helpful chance meetings in the Valley is so high
that it's still a significant increment on what we can deliver.
Chance meetings play a role like the role relaxation plays in having
ideas.  Most people have had the experience of working hard on some
problem, not being able to solve it, giving up and going to bed,
and then thinking of the answer in the shower in the morning.  What
makes the answer appear is letting your thoughts
drift
a bit—and thus drift off the wrong
path you'd been pursuing last night and onto the right one adjacent
to it.
Chance meetings let your acquaintance drift in the same way taking
a shower lets your thoughts drift. The critical thing in both cases
is that they drift just the right amount.  The meeting between Larry
Page and Sergey Brin was a good example.  They let their acquaintance
drift, but only a little; they were both meeting someone they had
a lot in common with.
For Larry Page the most important component of the antidote was
Sergey Brin, and vice versa.  The antidote is
people
.  It's not the
physical infrastructure of Silicon Valley that makes it work, or
the weather, or anything like that.  Those helped get it started,
but now that the reaction is self-sustaining what drives it is the
people.
Many observers have noticed that one of the most distinctive things
about startup hubs is the degree to which people help one another
out, with no expectation of getting anything in return.  I'm not
sure why this is so.  Perhaps it's because startups are less of a
zero sum game than most types of business; they are rarely killed
by competitors.  Or perhaps it's because so many startup founders
have backgrounds in the sciences, where collaboration is encouraged.
A large part of YC's function is to accelerate that process.  We're
a sort of Valley within the Valley, where the density of people
working on startups and their willingness to help one another are
both artificially amplified.
Numbers
Both components of the antidote—an environment that encourages
startups, and chance meetings with people who help you—are
driven by the same underlying cause: the number of startup people
around you.  To make a startup hub, you need a
lot
of people
interested in startups.
There are three reasons. The first, obviously, is that if you don't
have enough density, the chance meetings don't happen.
[
4
]
The second is that different startups need such different things, so
you need a lot of people to supply each startup with what they need
most.  Sean Parker was exactly what Facebook needed in 2004.  Another
startup might have needed a database guy, or someone with connections
in the movie business.
This is one of the reasons we fund such a large number of companies,
incidentally.  The bigger the community, the greater the chance it
will contain the person who has that one thing you need most.
The third reason you need a lot of people to make a startup hub is
that once you have enough people interested in the same problem,
they start to set the social norms.  And it is a particularly
valuable thing when the atmosphere around you encourages you to do
something that would otherwise seem too ambitious.  In most places
the atmosphere pulls you back toward the mean.
I flew into the Bay Area a few days ago.  I notice this every time
I fly over the Valley: somehow you can sense something is going on.
Obviously you can sense prosperity in how well kept a
place looks.  But there are different kinds of prosperity.  Silicon
Valley doesn't look like Boston, or New York, or LA, or DC.  I tried
asking myself what word I'd use to describe the feeling the Valley
radiated, and the word that came to mind was optimism.
Notes
[
1
]
I'm not saying it's impossible to succeed in a city with few
other startups, just harder.  If you're sufficiently good at
generating your own morale, you can survive without external
encouragement.  Wufoo was based in Tampa and they succeeded.  But
the Wufoos are exceptionally disciplined.
[
2
]
Incidentally, this phenomenon is not limited to startups.  Most
unusual ambitions fail, unless the person who has them manages to
find the right sort of community.
[
3
]
Starting a company is common, but starting a startup is rare.
I've talked about the distinction between the two elsewhere, but
essentially a startup is a new business designed for scale.  Most
new businesses are service businesses and except in rare cases those
don't scale.
[
4
]
As I was writing this, I had a demonstration of the density of
startup people in the Valley.  Jessica and I bicycled to University
Ave in Palo Alto to have lunch at the fabulous Oren's Hummus.  As
we walked in, we met Charlie Cheever sitting near the door.  Selina
Tobaccowala stopped to say hello on her way out.  Then Josh Wilson
came in to pick up a take out order.  After lunch we went to get
frozen yogurt.  On the way we met Rajat Suri.  When we got to the
yogurt place, we found Dave Shen there, and as we walked out we ran
into Yuri Sagalov.  We walked with him for a block or so and we ran
into Muzzammil Zaveri, and then a block later we met Aydin Senkut.
This is everyday life in Palo Alto.  I wasn't trying to meet people;
I was just having lunch.  And I'm sure for every startup founder
or investor I saw that I knew, there were 5 more I didn't.  If Ron
Conway had been with us he would have met 30 people he knew.
Thanks
to Sam Altman, Paul Buchheit, Jessica Livingston, and
Harj Taggar for reading drafts of this.


The List of N Things
September 2009
I bet you the current issue of
Cosmopolitan
has an article
whose title begins with a number. "7 Things He Won't Tell You about
Sex," or something like that.  Some popular magazines
feature articles of this type on the cover of every
issue.  That can't be happening by accident.  Editors must know
they attract readers.
Why do readers like the list of n things so much?   Mainly because
it's easier to read than a regular article.
[
1
]
Structurally, the list of n things is a degenerate case of essay.
An essay can go anywhere the writer wants.  In a list of n things
the writer agrees to constrain himself to a collection of points
of roughly equal importance, and he tells the reader explicitly
what they are.
Some of the work of reading an article is understanding its
structure—figuring out what in high school we'd have called
its "outline." Not explicitly, of course, but someone who really
understands an article probably has something in his brain afterward
that corresponds to such an outline.  In a list of n things, this
work is done for you.  Its structure is an exoskeleton.
As well as being explicit, the structure is guaranteed to be of the
simplest possible type: a few main points with few to no subordinate
ones, and no particular connection between them.
Because the main points are unconnected, the list of n things is
random access.  There's no thread of reasoning you have to follow.  You could
read the list in any order.  And because the points are independent
of one another, they work like watertight compartments in an
unsinkable ship.  If you get bored with, or can't understand, or
don't agree with one point, you don't have to give up on the article.
You can just abandon that one and skip to the next.  A list of n
things is parallel and therefore fault tolerant.
There are times when this format is what a writer wants.  One, obviously,
is when what you have to say actually is a list of n
things.  I once wrote an essay about the
mistakes that kill startups
, and a few people made fun of me
for writing something whose title began with a number.  But in that
case I really was trying to make a complete catalog of a number of
independent things.  In fact, one of the questions I was trying to
answer was how many there were.
There are other less legitimate reasons for using this format.  For
example, I use it when I get close to a deadline.  If I have to
give a talk and I haven't started it a few days beforehand, I'll
sometimes play it safe and make the talk a list of n things.
The list of n things is easier for writers as well as readers.  When
you're writing a real essay, there's always a chance you'll hit a
dead end.  A real essay is a train of thought, and some trains of
thought just peter out.  That's an alarming possibility when you
have to give a talk in a few days.  What if you run out of ideas?
The compartmentalized structure of the list of n things protects
the writer from his own stupidity in much the same way it protects
the reader.  If you run out of ideas on one point, no problem: it
won't kill the essay.  You can take out the whole point if you need
to, and the essay will still survive.
Writing a list of n things is so relaxing.  You think of n/2 of
them in the first 5 minutes.  So bang, there's the structure, and
you just have to fill it in.  As you think of more points, you just
add them to the end.  Maybe you take out or rearrange or combine a
few, but at every stage you have a valid (though initially low-res)
list of n things.  It's like the sort of programming where you write
a version 1 very quickly and then gradually modify it, but at every
point have working code—or the style of painting where you begin
with a complete but very blurry sketch done in an hour, then spend
a week cranking up the resolution.
Because the list of n things is easier for writers too, it's not
always a damning sign when readers prefer it.  It's not necessarily
evidence readers are lazy; it could also mean they don't have
much confidence in the writer.  The list of n things is in that
respect the cheeseburger of essay forms.  If you're eating at a
restaurant you suspect is bad, your best bet is to order the
cheeseburger.  Even a bad cook can make a decent cheeseburger.  And
there are pretty strict conventions about what a cheeseburger should
look like.  You can assume the cook isn't going to try something
weird and artistic.  The list of n things similarly limits the
damage that can be done by a bad writer.  You know it's going to
be about whatever the title says, and the format prevents the writer
from indulging in any flights of fancy.
Because the list of n things is the easiest essay form, it should
be a good one for beginning writers.  And in fact it is what most
beginning writers are taught.  The classic 5 paragraph essay is
really a list of n things for n = 3.  But the students writing them
don't realize they're using the same structure as the articles they
read in
Cosmopolitan
. They're not allowed to include the numbers,
and they're expected to spackle over the gaps with gratuitous
transitions ("Furthermore...") and cap the thing at either end with
introductory and concluding paragraphs so it will look superficially
like a real essay.
[
2
]
It seems a fine plan to start students off with the list of n things.
It's the easiest form.  But if we're going to do that, why not do
it openly?  Let them write lists of n things like the pros, with
numbers and no transitions or "conclusion."
There is one case where the list of n things is a dishonest format:
when you use it to attract attention by falsely claiming the list
is an exhaustive one.  I.e. if you write an article that purports
to be about
the
7 secrets of success.  That kind of title is the
same sort of reflexive challenge as a whodunit. You have to at least
look at the article to check whether they're the same 7 you'd list.
Are you overlooking one of the secrets of success?  Better check.
It's fine to put "The" before the number if you really believe
you've made an exhaustive list.  But evidence suggests most things
with titles like this are linkbait.
The greatest weakness of the list of n things is that there's so
little room for new thought.  The main point of essay writing, when
done right, is the new ideas you have while doing it.  A real essay,
as the name implies, is
dynamic
: you don't know what you're going
to write when you start.  It will be about whatever you discover
in the course of writing it.
This can only happen in a very limited way in a list of n things.
You make the title first, and that's what it's going to be about.
You can't have more new ideas in the writing than will fit in the
watertight compartments you set up initially.  And your brain seems
to know this: because you don't have room for new ideas, you don't
have them.
Another advantage of admitting to beginning writers that the 5
paragraph essay is really a list of n things is that we can warn
them about this.  It only lets you experience the defining
characteristic of essay writing on a small scale: in thoughts of a
sentence or two.  And it's particularly dangerous that the 5 paragraph
essay buries the list of n things within something that looks like
a more sophisticated type of essay.  If you don't know you're using
this form, you don't know you need to escape it.
Notes
[
1
]
Articles of this type are also startlingly popular on Delicious,
but I think that's because
delicious/popular
is driven by bookmarking,
not because Delicious users are stupid.  Delicious users are
collectors, and a list of n things seems particularly collectible
because it's a collection itself.
[
2
]
Most "word problems" in school math textbooks are similarly
misleading.  They look superficially like the application of math
to real problems, but they're not.  So if anything they reinforce
the impression that math is merely a complicated but pointless
collection of stuff to be memorized.
Russian Translation


6,631,372
March 2006, rev August 2009
A couple days ago I found to my surprise that I'd been granted a
patent
.
It issued in 2003, but no one told me.  I wouldn't know about it
now except that a few months ago, while visiting Yahoo, I happened
to run into a Big Cheese I knew from working there in the late
nineties.  He brought up something called Revenue Loop, which Viaweb
had been working on when they bought us.
The idea is basically that you sort search results not in order of
textual "relevance" (as search engines did then) nor in order of
how much advertisers bid (as Overture did) but in order of the bid
times the number of transactions.  Ordinarily you'd do this for
shopping searches, though in fact one of the features of our scheme
is that it automatically detects which searches are shopping searches.
If you just order the results in order of bids, you can make the
search results useless, because the first results could be dominated
by lame sites that had bid the most.  But if you order results by
bid multiplied by transactions, far from selling out, you're getting
a
better
measure of relevance.  What could be a better sign that
someone was satisfied with a search result than going to the site
and buying something?
And, of course, this algorithm automatically maximizes the revenue
of the search engine.
Everyone is focused on this type of approach now, but few were in
1998.  In 1998 it was all about selling banner ads.  We didn't know
that, so we were pretty excited when we figured out what seemed to
us the optimal way of doing shopping searches.
When Yahoo was thinking of buying us, we had a meeting with Jerry
Yang in New York.  For him, I now realize, this was supposed to be
one of those meetings when you check out a company you've pretty
much decided to buy, just to make sure they're ok guys.  We weren't
expected to do more than chat and seem smart and reasonable.  He
must have been dismayed when I jumped up to the whiteboard and
launched into a presentation of our exciting new technology.
I was just as dismayed when he didn't seem to care at all about it.
At the time I thought, "boy, is this guy poker-faced.  We present
to him what has to be the optimal way of sorting product search
results, and he's not even curious."  I didn't realize till much later
why he didn't care.  In 1998, advertisers were overpaying enormously
for ads on web sites.
In 1998, if advertisers paid the maximum that traffic was worth to
them, Yahoo's revenues would have
decreased.
Things are different now, of course.  Now this sort of thing is all
the rage.  So when I ran into the Yahoo exec I knew from the old
days in the Yahoo cafeteria a few months ago, the first thing he
remembered was not (fortunately) all the fights I had with him, but
Revenue Loop.
"Well," I said, "I think we actually applied for a patent on it.
I'm not sure what happened to the application after I left."
"Really?  That would be an important patent."
So someone investigated, and sure enough, that patent application
had continued in the pipeline for several years after, and finally
issued in 2003.
The main thing that struck me on reading it, actually, is that
lawyers at some point messed up my nice clear writing.  Some clever
person with a spell checker reduced one section to Zen-like incomprehensibility:
Also, common spelling errors will tend to get fixed. For example,
  if users searching for "compact disc player" end up spending
  considerable money at sites offering compact disc players, then
  those pages will have a higher relevance for that search phrase,
  even though the phrase "compact disc player" is not present on
  those pages.
(That "compat disc player" wasn't a typo, guys.)
For the fine prose of the original, see the provisional application
of February 1998, back when we were still Viaweb and couldn't afford
to pay lawyers to turn every "a lot of" into "considerable."


I support jail terms for protestors who deface famous art to get attention. They're cheating. They're taking advantage of a feature of high-trust societies — art displayed publicly in museums — and violating that trust.

12 yo about to have a character-building experience. https://t.co/Xz1NhT1vj5

@ahistoryinart Reminds me of Stephen Bone. In fact not just similar but indistinguishable.

@parkerconrad The frightening but I fear inevitable implication is that a lot of VCs don't even rise to amoral.

@parkerconrad And the weird thing is that it's not even in the VCs' interest. All the returns are in the big outcomes. The small ones are rounding error. So even if you were completely amoral, why would you bother nickle-and-diming the founders?

@jjmaxwell4 @parkerconrad @svangel @NFXGuild @Initialized For sure SV Angel and Initialized won't screw founders over.

@Noahpinion An encouraging metaphor; it implies most people are good.

This would seem incomprehensible, or dystopian fiction, to people 100 years ago, when kids this age regularly walked around large cities on their own.

@BenjaminDEKR Ugh. Twitter.

I explained why it can't be right in the tweet you replied to.

I'm sorry, but I'm going to have to block you now. You are wasting my time.

@BenjaminDEKR That can't be right. People leave jobs at every organization. If toxic work culture were the main reason people left jobs, it would imply that every organization had a toxic work culture.


I told 15 yo that I also started lifting weights at around that age, but that in my case one of my motivations was to be able to defend myself.

(For context here's a photo from my high school yearbook. The guy in front later played defensive end for the Raiders.) https://t.co/VgZFHkvAe5

Time for some Streisand Effect.

@ryangrim @ytirawi Wow, it's true. I clicked on the follow button and when I reloaded the page, it hadn't worked.

@elonmusk If I write a new essay and tweet a link to it, that's "lazy linking," but if I tweet that I've written a new essay and then put the link in a reply, that's somehow better?

The deprioritization of tweets with links in them is Twitter's biggest flaw. It bothers me more than all the new right-wing trolls. Trolls I'm used to, but what draws me to Twitter is to find out what's going on, and you can't do that without links.

@PalmerLuckey There might not be many such people, but those that exist are disproportionately likely to be rich and powerful.

@PalmerLuckey Maybe you should fund a new news outlet. Now that the old papers of record have given up that job, there's room for a new one to take their place. And there are *some* people who genuinely want to know what's going on, rather than merely having their biases confirmed.

@aidangch If there was ever a startup that was just a feature, it was Dropbox. As many people pointed out at the time, all it did was back up files to the cloud. But now that it's an $8.7 billion feature, people don't say that so much.

@aidangch The reason it seems there are "more and more" startups that are just features is due to another, separate problem in your head: recency bias. You forget how lame companies like Dropbox and Airbnb and Coinbase seemed when they first launched.

@aidangch Sure, I can explain. The problem is in your own head. Startup ideas often look like features at first. Especially if the founders launch as soon as they have a quantum of utility, as I recommend.


@Spotlight_Abby It sounds like there is some confusion about how AI is involved here. When you talk about "the use of AI to moderate the experiences" of people, are you talking about people's writing being judged by AI or something like that?

@pashadelics YC doesn't get any financial gain. The money it gives to the nonprofits is just a donation. They succeed in about the same proportion as the for-profit startups.

@LewisCTech @mkaleel When I started writing essays, "blog" still had its original sense of "web log," meaning an entry written on a specific day, which would have been inaccurate.

Clearly all this attention is going to your head, so goodbye...

@joshnekoff No, it wasn't. It's not complex writing that's an AI tell, just specific words, one of which in the current version of ChatGPT is "delve".

@tamuramble Some are unable to, others don't want to, apparently in roughly equal proportions.

@ZubaydaRingim @buffys No, that is pretty much is all I said.

@barisukafx @buffys Decades of practice writing essays.

@_romeopeter @buffys No, but it is a slightly bogus one. There are almost always better alternatives.

@Rohan_Choksi @buffys I know. But they're greatly overestimating the differences. When x is a more basic word than y in one country, it almost always is in another too.

@macroflaneur @buffys Not always. Sometimes maximum precision would be overkill.


@josephjojoe Don't hire stupid people or create bureaucracy. It really is as simple as that.

@ClaudiuDP Yes, I know.

@Austen @patrickc @DanielleFong "School closures explained only 11 percent of the variation."

It's good for any organization to be on the way up rather than the way down, of course, but it's especially important for YC to feel like a startup, because it has to transmit startup culture to the founders. You can't be a bureaucracy teaching people about startups.

When YC partners describe what YC feels like now, under Garry, they often use words like "early." It feels like the early days of a startup, when the people are smart and there isn't too much bureaucracy. There is no way I'd rather hear it described.

@pitdesi It's not a nice house.

I screenshotted this because it will seem so funny a couple years from now. Though to be fair it seems pretty funny already. https://t.co/qMwifbOLuz

@TimCodesStuff @robinhanson It's hard to say for sure but maybe 75.

@patrickc I think book publishing, specifically, is not the business it used to be. Similarly for journalism.

@robinhanson Because very political people consider their position on the spectrum to be optimal. So for a San Francisco SJW who's at the 99th percentile of leftness, anything below about the 95th is far right. Ditto at the other end.


The High-Res Society
December 2008
For nearly all of history the success of a society was proportionate
to its ability to assemble large and disciplined organizations.
Those who bet on economies of scale generally won, which meant the
largest organizations were the most successful ones.
Things have already changed so much that this is hard for us to
believe, but till just a few decades ago the largest organizations
tended to be the most progressive.  An ambitious kid graduating
from college in 1960 wanted to work in the huge, gleaming offices
of Ford, or General Electric, or NASA.  Small meant small-time.
Small in 1960 didn't mean a cool little startup.  It meant uncle
Sid's shoe store.
When I grew up in the 1970s, the idea of the "corporate ladder" was
still very much alive.  The standard plan was to try to get into a
good college, from which one would be drafted into some organization
and then rise to positions of gradually increasing responsibility.
The more ambitious merely hoped to climb the same ladder faster.
[
1
]
But in the late twentieth century something changed.  It turned out
that economies of scale were not the only force at work.  Particularly
in technology, the increase in speed one could get from smaller
groups started to trump the advantages of size.
The future turned out to be different from the one we were expecting
in 1970.  The domed cities and flying cars we expected have failed
to materialize.  But fortunately so have the jumpsuits with badges
indicating our specialty and rank.  Instead of being dominated by
a few, giant tree-structured organizations, it's now looking like
the economy of the future will be a fluid network of smaller,
independent units.
It's not so much that large organizations stopped working.  There's
no evidence that famously successful organizations like the Roman
army or the British East India Company were any less afflicted by
protocol and politics than organizations of the same size today.
But they were competing against opponents who couldn't change the
rules on the fly by discovering new technology.  Now it turns out
the rule "large and disciplined organizations win" needs to have a
qualification appended: "at games that change slowly." No one knew
till change reached a sufficient speed.
Large organizations
will
start to do worse now, though,
because for the first time in history they're no longer getting the
best people.  An ambitious kid graduating from college now doesn't
want to work for a big company.  They want to work for the hot
startup that's rapidly growing into one.  If they're really ambitious,
they want to start it.
[
2
]
This doesn't mean big companies will disappear.  To say that
startups will succeed implies that big companies will exist, because
startups that succeed either become big companies or are acquired
by them.
[
3
]
But large organizations will probably never again
play the leading role they did up till the last quarter of the
twentieth century.
It's kind of surprising that a trend that lasted so long would ever
run out.  How often does it happen that a rule works for thousands
of years, then switches polarity?
The millennia-long run of bigger-is-better left us with a lot of
traditions
that are now obsolete,
but extremely deeply rooted.
Which means the ambitious can now do arbitrage on them.  It will
be very valuable to understand precisely which ideas to keep and
which can now be discarded.
The place to look is where the spread of smallness began: in the
world of startups.
There have always been occasional cases, particularly in the US,
of ambitious people who grew the ladder under them instead of
climbing it.  But till recently this was an anomalous route that
tended to be followed only by outsiders.  It was no coincidence
that the great industrialists of the nineteenth century had so
little formal education.  As huge as their companies eventually
became, they were all essentially mechanics and shopkeepers at
first.  That was a social step no one with a college education would
take if they could avoid it.  Till the rise of technology startups,
and in particular, Internet startups, it was very unusual for
educated people to start their own businesses.
The eight men who left Shockley Semiconductor to found Fairchild
Semiconductor, the original Silicon Valley startup, weren't even
trying to start a company at first.  They were just looking for a
company willing to hire them as a group.  Then one of their parents
introduced them to a small investment bank that offered to find
funding for them to start their own, so they did.  But starting a
company was an alien idea to them; it was something they backed
into.
[
4
]
Now I would guess that practically every Stanford or Berkeley
undergrad who knows how to program has at least considered the idea
of starting a startup.  East Coast universities are not far behind,
and British universities only a little behind them.  This pattern
suggests that attitudes at Stanford and Berkeley are not an anomaly,
but a leading indicator.  This is the way the world is going.
Of course, Internet startups are still only a fraction of the world's
economy.  Could a trend based on them be that powerful?
I think so.  There's no reason to suppose there's any limit to the
amount of work that could be done in this area.  Like science,
wealth seems to expand fractally.  Steam power was a sliver of the
British economy when Watt started working on it.  But his work led
to more work till that sliver had expanded into something bigger
than the whole economy of which it had initially been a part.
The same thing could happen with the Internet.  If Internet startups
offer the best opportunity for ambitious people, then a lot of
ambitious people will start them, and this bit of the economy will
balloon in the usual fractal way.
Even if Internet-related applications only become a tenth of the
world's economy, this component will set the tone for the rest.
The most dynamic part of the economy always does, in everything
from salaries to standards of dress.  Not just because of its
prestige, but because the principles underlying the most dynamic
part of the economy tend to be ones that work.
For the future, the trend to bet on seems to be networks of small,
autonomous groups whose performance is measured individually.  And
the societies that win will be the ones with the least impedance.
As with the original industrial revolution, some societies are going
to be better at this than others.  Within a generation of its birth
in England, the Industrial Revolution had spread to continental
Europe and North America.  But it didn't spread everywhere.  This
new way of doing things could only take root in places that were
prepared for it.  It could only spread to places that already had
a vigorous middle class.
There is a similar social component to the transformation that began
in Silicon Valley in the 1960s.  Two new kinds of techniques were
developed there: techniques for building integrated circuits, and
techniques for building a new type of company designed to grow fast
by creating new technology.  The techniques for building integrated
circuits spread rapidly to other countries.  But the techniques for
building startups didn't.  Fifty years later, startups are ubiquitous
in Silicon Valley and common in a handful of other US cities, but
they're still an anomaly in most of the world.
Part of the reason—possibly the main reason—that startups
have not spread as broadly as the Industrial Revolution did is their
social disruptiveness.  Though it brought many social changes, the
Industrial Revolution was not fighting the principle that bigger
is better.  Quite the opposite: the two dovetailed beautifully.
The new industrial companies adapted the customs of existing large
organizations like the military and the civil service, and the
resulting hybrid worked well.  "Captains of industry" issued orders
to "armies of workers," and everyone knew what they were supposed
to do.
Startups seem to go more against the grain, socially.  It's hard
for them to flourish in societies that value hierarchy and stability,
just as it was hard for industrialization to flourish in societies
ruled by people who stole at will from the merchant class.  But
there were already a handful of countries past that stage when the
Industrial Revolution happened.   There do not seem to be that many
ready this time.
Notes
[
1
]
One of the bizarre consequences of this model was that the usual
way to make more money was to become a manager.  This is one of the
things startups fix.
[
2
]
There are a lot of reasons American car companies have been
doing so much worse than Japanese car companies, but at least one
of them is a cause for optimism: American graduates have more
options.
[
3
]
It's possible that companies will one day be able to grow big
in revenues without growing big in people, but we are not very far
along that trend yet.
[
4
]
Lecuyer, Christophe,
Making Silicon Valley
, MIT Press, 2006.
Thanks
to Trevor Blackwell, Paul Buchheit, Jessica Livingston,
and Robert Morris for reading drafts of this.