Lies We Tell Kids
May 2008
Adults lie constantly to kids.  I'm not saying we should stop, but
I think we should at least examine which lies we tell and why.
There may also be a benefit to us.  We were all lied to as kids,
and some of the lies we were told still affect us.  So by studying
the ways adults lie to kids, we may be able to clear our heads of
lies we were told.
I'm using the word "lie" in a very general sense: not just overt
falsehoods, but also all the more subtle ways we mislead kids.
Though "lie" has negative connotations, I don't mean to suggest we
should never do this—just that we should pay attention when
we do.
[
1
]
One of the most remarkable things about the way we lie to kids is
how broad the conspiracy is.  All adults know what their culture
lies to kids about: they're the questions you answer "Ask
your parents."  If a kid asked who won the World Series in 1982
or what the atomic weight of carbon was, you could just tell him.
But if a kid asks you "Is there a God?" or "What's a prostitute?"
you'll probably say "Ask your parents."
Since we all agree, kids see few cracks in the view of the world
presented to them.  The biggest disagreements are between parents
and schools, but even those are small. Schools are careful what
they say about controversial topics, and if they do contradict what
parents want their kids to believe, parents either pressure the
school into keeping
quiet
or move their kids to a new school.
The conspiracy is so thorough that most kids who discover it do so
only by discovering internal contradictions in what they're told.
It can be traumatic for the ones who wake up during the operation.
Here's what happened to Einstein:
Through the reading of popular scientific books I soon reached
  the conviction that much in the stories of the Bible could not
  be true.  The consequence was a positively fanatic freethinking
  coupled with the impression that youth is intentionally being
  deceived by the state through lies: it was a crushing impression.
[
2
]
I remember that feeling.  By 15 I was convinced the world was corrupt
from end to end.  That's why movies like
The Matrix
have such
resonance.  Every kid grows up in a fake world.  In a way it would
be easier if the forces behind it were as clearly differentiated
as a bunch of evil machines, and one could make a clean break just by
taking a pill.
Protection
If you ask adults why they lie to kids, the most common reason they
give is to protect them.  And kids do need protecting.  The environment
you want to create for a newborn child will be quite unlike the
streets of a big city.
That seems so obvious it seems wrong to call it a lie.  It's certainly
not a bad lie to tell, to give a baby the impression the world is
quiet and warm and safe.  But this harmless type of lie can turn
sour if left unexamined.
Imagine if you tried to keep someone in as protected an environment
as a newborn till age 18.  To mislead someone so grossly about the
world would seem not protection but abuse.  That's an extreme
example, of course; when parents do that sort of thing it becomes
national news.  But you see the same problem on a smaller scale in
the malaise teenagers feel in suburbia.
The main purpose of suburbia is to provide a protected environment
for children to grow up in.  And it seems great for 10 year olds.
I liked living in suburbia when I was 10.  I didn't notice how
sterile it was.  My whole world was no bigger than a few friends'
houses I bicycled to and some woods I ran around in.  On a log scale
I was midway between crib and globe.  A suburban street was just
the right size.  But as I grew older, suburbia started to feel
suffocatingly fake.
Life can be pretty good at 10 or 20, but it's often frustrating at
15.  This is too big a problem to solve here, but certainly one
reason life sucks at 15 is that kids are trapped in a world designed
for 10 year olds.
What do parents hope to protect their children from by raising them
in suburbia?  A friend who moved out of Manhattan said merely that
her 3 year old daughter "saw too much."  Off the top of my head,
that might include: people who are high or drunk, poverty, madness,
gruesome medical conditions, sexual behavior of various degrees of
oddness, and violent anger.
I think it's the anger that would worry me most if I had a 3 year
old.  I was 29 when I moved to New York and I was surprised even
then.  I wouldn't want a 3 year old to see some of the disputes I
saw.  It would be too frightening.  A lot of the things adults
conceal from smaller children, they conceal because they'd be
frightening, not because they want to conceal the existence of such
things.  Misleading the child is just a byproduct.
This seems one of the most justifiable types of lying adults do to
kids.  But because the lies are indirect we don't keep a very strict
accounting of them.  Parents know they've concealed the facts about
sex, and many at some point sit their kids down and explain more.
But few tell their kids about the differences between the real world
and the cocoon they grew up in.  Combine this with the confidence
parents try to instill in their kids, and every year you get a new
crop of 18 year olds who think they know how to run the world.
Don't all 18 year olds think they know how to run the world?  Actually
this seems to be a recent innovation, no more than about 100 years old.
In preindustrial times teenage kids were junior members of the adult
world and comparatively well aware of their shortcomings.  They
could see they weren't as strong or skillful as the village smith.
In past times people lied to kids about some things more than we
do now, but the lies implicit in an artificial, protected environment
are a recent invention.  Like a lot of new inventions, the rich got
this first.  Children of kings and great magnates were the first
to grow up out of touch with the world.  Suburbia means half the
population can live like kings in that respect.
Sex (and Drugs)
I'd have different worries about raising teenage kids in New York.
I'd worry less about what they'd see, and more about what they'd
do.  I went to college with a lot of kids who grew up in Manhattan,
and as a rule they seemed pretty jaded.  They seemed to have lost
their virginity at an average of about 14 and by college had tried
more drugs than I'd even heard of.
The reasons parents don't want their teenage kids having sex are
complex.  There are some obvious dangers: pregnancy and sexually
transmitted diseases.  But those aren't the only reasons parents
don't want their kids having sex.  The average parents of a 14 year
old girl would hate the idea of her having sex even if there were
zero risk of pregnancy or sexually transmitted diseases.
Kids can probably sense they aren't being told the whole story.
After all, pregnancy and sexually transmitted diseases are just as
much a problem for adults, and they have sex.
What really bothers parents about their teenage kids having sex?
Their dislike of the idea is so visceral it's probably inborn.  But
if it's inborn it should be universal, and there are plenty of
societies where parents don't mind if their teenage kids have
sex—indeed, where it's normal for 14 year olds to become
mothers.  So what's going on?  There does seem to be a universal
taboo against sex with prepubescent children.  One can imagine
evolutionary reasons for that.  And I think this is the main reason
parents in industrialized societies dislike teenage kids having
sex.  They still think of them as children, even though biologically
they're not, so the taboo against child sex still has force.
One thing adults conceal about sex they also conceal about drugs:
that it can cause great pleasure.  That's what makes sex and drugs
so dangerous. The desire for them can cloud one's judgement—which
is especially frightening when the judgement being clouded is the
already wretched judgement of a teenage kid.
Here parents' desires conflict.  Older societies told kids they had
bad judgement, but modern parents want their children to be confident.
This may well be a better plan than the old one of putting them in
their place, but it has the side effect that after having implicitly
lied to kids about how good their judgement is, we then have to lie
again about all the things they might get into trouble with if they
believed us.
If parents told their kids the truth about sex and drugs, it would
be: the reason you should avoid these things is that you have lousy
judgement.  People with twice your experience still get burned by
them.  But this may be one of those cases where the truth wouldn't
be convincing, because one of the symptoms of bad judgement is
believing you have good judgement.  When you're too weak to lift
something, you can tell, but when you're making a decision impetuously,
you're all the more sure of it.
Innocence
Another reason parents don't want their kids having sex is that
they want to keep them innocent.  Adults have a certain model of
how kids are supposed to behave, and it's different from what they
expect of other adults.
One of the most obvious differences is the words kids are allowed
to use.  Most parents use words when talking to other adults that
they wouldn't want their kids using.  They try to hide even the
existence of these words for as long as they can.  And this is
another of those conspiracies everyone participates in: everyone
knows you're not supposed to swear in front of kids.
I've never heard more different explanations for anything parents
tell kids than why they shouldn't swear.  Every parent I know forbids
their children to swear, and yet no two of them have the same
justification.  It's clear most start with not wanting kids to
swear, then make up the reason afterward.
So my theory about what's going on is that the
function
of
swearwords is to mark the speaker as an adult.  There's no difference
in the meaning of "shit" and "poopoo."  So why should one be ok for
kids to say and one forbidden?  The only explanation is: by definition.
[
3
]
Why does it bother adults so much when kids do things reserved for
adults?   The idea of a foul-mouthed, cynical 10 year old leaning
against a lamppost with a cigarette hanging out of the corner of
his mouth is very disconcerting.  But why?
One reason we want kids to be innocent is that we're programmed to
like certain kinds of helplessness.  I've several times heard mothers
say they deliberately refrained from correcting their young children's
mispronunciations because they were so cute.  And if you think about
it, cuteness is helplessness. Toys and cartoon characters meant to
be cute always have clueless expressions and stubby, ineffectual
limbs.
It's not surprising we'd have an inborn desire to love and protect
helpless creatures, considering human offspring are so helpless for
so long.  Without the helplessness that makes kids cute, they'd be
very annoying.  They'd merely seem like incompetent adults.  But
there's more to it than that.  The reason our hypothetical jaded
10 year old bothers me so much is not just that he'd be annoying,
but that he'd have cut off his prospects for growth so early.  To
be jaded you have to think you know how the world works, and any
theory a 10 year old had about that would probably be a pretty
narrow one.
Innocence is also open-mindedness.  We want kids to be innocent so
they can continue to learn.  Paradoxical as it sounds, there are
some kinds of knowledge that get in the way of other kinds of
knowledge.  If you're going to learn that the world is a brutal
place full of people trying to take advantage of one another, you're
better off learning it last.  Otherwise you won't bother learning
much more.
Very smart adults often seem unusually innocent, and I don't think
this is a coincidence.  I think they've deliberately avoided learning
about certain things.  Certainly I do.  I used to think I wanted
to know everything.  Now I know I don't.
Death
After sex, death is the topic adults lie most conspicuously about
to kids.  Sex I believe they conceal because of deep taboos.  But
why do we conceal death from kids?   Probably because small children
are particularly horrified by it.  They want to feel safe, and death
is the ultimate threat.
One of the most spectacular lies our parents told us was about the
death of our first cat.  Over the years, as we asked for more
details, they were compelled to invent more, so the story grew quite
elaborate.  The cat had died at the vet's office.  Of what?  Of the
anaesthesia itself.  Why was the cat at the vet's office?  To be
fixed.  And why had such a routine operation killed it?  It wasn't
the vet's fault; the cat had a congenitally weak heart; the anaesthesia
was too much for it; but there was no way anyone could have
known this in advance.  It was not till we were in our twenties
that the truth came out: my sister, then about three, had accidentally
stepped on the cat and broken its back.
They didn't feel the need to tell us the cat was now happily in cat
heaven.  My parents never claimed that people or animals who died
had "gone to a better place," or that we'd meet them again.  It
didn't seem to harm us.
My grandmother told us an edited version of the death of my
grandfather.  She said they'd been sitting reading one day, and
when she said something to him, he didn't answer.  He seemed to be
asleep, but when she tried to rouse him, she couldn't.  "He was
gone." Having a heart attack sounded like falling asleep.  Later I
learned it hadn't been so neat, and the heart attack had taken most
of a day to kill him.
Along with such outright lies, there must have been a lot of changing
the subject when death came up.  I can't remember that, of course,
but I can infer it from the fact that I didn't really grasp I was
going to die till I was about 19.  How could I have missed something
so obvious for so long?  Now that I've seen parents managing the
subject, I can see how: questions about death are gently but firmly
turned aside.
On this topic, especially, they're met half-way by kids.  Kids often
want to be lied to.  They want to believe they're living in a
comfortable, safe world as much as their parents want them to believe
it.
[
4
]
Identity
Some parents feel a strong adherence to an ethnic or religious group
and want their kids to feel it too.  This usually requires two
different kinds of lying: the first is to tell the child that he
or she is an X, and the second is whatever specific lies Xes
differentiate themselves by believing.
[
5
]
Telling a child they have a particular ethnic or religious identity
is one of the stickiest things you can tell them.  Almost anything
else you tell a kid, they can change their mind about later when
they start to think for themselves.  But if you tell a kid they're
a member of a certain group, that seems nearly impossible to shake.
This despite the fact that it can be one of the most premeditated
lies parents tell.  When parents are of different religions, they'll
often agree between themselves that their children will be "raised
as Xes."  And it works. The kids obligingly grow up considering
themselves as Xes, despite the fact that if their parents had chosen
the other way, they'd have grown up considering themselves as Ys.
One reason this works so well is the second kind of lie involved.
The truth is common property.  You can't distinguish your group by
doing things that are rational, and believing things that are true.
If you want to set yourself apart from other people, you have to
do things that are arbitrary, and believe things that are false.
And after having spent their whole lives doing things that are arbitrary
and believing things that are false, and being regarded as odd by
"outsiders" on that account, the cognitive dissonance pushing
children to regard themselves as Xes must be enormous.  If they
aren't an X, why are they attached to all these arbitrary beliefs
and customs?  If they aren't an X, why do all the non-Xes call them
one?
This form of lie is not without its uses.  You can use it to carry
a payload of beneficial beliefs, and they will also become part of
the child's identity.  You can tell the child that in addition to
never wearing the color yellow, believing the world was created by
a giant rabbit, and always snapping their fingers before eating
fish, Xes are also particularly honest and industrious.  Then X
children will grow up feeling it's part of their identity to be
honest and industrious.
This probably accounts for a lot of the spread of modern religions,
and explains why their doctrines are a combination of the useful
and the bizarre.  The bizarre half is what makes the religion stick,
and the useful half is the payload.
[
6
]
Authority
One of the least excusable reasons adults lie to kids is to maintain
power over them.  Sometimes these lies are truly sinister, like a
child molester telling his victims they'll get in trouble if they
tell anyone what happened to them.  Others seem more innocent; it
depends how badly adults lie to maintain their power, and what they
use it for.
Most adults make some effort to conceal their flaws from children.
Usually their motives are mixed.  For example, a father who has an
affair generally conceals it from his children.  His motive is
partly that it would worry them, partly that this would introduce
the topic of sex, and partly (a larger part than he would admit)
that he doesn't want to tarnish himself in their eyes.
If you want to learn what lies are told to kids, read almost any
book written to teach them about "issues."
[
7
]
Peter Mayle wrote
one called
Why Are We Getting a Divorce?
It begins with the three
most important things to remember about divorce, one of which is:
You shouldn't put the blame on one parent, because divorce is
  never only one person's fault.
[
8
]
Really?  When a man runs off with his secretary, is it always partly
his wife's fault?  But I can see why Mayle might have said this.
Maybe it's more important for kids to respect their parents than
to know the truth about them.
But because adults conceal their flaws, and at the same time insist
on high standards of behavior for kids, a lot of kids grow up feeling
they fall hopelessly short.  They walk around feeling horribly evil
for having used a swearword, while in fact most of the adults around
them are doing much worse things.
This happens in intellectual as well as moral questions.  The more
confident people are, the more willing they seem to be to answer a
question "I don't know."  Less confident people feel they have to
have an answer or they'll look bad.  My parents were pretty good
about admitting when they didn't know things, but I must have been
told a lot of lies of this type by teachers, because I rarely heard
a teacher say "I don't know" till I got to college.  I remember
because it was so surprising to hear someone say that in front of
a class.
The first hint I had that teachers weren't omniscient came in sixth
grade, after my father contradicted something I'd learned in school.
When I protested that the teacher had said the opposite, my father
replied that the guy had no idea what he was talking about—that
he was just an elementary school teacher, after all.
Just
a teacher?  The phrase seemed almost grammatically ill-formed.
Didn't teachers know everything about the subjects they taught?
And if not, why were they the ones teaching us?
The sad fact is, US public school teachers don't generally understand
the stuff they're teaching very well.  There are some sterling
exceptions, but as a rule people planning to go into teaching rank
academically near the bottom of the college population.  So the
fact that I still thought at age 11 that teachers were infallible
shows what a job the system must have done on my brain.
School
What kids get taught in school is a complex mix of lies.  The most
excusable are those told to simplify ideas to make them easy to
learn.  The problem is, a lot of propaganda gets slipped into the
curriculum in the name of simplification.
Public school textbooks represent a compromise between what various
powerful groups want kids to be told.  The lies are rarely overt.
Usually they consist either of omissions or of over-emphasizing
certain topics at the expense of others.  The view of history we
got in elementary school was a crude hagiography, with at least one
representative of each powerful group.
The famous scientists I remember were Einstein, Marie Curie, and
George Washington Carver.   Einstein was a big deal because his
work led to the atom bomb.  Marie Curie was involved with X-rays.
But I was mystified about Carver.  He seemed to have done stuff
with peanuts.
It's obvious now that he was on the list because he was black (and
for that matter that Marie Curie was on it because she was a woman),
but as a kid I was confused for years about him.  I wonder if it
wouldn't have been better just to tell us the truth: that there
weren't any famous black scientists.  Ranking George Washington
Carver with Einstein misled us not only about science, but about
the obstacles blacks faced in his time.
As subjects got softer, the lies got more frequent.  By the time
you got to politics and recent history, what we were taught was
pretty much pure propaganda.  For example, we were taught to regard
political leaders as saints—especially the recently martyred
Kennedy and King.  It was astonishing to learn later that they'd
both been serial womanizers, and that Kennedy was a speed freak to
boot.  (By the time King's plagiarism emerged, I'd lost the ability
to be surprised by the misdeeds of famous people.)
I doubt you could teach kids recent history without teaching them
lies, because practically everyone who has anything to say about
it has some kind of spin to put on it.  Much recent history
consists
of spin.  It would probably be better just to teach them metafacts
like that.
Probably the biggest lie told in schools, though, is that the way
to succeed is through following "the rules."  In fact most such
rules are just hacks to manage large groups efficiently.
Peace
Of all the reasons we lie to kids, the most powerful is probably
the same mundane reason they lie to us.
Often when we lie to people it's not part of any conscious strategy,
but because they'd react violently to the truth.  Kids, almost by
definition, lack self-control.  They react violently to things—and
so they get lied to a lot.
[
9
]
A few Thanksgivings ago, a friend of mine found himself in a situation
that perfectly illustrates the complex motives we have when we lie
to kids.  As the roast turkey appeared on the table, his alarmingly
perceptive 5 year old son suddenly asked if the turkey had wanted
to die.  Foreseeing disaster, my friend and his wife rapidly
improvised: yes, the turkey had wanted to die, and in fact had lived
its whole life with the aim of being their Thanksgiving dinner.
And that (phew) was the end of that.
Whenever we lie to kids to protect them, we're usually also lying
to keep the peace.
One consequence of this sort of calming lie is that we grow up
thinking horrible things are normal.  It's hard for us to feel a
sense of urgency as adults over something we've literally been
trained not to worry about.  When I was about 10 I saw a documentary
on pollution that put me into a panic.  It seemed the planet was
being irretrievably ruined.  I went to my mother afterward to ask
if this was so.  I don't remember what she said, but she made me
feel better, so I stopped worrying about it.
That was probably the best way to handle a frightened 10 year old.
But we should understand the price.  This sort of lie is one of the
main reasons bad things persist: we're all trained to ignore them.
Detox
A sprinter in a race almost immediately enters a state called "oxygen
debt."  His body switches to an emergency source of energy that's
faster than regular aerobic respiration.  But this process builds
up waste products that ultimately require extra oxygen to break
down, so at the end of the race he has to stop and pant for a while
to recover.
We arrive at adulthood with a kind of truth debt.  We were told a
lot of lies to get us (and our parents) through our childhood.  Some
may have been necessary.  Some probably weren't.  But we all arrive
at adulthood with heads full of lies.
There's never a point where the adults sit you down and explain all
the lies they told you.  They've forgotten most of them.  So if
you're going to clear these lies out of your head, you're going to
have to do it yourself.
Few do.  Most people go through life with bits of packing material
adhering to their minds and never know it.  You probably never can
completely undo the effects of lies you were told as a kid, but
it's worth trying.  I've found that whenever I've been able to undo
a lie I was told, a lot of other things fell into place.
Fortunately, once you arrive at adulthood you get a valuable new
resource you can use to figure out what lies you were told.  You're
now one of the liars.  You get to watch behind the scenes as adults
spin the world for the next generation of kids.
The first step in clearing your head is to realize how far you are
from a neutral observer.  When I left high school I was, I thought,
a complete skeptic.  I'd realized high school was crap.  I thought
I was ready to question everything I knew.  But among the many other
things I was ignorant of was how much debris there already was in
my head.  It's not enough to consider your mind a blank slate.  You
have to consciously erase it.
Notes
[
1
]
One reason I stuck with such a brutally simple word is that
the lies we tell kids are probably not quite as harmless as we
think.  If you look at what adults told children in the past, it's
shocking how much they lied to them.  Like us, they did it with the
best intentions.  So if we think we're as open as one could reasonably
be with children, we're probably fooling ourselves.  Odds are people
in 100 years will be as shocked at some of the lies we tell as we
are at some of the lies people told 100 years ago.
I can't predict which these will be, and I don't want to write an
essay that will seem dumb in 100 years.  So instead of using special
euphemisms for lies that seem excusable according to present fashions,
I'm just going to call all our lies lies.
(I have omitted one type: lies told to play games with kids'
credulity.  These range from "make-believe," which is not really a
lie because it's told with a wink, to the frightening lies told by
older siblings.   There's not much to say about these: I wouldn't
want the first type to go away, and wouldn't expect the second type
to.)
[
2
]
Calaprice, Alice (ed.),
The Quotable Einstein
, Princeton
University Press, 1996.
[
3
]
If you ask parents why kids shouldn't swear, the less educated
ones usually reply with some question-begging answer like "it's
inappropriate," while the more educated ones come up with elaborate
rationalizations.  In fact the less educated parents seem closer
to the truth.
[
4
]
As a friend with small children pointed out, it's easy for small
children to consider themselves immortal, because time seems to
pass so slowly for them.  To a 3 year old, a day feels like a month
might to an adult.  So 80 years sounds to him like 2400 years would
to us.
[
5
]
I realize I'm going to get endless grief for classifying religion
as a type of lie.  Usually people skirt that issue with some
equivocation implying that lies believed for a sufficiently long
time by sufficiently large numbers of people are immune to the usual
standards for truth.  But because I can't predict which lies future
generations will consider inexcusable, I can't safely omit any type
we tell.  Yes, it seems unlikely that religion will be out of fashion
in 100 years, but no more unlikely than it would have seemed to
someone in 1880 that schoolchildren in 1980 would be taught that
masturbation was perfectly normal and not to feel guilty about it.
[
6
]
Unfortunately the payload can consist of bad customs as well
as good ones.  For example, there are certain qualities that some
groups in America consider "acting white."  In fact most of them
could as accurately be called "acting Japanese."  There's nothing
specifically white about such customs. They're common to all cultures
with long traditions of living in cities.  So it is probably a
losing bet for a group to consider behaving the opposite way as
part of its identity.
[
7
]
In this context, "issues" basically means "things we're going
to lie to them about."  That's why there's a special name for these
topics.
[
8
]
Mayle, Peter,
Why Are We Getting a Divorce?
, Harmony, 1988.
[
9
]
The ironic thing is, this is also the main reason kids lie to
adults.  If you freak out when people tell you alarming things,
they won't tell you them.  Teenagers don't tell their parents what
happened that night they were supposed to be staying at a friend's
house for the same reason parents don't tell 5 year olds the truth
about the Thanksgiving turkey.  They'd freak if they knew.
Thanks
to Sam Altman, Marc Andreessen, Trevor Blackwell,
Patrick Collison, Jessica Livingston, Jackie McDonough, Robert
Morris, and David Sloo for reading drafts of this.  And since there
are some controversial ideas here, I should add that none of them
agreed with everything in it.
German Translation
French Translation
Russian Translation


Interesting comparison by someone who did YC in 2011, 2016, and 2024.

@NecktieSalvage If you believe our society is declining you're probably going to hate a lot of things.

@growindiedev It wasn't just loyalty but principle.

I'm glad Sam didn't throw Peter Thiel under the bus when the mob came for him.

@arturtavr Apparently not, but you could cite the tweet.

Incidentally here's a related point: https://t.co/qak61MNH7A

@cremieuxrecueil @willrobinson22 Wow, really? That is surprising. What's the best source on this?

@mysticmarkets The salaries people pay programmers prove that's not true.

@sharinganlou By default start here: https://t.co/pJEKmmY3Tg

@agent_A98 This is what drives the best programmers.

@RobertGrootjen If you have a friend who uses a particular language, use that one.


@kane And why do they call fines fees? Does that mean speeding is legal, but they charge for it?

@Noahpinion Why don't banks loan?

@AwayBerk Why not just wait to tweet about it till after reading it?

@hamandcheese It's true. I thought he was nuts, but it seems he wasn't far off.

@sprout94 @NAChristakis The confidence is more a product of going there than the way students get in. The one thing every Harvard undergrad learns is how to sound like they know what they're talking about.

@sprout94 @NAChristakis The admissions process at American universities is much more arbitrary than at British ones. You get in by pleasing the admissions officers. Intelligence is one thing that pleases them, but not the only thing.

@Arp_it1 @OpenAI That 40% is an old number. Apparently it's higher now.

@Noahpinion And indeed, this question doesn't so much focus the discussion as end it. But I am usually happy to.

@Noahpinion It took me a long time to realize that most of the people who express an opinion about some essay I've written haven't actually read it, but instead are responding to what they imagine I must have said. I was very confused before I understood this.

@ashthaker1 It's not cricket season yet.


@AlecStapp Who initiated these bills and why?

@jeremycorbyn Their *use* causes mass death and destruction, but empirically their existence seems to cause peace.

@artist_romney Those dates have to be wrong.

@lacker @dflieb I tried asking ChatGPT. At first it tried to tell me it was clumsy, which I already knew. But when I told it that I wanted to know whether it was strictly ungrammatical — whether there was no valid English parsing — it said there wasn't.

@atajik It represents an odd number.

Explained to 13 yo why n makes a better variable name than o. https://t.co/GQUA3iUeFO

@levie Initially we thought AI would help humans work, now we think humans will manage AI, next we'll realize AI will manage AI.

@stephsmithio I do stealth yoga while waiting in lines. E.g. tree. People are cool with it as long as it looks unintentional.

@PeterDiamandis It would be stupid to try. Who fights fires if you get the flu?

@reidhoffman Depends how old you are. Young people should take more risk, because they can afford the most and yet (having just emerged from childhood) are very risk-averse. But I wouldn't recommend more risk for 80 year olds.


@txnftxnf @mikekarnj Actually western Colorado has always seemed especially promising to me. I was just recommending it to my son last week.

The rich love quiet because they're trying to work.

@ravisyal I doubt I ever will be.

@matthew_d_green When they do something that doesn't seem to make sense, the reason could be something that they know that we don't. Do you think that's the case here? And if so what could it be?

@Venice_Wes I'm not 10 anymore. I'd rather have this world. https://t.co/tpboL8fXLm

@scienceisstrat1 For graduate school I presume?

When I was 10, this was where I thought I'd be now.

@mikekarnj I agree with this list, but if they really were the new status symbols a lot more people would be moving to the English countryside.

@RealTimeWWII Imagine how much spit they must have eaten.

@ktlmld @cremieuxrecueil I think the correct word here would be "innumerate" rather than "human." There's a high correlation between the two, but they're not identical.


@JHWeissmann Maybe he's not as imprudent as he seems. Tempting fate means inviting something that would be bad for you. But think what happens to JD Vance if Trump is forced out of office.

@Rasika012 Wouldn't that be wonderful?

@RepSwalwell Man, what is it with Florida?

@traestephens Boston is the intellectual capital. Not just of the US but of the world.

@Austen If you did this in July you must mean it.

@sharmutal More like old Defenders. Bentleys are a London thing.

@stocksgoupndown 15 year old boys aren't thinking about bastions of high trust society. They're thinking about whacking a ball two hundred yards.

@brian_armstrong @base What did you do to make it 10x faster?

@QuasLacrimas "hard to be sure" = confident? Are you this dishonest or do you just have trouble reading?

Teenage boys in the UK have spontaneously taken up golf. Not because of a marketing campaign or a charismatic young golfer. They just like it.


@AgentBigfoot It was good 20 years ago. A bit of a self-caricature, but interesting.

@_elzubeir Very, very unlikely.

@Viktor_shinski I always did.

@jason_kerr @nickcammarata Sounds like someone has been reading dictionaries. But words can be tacky, just as other things can be.

@skdh Does the Elements of Style need an update? I don't mean this as a rhetorical question. It would be interesting if there were something new to say on the subjects it talks about. But IIRC it was pretty timeless.

"Our reverence for the truth might be a distraction that's getting in the way of finding common ground and getting things done."

—  NPR CEO Katherine Maher

@ograuc No. If they get wronger, you'll get increasingly different even if you do nothing.

@MarcHoag @JanelCubbage That would sound awkward.

@noornet @korbencopy @Levi7hart Again, not precise enough.

@MandelsonC I am glad to say that it does not.


Better Bayesian Filtering
January 2003
(This article was given as a talk at the 2003 Spam Conference.
It describes the work I've done to improve the performance of
the algorithm described in
A Plan for Spam
,
and what I plan to do in the future.)
The first discovery I'd like to present here is an algorithm for
lazy evaluation of research papers.  Just
write whatever you want and don't cite any previous work, and
indignant readers will send you references to all the papers you
should have cited.   I discovered this algorithm
after ``A Plan for Spam'' [1] was on Slashdot.
Spam filtering is a subset of text classification,
which is a well established field, but the first papers about
Bayesian
spam filtering per se seem to have been two
given at the same conference in 1998,
one by Pantel and Lin [2],
and another by a group from
Microsoft Research [3].
When I heard about this work I was a bit surprised.  If
people had been onto Bayesian filtering four years ago,
why wasn't everyone using it?
When I read the papers I found out why.  Pantel and Lin's filter was the
more effective of the two, but it
only caught 92% of spam, with 1.16% false positives.
When I tried writing a Bayesian spam filter,
it caught 99.5% of spam with less than .03% false
positives [4].
It's always alarming when two people
trying the same experiment get widely divergent results.
It's especially alarming here because those two sets of numbers
might yield opposite conclusions.
Different users have different requirements, but I think for
many people a filtering rate of 92% with 1.16% false positives means
that filtering is not an acceptable solution, whereas
99.5% with less than .03% false positives means that it is.
So why did we get such different numbers?
I haven't tried to reproduce Pantel and Lin's results, but
from reading the paper I see five things that probably account
for the difference.
One is simply that they trained their filter on very little
data: 160 spam and 466 nonspam mails.
Filter performance should still be climbing with data
sets that small.  So their numbers may not even be an accurate
measure of the performance of their algorithm, let alone of
Bayesian spam filtering in general.
But I think the most important difference is probably
that they ignored message headers.  To anyone who has worked
on spam filters, this will seem a perverse decision.
And yet in the very first filters I tried writing, I ignored the
headers too.  Why?  Because I wanted to keep the problem neat.
I didn't know much about mail headers then, and they seemed to me
full of random stuff.  There is a lesson here for filter
writers: don't ignore data.  You'd think this lesson would
be too obvious to mention, but I've had to learn it several times.
Third, Pantel and Lin stemmed the tokens, meaning they reduced e.g. both
``mailing'' and ``mailed'' to the root ``mail''.   They may
have felt they were forced to do this by the small size
of their corpus, but if so this is a kind of premature
optimization.
Fourth, they calculated probabilities differently.
They used all the tokens, whereas I only
use the 15 most significant.  If you use all the tokens
you'll tend to miss longer spams, the type where someone tells you their life
story up to the point where they got rich from some multilevel
marketing scheme.  And such an algorithm
would be easy for spammers to spoof: just add a big
chunk of random text to counterbalance the spam terms.
Finally, they didn't bias against false positives.
I think
any spam filtering algorithm ought to have a convenient
knob you can twist to decrease the
false positive rate at the expense of the filtering rate.
I do this by counting the occurrences
of tokens in the nonspam corpus double.
I don't think it's a good idea to treat spam filtering as
a straight text classification problem.  You can use
text classification techniques, but solutions can and should
reflect the fact that the text is email, and spam
in particular.  Email is not just text; it has structure.
Spam filtering is not just classification, because
false positives are so much worse than false negatives
that you should treat them as a different kind of error.
And the source of error is not just random variation, but
a live human spammer working actively to defeat your filter.
Tokens
Another project I heard about
after the Slashdot article was Bill Yerazunis'
CRM114
[5].
This is the counterexample to the design principle I
just mentioned.  It's a straight text classifier,
but such a stunningly effective one that it manages to filter
spam almost perfectly without even knowing that's
what it's doing.
Once I understood how CRM114 worked, it seemed
inevitable that I would eventually have to move from filtering based
on single words to an approach like this.  But first, I thought,
I'll see how far I can get with single words.  And the answer is,
surprisingly far.
Mostly I've been working on smarter tokenization.  On
current spam, I've been able to achieve filtering rates that
approach CRM114's.  These techniques are mostly orthogonal to Bill's;
an optimal solution might incorporate both.
``A Plan for Spam'' uses a very simple
definition of a token.  Letters, digits, dashes, apostrophes,
and dollar signs are constituent characters, and everything
else is a token separator.  I also ignored case.
Now I have a more complicated definition of a token:
Case is preserved.
Exclamation points are constituent characters.
Periods and commas are constituents if they occur
 between two digits.  This lets me get ip addresses
 and prices intact.
A price range like $20-25 yields two tokens,
 $20 and $25.
Tokens that occur within the
 To, From, Subject, and Return-Path lines, or within urls,
 get marked accordingly.  E.g. ``foo'' in the Subject line
 becomes ``Subject*foo''.  (The asterisk could
 be any character you don't allow as a constituent.)
Such measures increase the filter's vocabulary, which
makes it more discriminating.  For example, in the current
filter, ``free'' in the Subject line
has a spam probability of 98%, whereas the same token
in the body has a spam probability of only 65%.
Here are some of the current probabilities [6]:
Subject*FREE      0.9999
free!!            0.9999
To*free           0.9998
Subject*free      0.9782
free!             0.9199
Free              0.9198
Url*free          0.9091
FREE              0.8747
From*free         0.7636
free              0.6546
In the Plan for Spam filter, all these tokens would have had the
same probability, .7602.  That filter recognized about 23,000
tokens.  The current one recognizes about 187,000.
The disadvantage of having a larger universe of tokens
is that there is more
chance of misses.
Spreading your corpus out over more tokens
has the same effect as making it smaller.
If you consider exclamation points as
constituents, for example, then you could end up
not having a spam probability for free with seven exclamation
points, even though you know that free with just two
exclamation points has a probability of 99.99%.
One solution to this is what I call degeneration.  If you
can't find an exact match for a token,
treat it as if it were a less specific
version.  I consider terminal exclamation
points, uppercase letters, and occurring in one of the
five marked contexts as making a token more specific.
For example, if I don't find a probability for
``Subject*free!'', I look for probabilities for
``Subject*free'', ``free!'', and ``free'', and take whichever one
is farthest from .5.
Here are the alternatives [7]
considered if the filter sees ``FREE!!!'' in the
Subject line and doesn't have a probability for it.
Subject*Free!!!
Subject*free!!!
Subject*FREE!
Subject*Free!
Subject*free!
Subject*FREE
Subject*Free
Subject*free
FREE!!!
Free!!!
free!!!
FREE!
Free!
free!
FREE
Free
free
If you do this, be sure to consider versions with initial
caps as well as all uppercase and all lowercase.  Spams
tend to have more sentences in imperative mood, and in
those the first word is a verb.  So verbs with initial caps
have higher spam probabilities than they would in all
lowercase.  In my filter, the spam probability of ``Act''
is 98% and for ``act'' only 62%.
If you increase your filter's vocabulary, you can end up
counting the same word multiple times, according to your old
definition of ``same''.
Logically, they're not the
same token anymore.  But if this still bothers you, let
me add from experience that the words you seem to be
counting multiple times tend to be exactly the ones you'd
want to.
Another effect of a larger vocabulary is that when you
look at an incoming mail you find more interesting tokens,
meaning those with probabilities far from .5.  I use the
15 most interesting to decide if mail is spam.
But you can run into a problem when you use a fixed number
like this.  If you find a lot of maximally interesting tokens,
the result can end up being decided by whatever random factor
determines the ordering of equally interesting tokens.
One way to deal with this is to treat some
as more interesting than others.
For example, the
token ``dalco'' occurs 3 times in my spam corpus and never
in my legitimate corpus.  The token ``Url*optmails''
(meaning ``optmails'' within a url) occurs 1223 times.
And yet, as I used to calculate probabilities for tokens,
both would have the same spam probability, the threshold of .99.
That doesn't feel right.  There are theoretical
arguments for giving these two tokens substantially different
probabilities (Pantel and Lin do), but I haven't tried that yet.
It does seem at least that if we find more than 15 tokens
that only occur in one corpus or the other, we ought to
give priority to the ones that occur a lot.  So now
there are two threshold values.  For tokens that occur only
in the spam corpus, the probability is .9999 if they
occur more than 10 times and .9998 otherwise.  Ditto
at the other end of the scale for tokens found
only in the legitimate corpus.
I may later scale token probabilities substantially,
but this tiny amount of scaling at least ensures that
tokens get sorted the right way.
Another possibility would be to consider not
just 15 tokens, but all the tokens over a certain
threshold of interestingness.  Steven Hauser does this
in his statistical spam filter [8].
If you use a threshold, make it very high, or
spammers could spoof you by packing messages with
more innocent words.
Finally, what should one do
about html?  I've tried the whole spectrum of options, from
ignoring it to parsing it all.  Ignoring html is a bad idea,
because it's full of useful spam signs.  But if you parse
it all, your filter might degenerate into a mere html
recognizer.  The most effective approach
seems to be the middle course, to notice some tokens but not
others.  I look at a, img, and font tags, and ignore the
rest.  Links and images you should certainly look at, because
they contain urls.
I could probably be smarter about dealing with html, but I
don't think it's worth putting a lot of time into this.
Spams full of html are easy to filter.  The smarter
spammers already avoid it.  So
performance in the future should not depend much on how
you deal with html.
Performance
Between December 10 2002 and January 10 2003 I got about
1750 spams.
Of these, 4 got through.  That's a filtering
rate of about 99.75%.
Two of the four spams I missed got through because they
happened to use words that occur often in my legitimate
email.
The third was one of those that exploit
an insecure cgi script to send mail to third parties.
They're hard to filter based just
on the content because the headers are innocent and
they're careful about the words they use.  Even so I can
usually catch them.  This one squeaked by with a
probability of .88, just under the threshold of .9.
Of course, looking at multiple token sequences
would catch it easily.  ``Below is the result of
your feedback form'' is an instant giveaway.
The fourth spam was what I call
a spam-of-the-future, because this is what I expect spam to
evolve into: some completely neutral
text followed by a url.  In this case it was was from
someone saying they had finally finished their homepage
and would I go look at it.  (The page was of course an
ad for a porn site.)
If the spammers are careful about the headers and use a
fresh url, there is nothing in spam-of-the-future for filters
to notice.  We can of course counter by sending a
crawler to look at the page.  But that might not be necessary.
The response rate for spam-of-the-future must
be low, or everyone would be doing it.
If it's low enough,
it
won't pay
for spammers to send it, and we won't
have to work too hard on filtering it.
Now for the really shocking news: during that same one-month
period I got
three
false positives.
In a way it's
a relief to get some false positives.  When I wrote ``A Plan
for Spam'' I hadn't had any, and I didn't know what they'd
be like.  Now that I've had a few, I'm relieved to find
they're not as bad as I feared.
False positives yielded by statistical
filters turn out to be mails that sound a lot like spam, and
these tend to be the ones you would least mind missing [9].
Two of the false positives were newsletters
from companies I've bought things from.  I never
asked to receive them, so arguably they
were spams, but I count them as false positives because
I hadn't been deleting them as spams before.  The reason
the filters caught them was that both companies in
January switched to commercial email senders
instead of sending the mails from their own servers,
and both the headers and the bodies became much spammier.
The third false positive was a bad one, though.  It was
from someone in Egypt and written in all uppercase.  This was
a direct result of making tokens case sensitive; the Plan
for Spam filter wouldn't have caught it.
It's hard to say what the overall false positive rate is,
because we're up in the noise, statistically.
Anyone who has worked on filters (at least, effective filters) will
be aware of this problem.
With some emails it's
hard to say whether they're spam or not, and these are
the ones you end up looking at when you get filters
really tight.  For example, so far the filter has
caught two emails that were sent to my address because
of a typo, and one sent to me in the belief that I was
someone else.  Arguably, these are neither my spam
nor my nonspam mail.
Another false positive was from a vice president at Virtumundo.
I wrote to them pretending to be a customer,
and since the reply came back through Virtumundo's
mail servers it had the most incriminating
headers imaginable.  Arguably this isn't a real false
positive either, but a sort of Heisenberg uncertainty
effect: I only got it because I was writing about spam
filtering.
Not counting these, I've had a total of five false positives
so far, out of about 7740 legitimate emails, a rate of .06%.
The other two were a notice that something I bought
was back-ordered, and a party reminder from Evite.
I don't think this number can be trusted, partly
because the sample is so small, and partly because
I think I can fix the filter not to catch
some of these.
False positives seem to me a different kind of error from
false negatives.
Filtering rate is a measure of performance.  False
positives I consider more like bugs.  I approach improving the
filtering rate as optimization, and decreasing false
positives as debugging.
So these five false positives are my bug list.  For example,
the mail from Egypt got nailed because the uppercase text
made it look to the filter like a Nigerian spam.
This really is kind of a bug.  As with
html, the email being all uppercase is really conceptually
one
feature, not one for each word.  I need to handle case in a
more sophisticated way.
So what to make of this .06%?  Not much, I think.  You could
treat it as an upper bound, bearing in mind the small sample size.
But at this stage it is more a measure of the bugs
in my implementation than some intrinsic false positive rate
of Bayesian filtering.
Future
What next?  Filtering is an optimization problem,
and the key to optimization is profiling.  Don't
try to guess where your code is slow, because you'll
guess wrong.
Look
at where your code is slow,
and fix that.  In filtering, this translates to:
look at the spams you miss, and figure out what you
could have done to catch them.
For example, spammers are now working aggressively to
evade filters, and one of the things they're doing is
breaking up and misspelling words to prevent filters from
recognizing them.  But working on this is not my first
priority, because I still have no trouble catching these
spams [10].
There are two kinds of spams I currently do
have trouble with.
One is the type that pretends to be an email from
a woman inviting you to go chat with her or see her profile on a dating
site.  These get through because they're the one type of
sales pitch you can make without using sales talk.  They use
the same vocabulary as ordinary email.
The other kind of spams I have trouble filtering are those
from companies in e.g. Bulgaria offering contract programming
services.   These get through because I'm a programmer too, and
the spams are full of the same words as my real mail.
I'll probably focus on the personal ad type first.  I think if
I look closer I'll be able to find statistical differences
between these and my real mail.  The style of writing is
certainly different, though it may take multiword filtering
to catch that.
Also, I notice they tend to repeat the url,
and someone including a url in a legitimate mail wouldn't do that [11].
The outsourcing type are going to be hard to catch.  Even if
you sent a crawler to the site, you wouldn't find a smoking
statistical gun.
Maybe the only answer is a central list of
domains advertised in spams [12].  But there can't be that
many of this type of mail.  If the only
spams left were unsolicited offers of contract programming
services from Bulgaria, we could all probably move on to
working on something else.
Will statistical filtering actually get us to that point?
I don't know.  Right now, for me personally, spam is
not a problem.  But spammers haven't yet made a serious
effort to spoof statistical filters.  What will happen when they do?
I'm not optimistic about filters that work at the
network level [13].
When there is a static obstacle worth getting past, spammers
are pretty efficient at getting past it.  There
is already a company called Assurance Systems that will
run your mail through Spamassassin and tell you whether
it will get filtered out.
Network-level filters won't be completely useless.
They may be enough to kill all the "opt-in"
spam, meaning spam from companies like Virtumundo and
Equalamail who claim that they're really running opt-in lists.
You can filter those based just on the headers, no
matter what they say in the body.  But anyone willing to
falsify headers or use open relays, presumably including
most porn spammers, should be able to get some message past
network-level filters if they want to.  (By no means the
message they'd like to send though, which is something.)
The kind of filters I'm optimistic about are ones that
calculate probabilities based on each individual user's mail.
These can be much more effective, not only in
avoiding false positives, but in filtering too: for example,
finding the recipient's email address base-64 encoded anywhere in
a message is a very good spam indicator.
But the real advantage of individual filters is that they'll all be
different.  If everyone's filters have different probabilities,
it will make the spammers' optimization loop, what programmers
would call their edit-compile-test cycle, appallingly slow.
Instead of just tweaking a spam till it gets through a copy of
some filter they have on their desktop, they'll have to do a
test mailing for each tweak.  It would be like programming in
a language without an interactive toplevel,
and I wouldn't wish that
on anyone.
Notes
[1]
Paul Graham.  ``A Plan for Spam.'' August 2002.
http://paulgraham.com/spam.html.
Probabilities in this algorithm are
calculated using a degenerate case of Bayes' Rule.  There are
two simplifying assumptions: that the probabilities
of features (i.e. words) are independent, and that we know
nothing about the prior probability of an email being
spam.
The first assumption is widespread in text classification.
Algorithms that use it are called ``naive Bayesian.''
The second assumption I made because the proportion of spam in
my incoming mail fluctuated so much from day to day (indeed,
from hour to hour) that the overall prior ratio seemed
worthless as a predictor.  If you assume that P(spam) and
P(nonspam) are both .5, they cancel out and you can
remove them from the formula.
If you were doing Bayesian filtering in a situation where
the ratio of spam to nonspam was consistently very high or
(especially) very low, you could probably improve filter
performance by incorporating prior probabilities.  To do
this right you'd have to track ratios by time of day, because
spam and legitimate mail volume both have distinct daily
patterns.
[2]
Patrick Pantel and Dekang Lin. ``SpamCop-- A Spam
Classification & Organization Program.''  Proceedings of AAAI-98
Workshop on Learning for Text Categorization.
[3]
Mehran Sahami, Susan Dumais, David Heckerman and Eric Horvitz.
``A Bayesian Approach to Filtering Junk E-Mail.'' Proceedings of AAAI-98
Workshop on Learning for Text Categorization.
[4] At the time I had zero false positives out of about 4,000
legitimate emails.  If the next legitimate email was
a false positive, this would give us .03%.  These false positive
rates are untrustworthy, as I explain later. I quote
a number here only to emphasize that whatever the false positive rate
is, it is less than 1.16%.
[5] Bill Yerazunis. ``Sparse Binary Polynomial Hash Message
Filtering and The CRM114 Discriminator.''  Proceedings of 2003
Spam Conference.
[6] In ``A Plan for Spam'' I used thresholds of .99 and .01.
It seems justifiable to use thresholds proportionate to the
size of the corpora.  Since I now have on the order of 10,000 of each
type of mail, I use .9999 and .0001.
[7] There is a flaw here I should probably fix.  Currently,
when ``Subject*foo'' degenerates to just ``foo'', what that means is
you're getting the stats for occurrences of ``foo'' in
the body or header lines other than those I mark.
What I should do is keep track of statistics for ``foo''
overall as well as specific versions, and degenerate from
``Subject*foo'' not to ``foo'' but to ``Anywhere*foo''.  Ditto for
case: I should degenerate from uppercase to any-case, not
lowercase.
It would probably be a win to do this with prices
too, e.g. to degenerate from ``$129.99'' to ``$--9.99'', ``$--.99'',
and ``$--''.
You could also degenerate from words to their stems,
but this would probably only improve filtering rates early on
when you had small corpora.
[8] Steven Hauser.  ``Statistical Spam Filter Works for Me.''
http://www.sofbot.com.
[9] False positives are not all equal, and we should remember
this when comparing techniques for stopping spam.
Whereas many of the false positives caused by filters
will be near-spams that you wouldn't mind missing,
false positives caused by blacklists, for example, will be just
mail from people who chose the wrong ISP.  In both
cases you catch mail that's near spam, but for blacklists nearness
is physical, and for filters it's textual.
[10] If spammers get good enough at obscuring tokens
for this to be a problem, we can respond by simply removing
whitespace, periods, commas, etc.  and using a dictionary to
pick the words out of the resulting sequence.
And of course finding words this way that weren't visible in
the original text would in itself be evidence of spam.
Picking out the words won't be trivial.  It will require
more than just reconstructing word boundaries; spammers
both add (``xHot nPorn cSite'') and omit (``P#rn'') letters.
Vision research may be useful here, since human vision is
the limit that such tricks will approach.
[11]
In general, spams are more repetitive than regular email.
They want to pound that message home.  I currently don't
allow duplicates in the top 15 tokens, because
you could get a false positive if the sender happens to use
some bad word multiple times. (In my current filter, ``dick'' has
a spam probabilty of .9999, but it's also a name.)
It seems we should at least notice duplication though,
so I may try allowing up to two of each token, as Brian Burton does in
SpamProbe.
[12]  This is what approaches like Brightmail's will
degenerate into once spammers are pushed into using mad-lib
techniques to generate everything else in the message.
[13]
It's sometimes argued that we should be working on filtering
at the network level, because it is more efficient.  What people
usually mean when they say this is: we currently filter at the
network level, and we don't want to start over from scratch.
But you can't dictate the problem to fit your solution.
Historically, scarce-resource arguments have been the losing
side in debates about software design.
People only tend to use them to justify choices
(inaction in particular) made for other reasons.
Thanks
to Sarah Harlin, Trevor Blackwell, and
Dan Giffin for reading drafts of this paper, and to Dan again
for most of the infrastructure that this filter runs on.
Related:
A Plan for Spam
Plan for Spam FAQ
2003 Spam Conference Proceedings
Japanese Translation
Chinese Translation
Test of These Suggestions


@CuriosMuseum That is surely AI.

@levie You seem to be a case of this:

https://t.co/WSrIC97iHQ

When Matthew Green says something like this, it's probably not an exaggeration. And that is alarming.

@mattyglesias I considered it a compliment when these people described such qualities as white. It seemed they were giving us more credit than we deserved.

@noah_foltz This is the price from a wine merchant.

@LifeThruBook Luxury brands would be very slack if they allowed this not to be true in their market.

I asked a friend who's been in the wine business for decades at what price wine quality is within a percent or two of the asymptote. He said £250 ($300). Beyond that you're just paying for brand.

@levie Do you think you will say later that the main importance of what you did at Box up till now was the position it got you into when AI arrived? Or is that putting it too strongly?

12 yo had a fever and was sleepwalking last night. As we led him back to bed, he explained his theory that The Simpsons is derived from Frasier. He remembers nothing of this.

Prediction: Gene editing will be huge. This is just Internet stocks in 2000.


I don't have to tell you what happened to these three boys. You already know. How awful is that? https://t.co/KsyAzW4nq3

There are few jobs where you could have more effect on the world, or work with better people, than this one.

Communism means the government owns everything. Authoritarianism means it controls everything. And ownership and control converge. So while it's well known that communism yields dictatorship, it's also true that dictatorship yields communism.

@Truthlovingguy Writing essays and writing fiction are two very different things.

@samtayyari Boy is that not true. I often spend weeks writing essays, and when I look back at the first versions they usually seem terrible by comparison.

Where the two versions differ, I almost always prefer the original. I had three weeks to work on it instead of two hours. But maybe a couple of those commas can go.

One consistent difference between the two versions is that the second flows slightly faster. When they differ by a comma, it's always because it's missing from the second. Presumably because I'm reproducing thoughts I've already had instead of working them out as I'm writing.

Only once did I actually omit an idea in the reproduction, when "replaying the thoughts in them and asking" became merely "each time asking".

For example, "You'd think that such an arbitrary constraint would" became "You'd expect such an arbitrary constraint to". In another place "you always end up sacrificing" became "it's always at the expense of". There was one whole paragraph that got turned around like this.

Instead I seem to have it (or at least parts of it) stored as ideas rather than words. Mostly these come out the same, right down to the punctuation, but occasionally they turn into different words.


@MasonJettericks What's the difference between per se impossible and impossible?

@wylfcen Or British photo editors have different tastes.

When you see photos of Western politicians in Japanese newspapers, they look Japanese, because the photo editors choose the pictures that look right to them.

@furnuwees It is a kind of roofing material usually made from yak felt.

@adilmania @nikitabier Startups!

@Spin0zasGod Exactly.

@Austen It certainly wouldn't have worked as well. I wrote a whole essay about this.

https://t.co/wRohkkMV6p

@nikitabier Meh, tax optimization isn't where the money is anyway.

One of the things I love about Jessica is that she has no tolerance for affectation of any kind. She sees through ideologues, social climbers, and pseudo-intellectuals like an airport baggage scanner.

Don't do this. No matter how much you dislike Trump, don't do this. Half the people in America voted for Trump. If you think voting for Trump makes people evil, then you have to believe half the people in America are evil, and that's just deluded. https://t.co/lYp15zNLeE

@loose_parts Half the people in America voted for Trump. Do you think half the people in America are evil? Come on.

Frankly what you did to him is far worse.


@waitbutwhy It's simply because Steve is gone. Imagine the fire he'd have lit under Apple's AI people if he was still there.

@skdh This is why we always tell startups that if they buy merch with their name on it, buy the highest quality things. Otherwise it will sit on a shelf.

@AyushKhatr21401 Make something you yourself want. Then you know it's something people want.

@naval I'd say it's laziness. I know lots of proud people who've prospered despite it, but I can't think of many people who've prospered despite being lazy.

@BorsaEsoterica @curtis_yarvin Thank you for that valuable... uh sequence of intellectual-sounding phrases, Grok.

@RetiefPeter Why should the police intercede to protect someone from a mob?

@BackTheBunny @curtis_yarvin Right on schedule: https://t.co/R7yg6gRHBE

@BackTheBunny @curtis_yarvin If by "steamrolled" you mean I'm going to get a tidal wave of replies from fake accounts or people indistinguishable from them, I'm used to that.

@broudsov @elonmusk He didn't go to school in America.

@curtis_yarvin The executive branch nominates the judges. Did you not go to school in America?


@dflieb Fascinating data point for grammarians incidentally. I'm a native English speaker and I'm not 100% sure whether "I’d have rather it been cut" is grammatical or not. It sounds off, but English is so permissive that practically everything turns out to be grammatical.

There was an air vent in the wall of the room that the Boston Museum of Fine Arts used for new shows, and it fooled me multiple times after my standards had been lowered by looking at a whole room of mediocre stuff.

@typesfast In particular it tended to act as a brake on vendettas.

@ATabarrok Fresh durian's pretty good. It tastes like a creamy pudding.

@Noahpinion You can't burn your bridges permanently with the people who are in the center accidentally. They're always willing to listen to new evidence.

https://t.co/ogMaJHrydF

"It is time for Europe to clarify specifically how it might place sanctions on Israel, and to develop its ad hoc sanctions decisions into a systematic policy framework."

— Martin Sandbu, the Financial Times's European economics commentator https://t.co/18d94vOr2Q

@malmaladei He is now.

The reason I like this picture so much is that he's not smiling at something, as little kids often do, but smiling at the thought of something. He's thinking about something that makes him happy.

I told 13 yo that if he ever writes an autobiography, he should put this on the cover. He's not convinced. (The reason he looks so happy is that we're about to go on a trip to the beach.) https://t.co/ovEuOBuFXq

@uncertainsys The scary thing is that if I had to choose between resembling that guy in body or soul, I'd have to choose body.


Snapshot: Viaweb, June 1998
January 2012
A few hours before the Yahoo acquisition was announced in June 1998
I took a
snapshot of Viaweb's
site
.  I thought it might be interesting to look at one day.
The first thing one notices is is how tiny the pages are.  Screens
were a lot smaller in 1998.  If I remember correctly, our frontpage
used to just fit in the size window people typically used then.
Browsers then (IE 6 was still 3 years in the future) had few fonts
and they weren't antialiased.  If you wanted to make pages that
looked good, you had to render display text as images.
You may notice a certain similarity between the Viaweb and
Y Combinator
logos.  We did that
as an inside joke when we started YC.  Considering how basic a red
circle is, it seemed surprising to me when we started Viaweb how
few other companies used one as their logo.  A bit later I realized
why
.
On the
Company
page
you'll notice a mysterious individual called John McArtyem.
Robert Morris (aka Rtm) was so publicity averse after the
Worm
that he
didn't want his name on the site.  I managed to get him to agree
to a compromise: we could use his bio but not his name.  He has
since
relaxed
a bit
on that point.
Trevor graduated at about the same time the acquisition closed, so in the
course of 4 days he went from impecunious grad student to millionaire
PhD.  The culmination of my career as a writer of press releases
was one
celebrating
his graduation
, illustrated with a drawing I did of him during
a meeting.
(Trevor also appears as
Trevino
Bagwell
in our directory of web designers merchants could hire
to build stores for them.  We inserted him as a ringer in case some
competitor tried to spam our web designers.   We assumed his logo
would deter any actual customers, but it did not.)
Back in the 90s, to get users you had to get mentioned in magazines
and newspapers.  There were not the same ways to get found online
that there are today.  So we used to pay a
PR
firm
$16,000 a month to get us mentioned in the press.  Fortunately
reporters
liked
us
.
In our
advice about
getting traffic from search engines
(I don't think the term SEO
had been coined yet), we say there are only 7 that matter: Yahoo,
AltaVista, Excite, WebCrawler, InfoSeek, Lycos, and HotBot.  Notice
anything missing?  Google was incorporated that September.
We supported online transactions via a company called
Cybercash
,
since if we lacked that feature we'd have gotten beaten up in product
comparisons.  But Cybercash was so bad and most stores' order volumes
were so low that it was better if merchants processed orders like phone orders.  We had a page in our site trying to
talk merchants
out of doing real time authorizations
.
The whole site was organized like a funnel, directing people to the
test drive
.
It was a novel thing to be able to try out software online.  We put
cgi-bin in our dynamic urls to fool competitors about how our
software worked.
We had some
well
known users
.  Needless to say, Frederick's of Hollywood got the
most traffic.  We charged a flat fee of $300/month for big stores,
so it was a little alarming to have users who got lots of traffic.
I once calculated how much Frederick's was costing us in bandwidth,
and it was about $300/month.
Since we hosted all the stores, which together were getting just
over 10 million page views per month in June 1998, we consumed what
at the time seemed a lot of bandwidth.  We had 2 T1s (3 Mb/sec)
coming into our offices.  In those days there was no AWS.  Even
colocating servers seemed too risky, considering how often things
went wrong with them.  So we had our servers in our offices.  Or
more precisely, in Trevor's office.  In return for the unique
privilege of sharing his office with no other humans, he had to
share it with 6 shrieking tower servers.  His office was nicknamed
the Hot Tub on account of the heat they generated.  Most days his
stack of window air conditioners could keep up.
For describing pages, we had a template language called
RTML
, which
supposedly stood for something, but which in fact I named after
Rtm.  RTML was Common Lisp augmented by some macros and libraries,
and concealed under a structure editor that made it look like it
had syntax.
Since we did continuous releases, our software didn't actually have
versions.  But in those days the trade press expected versions, so
we made them up.  If we wanted to get lots of attention, we made
the version number
an
integer
.  That "version 4.0" icon was generated by our own
button generator, incidentally.  The whole Viaweb site was made
with our software, even though it wasn't an online store, because
we wanted to experience what our users did.
At the end of 1997, we released a general purpose shopping search
engine called
Shopfind
.  It
was pretty advanced for the time.  It had a programmable crawler
that could crawl most of the different stores online and pick out
the products.


How to Start Google
March 2024
(This is a talk I gave to 14 and 15 year olds about what to do now
if they might want to start a startup later. Lots of schools think
they should tell students something about startups. This is what I
think they should tell them.)
Most of you probably think that when you're released into the
so-called real world you'll eventually have to get some kind of
job. That's not true, and today I'm going to talk about a trick you
can use to avoid ever having to get a job.
The trick is to start your own company. So it's not a trick for
avoiding
work
, because if you start your own company you'll
work harder than you would if you had an ordinary job. But you will
avoid many of the annoying things that come with a job, including
a boss telling you what to do.
It's more exciting to work on your own project than someone else's.
And you can also get a lot richer. In fact, this is the standard
way to get
really rich
. If you look at the lists of the richest
people that occasionally get published in the press, nearly all of
them did it by starting their own companies.
Starting your own company can mean anything from starting a barber
shop to starting Google. I'm here to talk about one extreme end of
that continuum. I'm going to tell you how to start Google.
The companies at the Google end of the continuum are called startups
when they're young. The reason I know about them is that my wife
Jessica and I started something called Y Combinator that is basically
a startup factory. Since 2005, Y Combinator has funded over 4000
startups. So we know exactly what you need to start a startup,
because we've helped people do it for the last 19 years.
You might have thought I was joking when I said I was going to tell
you how to start Google. You might be thinking "How could
we
start Google?" But that's effectively what the people who did start
Google were thinking before they started it. If you'd told Larry
Page and Sergey Brin, the founders of Google, that the company they
were about to start would one day be worth over a trillion dollars,
their heads would have exploded.
All you can know when you start working on a startup is that it
seems worth pursuing. You can't know whether it will turn into
a company worth billions or one that goes out of business. So when I
say I'm going to tell you how to start Google, I mean I'm going to
tell you how to get to the point where you can start a company that
has as much chance of being Google as Google had of being Google.
[
1
]
How do you get from where you are now to the point where you can
start a successful startup? You need three things. You need to be
good at some kind of technology, you need an idea for what you're
going to build, and you need cofounders to start the company with.
How do you get good at technology? And how do you choose which
technology to get good at? Both of those questions turn out to have
the same answer: work on your own projects. Don't try to guess
whether gene editing or LLMs or rockets will turn out to be the
most valuable technology to know about. No one can predict that.
Just work on whatever interests you the most. You'll work much
harder on something you're interested in than something you're doing
because you think you're supposed to.
If you're not sure what technology to get good at, get good at
programming. That has been the source of the median startup for the
last 30 years, and this is probably not going to change in the next
10.
Those of you who are taking computer science classes in school may
at this point be thinking, ok, we've got this sorted. We're already
being taught all about programming. But sorry, this is not enough.
You have to be working on your own projects, not just learning stuff
in classes. You can do well in computer science classes without
ever really learning to program. In fact you can graduate with a
degree in computer science from a top university and still not be
any good at programming. That's why tech companies all make you
take a coding test before they'll hire you, regardless of where you
went to university or how well you did there. They know grades and
exam results prove nothing.
If you really want to learn to program, you have to work on your
own projects. You learn so much faster that way. Imagine you're
writing a game and there's something you want to do in it, and you
don't know how. You're going to figure out how a lot faster than
you'd learn anything in a class.
You don't have to learn programming, though. If you're wondering
what counts as technology, it includes practically everything you
could describe using the words "make" or "build." So welding would
count, or making clothes, or making videos. Whatever you're most
interested in. The critical distinction is whether you're producing
or just consuming. Are you writing computer games, or just playing
them? That's the cutoff.
Steve Jobs, the founder of Apple, spent time when he was a teenager
studying calligraphy — the sort of beautiful writing that
you see in medieval manuscripts. No one, including him, thought
that this would help him in his career. He was just doing it because
he was interested in it. But it turned out to help him a lot. The
computer that made Apple really big, the Macintosh, came out at
just the moment when computers got powerful enough to make letters
like the ones in printed books instead of the computery-looking
letters you see in 8 bit games. Apple destroyed everyone else at
this, and one reason was that Steve was one of the few people in
the computer business who really got graphic design.
Don't feel like your projects have to be
serious
. They can
be as frivolous as you like, so long as you're building things
you're excited about. Probably 90% of programmers start out building
games. They and their friends like to play games. So they build
the kind of things they and their friends want. And that's exactly
what you should be doing at 15 if you want to start a startup one
day.
You don't have to do just one project. In fact it's good to learn
about multiple things. Steve Jobs didn't just learn calligraphy.
He also learned about electronics, which was even more valuable.
Whatever you're interested in. (Do you notice a theme here?)
So that's the first of the three things you need, to get good at
some kind or kinds of technology. You do it the same way you get
good at the violin or football: practice. If you start a startup
at 22, and you start writing your own programs now, then by the
time you start the company you'll have spent at least 7 years
practicing writing code, and you can get pretty good at anything
after practicing it for 7 years.
Let's suppose you're 22 and you've succeeded: You're now really
good at some technology. How do you get
startup ideas
? It might
seem like that's the hard part. Even if you are a good programmer,
how do you get the idea to start Google?
Actually it's easy to get startup ideas once you're good at technology.
Once you're good at some technology, when you look at the world you
see dotted outlines around the things that are missing. You start
to be able to see both the things that are missing from the technology
itself, and all the broken things that could be fixed using it, and
each one of these is a potential startup.
In the town near our house there's a shop with a sign warning that
the door is hard to close. The sign has been there for several
years. To the people in the shop it must seem like this mysterious
natural phenomenon that the door sticks, and all they can do is put
up a sign warning customers about it. But any carpenter looking at
this situation would think "why don't you just plane off the part
that sticks?"
Once you're good at programming, all the missing software in the
world starts to become as obvious as a sticking door to a carpenter.
I'll give you a real world example. Back in the 20th century,
American universities used to publish printed directories with all
the students' names and contact info. When I tell you what these
directories were called, you'll know which startup I'm talking
about. They were called facebooks, because they usually had a picture
of each student next to their name.
So Mark Zuckerberg shows up at Harvard in 2002, and the university
still hasn't gotten the facebook online. Each individual house has
an online facebook, but there isn't one for the whole university.
The university administration has been diligently having meetings
about this, and will probably have solved the problem in another
decade or so. Most of the students don't consciously notice that
anything is wrong. But Mark is a programmer. He looks at this
situation and thinks "Well, this is stupid. I could write a program
to fix this in one night. Just let people upload their own photos
and then combine the data into a new site for the whole university."
So he does. And almost literally overnight he has thousands of
users.
Of course Facebook was not a startup yet. It was just a... project.
There's that word again. Projects aren't just the best way to learn
about technology. They're also the best source of startup ideas.
Facebook was not unusual in this respect. Apple and Google also
began as projects. Apple wasn't meant to be a company. Steve Wozniak
just wanted to build his own computer. It only turned into a company
when Steve Jobs said "Hey, I wonder if we could sell plans for this
computer to other people." That's how Apple started. They weren't
even selling computers, just plans for computers. Can you imagine
how lame this company seemed?
Ditto for Google. Larry and Sergey weren't trying to start a company
at first. They were just trying to make search better. Before Google,
most search engines didn't try to sort the results they gave you
in order of importance. If you searched for "rugby" they just gave
you every web page that contained the word "rugby." And the web was
so small in 1997 that this actually worked! Kind of. There might
only be 20 or 30 pages with the word "rugby," but the web was growing
exponentially, which meant this way of doing search was becoming
exponentially more broken. Most users just thought, "Wow, I sure
have to look through a lot of search results to find what I want."
Door sticks. But like Mark, Larry and Sergey were programmers. Like
Mark, they looked at this situation and thought "Well, this is
stupid. Some pages about rugby matter more than others. Let's figure
out which those are and show them first."
It's obvious in retrospect that this was a great idea for a startup.
It wasn't obvious at the time. It's never obvious. If it was obviously
a good idea to start Apple or Google or Facebook, someone else would
have already done it. That's why the best startups grow out of
projects that aren't meant to be startups. You're not trying to
start a company. You're just following your instincts about what's
interesting. And if you're young and good at technology, then your
unconscious instincts about what's interesting are better than your
conscious ideas about what would be a good company.
So it's critical, if you're a young founder, to build things for
yourself and your friends to use. The biggest mistake young founders
make is to build something for some mysterious group of other people.
But if you can make something that you and your friends truly want
to use — something your friends aren't just using out of
loyalty to you, but would be really sad to lose if you shut it down
— then you almost certainly have the germ of a good startup
idea. It may not seem like a startup to you. It may not be obvious
how to make money from it. But trust me, there's a way.
What you need in a startup idea, and all you need, is something
your friends actually want. And those ideas aren't hard to see once
you're good at technology. There are sticking doors everywhere.
[
2
]
Now for the third and final thing you need: a cofounder, or cofounders.
The optimal startup has two or three founders, so you need one or
two cofounders. How do you find them? Can you predict what I'm going
to say next? It's the same thing: projects. You find cofounders by
working on projects with them. What you need in a cofounder is
someone who's good at what they do and that you work well with, and
the only way to judge this is to work with them on things.
At this point I'm going to tell you something you might not want
to hear. It really matters to do well in your classes, even the
ones that are just memorization or blathering about literature,
because you need to do well in your classes to get into a good
university. And if you want to start a startup you should try to
get into the best university you can, because that's where the best
cofounders are. It's also where the best employees are. When Larry
and Sergey started Google, they began by just hiring all the smartest
people they knew out of Stanford, and this was a real advantage for
them.
The empirical evidence is clear on this. If you look at where the
largest numbers of successful startups come from, it's pretty much
the same as the list of the most selective universities.
I don't think it's the prestigious names of these universities that
cause more good startups to come out of them. Nor do I think it's
because the quality of the teaching is better. What's driving this
is simply the difficulty of getting in. You have to be pretty smart
and determined to get into MIT or Cambridge, so if you do manage
to get in, you'll find the other students include a lot of smart
and determined people.
[
3
]
You don't have to start a startup with someone you meet at university.
The founders of Twitch met when they were seven. The founders of
Stripe, Patrick and John Collison, met when John was born. But
universities are the main source of cofounders. And because they're
where the cofounders are, they're also where the ideas are, because
the best ideas grow out of projects you do with the people who
become your cofounders.
So the list of what you need to do to get from here to starting a
startup is quite short. You need to get good at technology, and the
way to do that is to work on your own projects. And you need to do
as well in school as you can, so you can get into a good university,
because that's where the cofounders and the ideas are.
That's it, just two things, build stuff and do well in school.
Notes
[
1
]
The rhetorical trick in this sentence is that the "Google"s
refer to different things. What I mean is: a company that has as
much chance of growing as big as Google ultimately did as Larry and
Sergey could have reasonably expected Google itself would at the
time they started it. But I think the original version is zippier.
[
2
]
Making something for your friends isn't the only source of
startup ideas. It's just the best source for young founders, who
have the least knowledge of what other people want, and whose own
wants are most predictive of future demand.
[
3
]
Strangely enough this is particularly true in countries like
the US where undergraduate admissions are done badly. US admissions
departments make applicants jump through a lot of arbitrary hoops
that have little to do with their intellectual ability. But the
more arbitrary a test, the more it becomes a test of mere determination
and resourcefulness. And those are the two most important qualities
in startup founders. So US admissions departments are better at
selecting founders than they would be if they were better at selecting
students.
Thanks
to Jared Friedman, Carolynn Levy, Jessica Livingston, Harj Taggar, and Garry Tan for reading drafts of this.