# Genetic fitness is a measure of selection strength, not the selection target

*Alternative title: "Evolution suggests robust rather than fragile generalization of alignment properties."*

A [frequently](https://www.lesswrong.com/posts/GNhMPAWcfBCASy8e6/a-central-ai-alignment-problem-capabilities-generalization) [repeated](https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities) [argument](https://www.lesswrong.com/posts/JcLhYQQADzTsAEaXd/ai-as-a-science-and-three-obstacles-to-alignment-strategies) goes something like this: 

1.  Evolution has optimized humans for inclusive genetic fitness (IGF)
2.  However, humans didn’t end up explicitly optimizing for genetic fitness (e.g. they use contraception to avoid having children)
3.  Therefore, even if we optimize an AI for X (typically something like "human values"), we shouldn’t expect it to explicitly optimize for X 

My argument is that premise 1 is a verbal shorthand that’s technically incorrect, and premise 2 is at least misleading. As for the overall conclusion, I think that the case from evolution might be interpreted as weak evidence for why AI should be expected to *continue* optimizing human values even as its capability increases.

**Summary of how premise 1 is wrong:** If we look closely at what evolution does, we can see that it selects for traits that are beneficial for surviving, reproducing, and passing one’s genes to the next generation. This is often described as “optimizing for IGF”, because the traits that are beneficial for these purposes are *usually* the ones that have the highest IGF. (This has some important exceptions, discussed later.) However, if we look closely at that process of selection, we can see that this kind of trait selection is *not* “optimizing for IGF” in the sense that, for example, we might optimize an AI to classify pictures.

The model that I’m sketching is something like this: evolution is an optimization function that, at any given time, is selecting for some traits that are in an important sense chosen at random. At any time, it might randomly shift to selecting for some other traits. Observing this selection process, we can calculate the IGF of traits currently under selection, as a measure of how strongly those are being selected. But evolution is not *optimizing for this measure*; evolution is *optimizing for* *the traits that have currently been chosen for optimization*. Resultingly, there is no reason to expect that the minds created by evolution should optimize for IGF, but there *is* reason to expect that they would optimize for the traits that were actually under selection. This is something that we observe any time that humans optimize for some biological need. 

In contrast, if we were optimizing an AI to classify pictures, we would not be randomly changing the selection criteria the way that evolution does. We would keep the selection criteria constant: always selecting for the property of classifying pictures the way we want. To the extent that the analogy to evolution holds, AIs should be much more likely to just do the thing they were selected for. 

**Summary of how premise 2 is misleading: **It is often implied that evolution selected humans to care about sex, and then sex led to offspring, and it was only recently with the evolution of contraception that this connection was severed. For example: 

> 15\. \[...\] We didn't break alignment with the 'inclusive reproductive fitness' outer loss function, immediately after the introduction of farming - something like 40,000 years into a 50,000 year Cro-Magnon takeoff, as was itself running very quickly relative to the outer optimization loop of natural selection.  Instead, we got a lot of technology more advanced than was in the ancestral environment, including contraception, in one very fast burst relative to the speed of the outer optimization loop, late in the general intelligence game.
> 
> – Eliezer Yudkowsky, [AGI Ruin: A List of Lethalities](https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities)

This seems wrong to me. Contraception may be a very recent invention, but infanticide or killing children by neglect is not; there have always been methods for controlling the population size even without contraception. According to the book [Anthropology of Childhood](https://www.lesswrong.com/posts/vwM7hnT9ysE3suwfk/notes-on-the-anthropology-of-childhood), family sizes and the economic value of having children have always been correlated. Children are more of a burden on foragers and foragers correspondingly have smaller family sizes, whereas children are an asset for farmers who have larger family sizes. 

Rather than evolution having selected humans for IGF and this linkage then breaking with the invention of contraception, evolution has selected humans to have an optimization function that weighs various factors in considering how many children to have. In forager-like environments, this function leads to a preference for fewer children and smaller family sizes; in farmer-like environments, this functions leads to a preference for more children and larger family sizes. [@RobinHanson](https://www.lesswrong.com/users/robinhanson?mention=user) [has suggested](https://www.overcomingbias.com/p/forager-v-farmer-elaboratedhtml) that modern society is more forager-like than farmer-like and that our increased wealth is causing us to revert to forager-like ways and psychology. To the extent that this argument is true, there has been no breakage between what evolution “intended” and how humans behave; rather, the optimization function that evolution created continues operating the way it always has. 

The invention of modern forms of contraception may have made it easier to limit family sizes in a farmer-type culture that had evolved cultural taboos against practices like infanticide. But rather than creating an entirely new evolutionary environment, finding a way to bypass those taboos brought us *closer* to how things had been in our original evolutionary environment. 

If we look at what humans were selected to optimize for, it looks like we are mostly continuing to optimize for those same things. The reason why a minority of people are choosing not to have children is because our evolved optimization function also values things other than children, and we have “stayed loyal” to this optimization function. In the case of an AI that was trained to act according to something like “human values” and nothing else, the historical example seems to suggest that its alignment properties might generalize even more robustly than ours, as it had not been selected for a mixture of many competing values.

Evolution as a force that selects for traits at random
======================================================

For this post, I skimmed two textbooks on evolution: [*Evolution (4th edition)* by Futuyama & Kirkpatrick](https://www.amazon.com/Evolution-Douglas-J-Futuyma/dp/1605356050/) and [*Evolutionary Analysis (5th edition)* by Herron & Freeman](https://www.amazon.com/Evolutionary-Analysis-5th-Jon-Herron/dp/0321616677/). The first one was selected based on Googling “what’s the best textbook on evolutionary biology” and the second was selected because an earlier edition was used in an undergraduate course on evolutionary psychology that I once took and I recalled it being good.

As far as I could tell, neither one talked about evolution as a process that optimizes for genetic fitness (though this was a rather light skim so I may have missed it even if it was there). Rather, they cautioned *against* thinking of evolution as an active agent that “does” anything in the first place. Evolution does *increase a population’s average adaptation to its environment* (Herron & Freeman, p. 107), but what this means can constantly change as the environment itself changes. At one time in history, a region may have a cold climate, selecting the species there for an ability to deal with the cold; and then the climate may shift to a warmer one, and previously beneficial adaptations like fur may suddenly become a liability. 

Another classic example is that of [peppered moth evolution](https://en.wikipedia.org/wiki/Peppered_moth_evolution). Light-colored moths used to be the norm in England, with dark-colored ones being very rare, as a light coloration was a better camouflage against birds than a dark one. With the Industrial Revolution and the appearance of polluting factories, some cities became so black that dark color became better camouflage, leading to an increase in dark-colored moths relative to the light-colored ones. And once pollution was reduced, the light-colored moths came to dominate again.

If we were modeling evolution as a mathematical function, we could say that it was first selecting for light coloration in moths, then changed to select for dark, then changed to select for light again. 

The closest that one gets to something like “evolution optimizing for genetic fitness” is what’s called “[the fundamental theorem of natural selection](https://en.wikipedia.org/wiki/Fisher%27s_fundamental_theorem_of_natural_selection)”, which among other things implies that natural selection will cause the mean fitness of a population to increase over time. However, here we are assuming that *the thing we are selecting for remains constant. *Light-colored moths will continue to become more common over time, up until a dark coloration becomes the trait with higher fitness and the dark coloration starts becoming more common. In both situations we might say that the “mean fitness of the population is increasing”, but this means a *different thing* in those two situations: in one situation it means selecting for white coloration, and in another situation, it means selecting for dark coloration. The thing that was first being selected *for*, is then being selected *against*, even as our terminology implies that the *same* thing is being selected for. 

What happened was that the mean fitness of the population went up as a particular coloration was selected for, then a random change (first the increased pollution, then the decreased pollution) caused the mean fitness to fall, and then it started climbing again.

> *All else being equal, the fundamental theorem would lead us to expect that the mean fitness of species should increase by a few percent per generation. But all else is not equal: what selection gives, other evolutionary forces take away. The fitness gains made by selection are continuously offset by environments that change in space and time, deleterious mutations, and other factors.* (Futuyama & Kirkpatrick, p. 127)

Even taking this into account, evolution does not even consistently increase the mean fitness of the population: sometimes evolution ends up selecting for a *decrease* in the mean fitness of the population.

> *The fundamental theorem and the adaptive landscape make assumptions that do not apply exactly to any natural populations. In many cases, though, they give very good approximations that are useful to guide our thinking about evolution. In other cases, the assumptions are violated in ways that make evolution behave very differently. A particularly important situation where the fundamental theorem and adaptive landscape do not apply is when selection is frequency dependent. In some cases, this can cause the mean fitness of a population to decline *(Futuyama & Kirkpatrick, p. 128)

An example of frequency-dependent selection leading to *lower* mean fitness is the case of a bush that produces many fruits (Futuyama & Kirkpatrick, p. 129). Some bushes then evolve a trunk that causes them to cast shade over their neighbors. As a result, those neighbors weaken and die, allowing the bushes that have become trees to get more water and nutrients.

This leads to the trees becoming more common than the bushes. But since trees need to spend much more energy on producing and maintaining their trunk, they don’t have as much energy to spend on growing fruit. When trees were rare and mostly stealing energy from the bushes, this wasn’t as much of a problem; but once the whole population consists of trees, they can end up shading each other. At this point, they end up producing much less fruit from which new trees could grow, so have fewer offspring and thus a lower mean fitness.

![](https://lh7-us.googleusercontent.com/2ulIWVX2urh2Xx8oU5Ix4RTOz_0UsZOkhyw5WTQ2YEKgXIo1Ej00p3cfhbHNRHt55oMhAOJvj9Vmd4Z7ZU6aF5k7kfLpwHmta6ZwpMLZcCV-hx5JJsUsqWkVsm2UtvzAaPBqEEL8hGE6d-4rZ33oP2o)

This kind of frequency-dependent selection is common. Another example (Futuyama & Kirkpatrick, p. 129) is that of bacteria that evolve both toxins that kill other bacteria, while also evolving an antidote against the toxin. Both cost energy to produce, but as long as these bacteria are rare, it’s worth the cost as the toxicity allows them to kill off their competitors. 

But once these toxic bacteria establish themselves, there’s no longer any benefit to producing the toxin - all the surviving bacteria are immune to it - so continuing to spend energy on producing it means there’s less energy available for replication. It now becomes more beneficial to keep the antidote production but lose the toxin production: the toxin production goes from being selected for, to being selected against. 

Once this selection process has happened for long enough and non-toxin-producing bacteria have come to predominate, the antidote production *also* becomes an unnecessary liability. Nobody is producing the toxin anymore, so there’s no reason to waste energy on maintaining a defense against it, so the antidote also goes from being selected for to being selected against.

But then what happens once none of the bacteria are producing the toxin *or* the antidote anymore? Now that nobody has a defense against the toxin, it becomes advantageous to start producing the toxin + antidote combination again, thus killing all the other bacteria that don’t have the antidote… and thus the cycle repeats. 

In this section, I have argued that to the extent that evolution is “optimizing a species for fitness”, this actually means different things (selecting for different traits) in different circumstances; and also evolution optimizing for fitness is more of a rough heuristic rather than a literal law anyway since there are many circumstances where evolution ends up *lowering* the fitness of a population. This alone should make us suspicious of the argument that “evolution selected humans for IGF”; what that means isn't that there's a single thing that was being optimized for, but rather that there was a wide variety of traits that were selected for at different times.

What exactly is fitness, again?
===============================

So far I’ve been talking about fitness in general terms, but let’s recap some of the technical details. What exactly *is* inclusive genetic fitness, again?

There are several different definitions; here’s one set of them.

A simple definition of *fitness* is that it’s the number of offspring that an individual leaves for the next generation[^ns0k7bysais]. Suppose that 1% of a peppered moth’s offspring survive to reproductive age and that the surviving moths have an average of 300 offspring. In this case, the average fitness of these individuals is 0.01 * 300 = 3.

For evolution by natural selection to occur, fitness differences among individuals need to be inherited. In biological evolution, inheritance happens through genes, so we are usually interested in *genetic fitness* \- the fitness of genes. Suppose that these are all light-colored moths in a polluted city. Suppose a gene allele for dark coloration increases the survivability by 0.33 percentage points, for an overall fitness of 0.0133 * 300 = 4. The fitnesses of the alleles are now 3 and 4. 

![](https://lh7-us.googleusercontent.com/XY1BT86uYuu7cxNldiP9QINMHpMEKdkqDo2WUMjKfL0zkoNt-_8j7SuNOJHWdzmkqzCODgfUW2-WNABcqYjT6M4vml07p2eJr_nK6eZHBubIwQ0Is8Vat9T2VCqOabrs5g8lKmv9YFaubu2zjUUkzSE)

Image from Futuyama & Kirkpatrick. Caption in the original: *Genotype A has a fitness of 3, while genotype B has a fitness of 4. Both genotypes start with 10 individuals. (A) The population size of genotype B grows much more rapidly. (B) Plotting the frequencies of the two genotypes shows that genotype B, which starts at a frequency of 0.5, makes up almost 90% of the population just 7 generations later.*

Often what matters is the *difference* in fitness between two alleles: for example, an allele with a fitness of 2 may become more common in the population if its competitor has a fitness of 1, but will become more rare if its competitor has a fitness of 3. Thus it’s common to indicate fitness *relative* to some common reference, such as the average fitness of the population or the genotype with the highest absolute fitness.

Genetic fitness can be divided into two components. An individual can pass a gene directly onto their offspring - this is called *direct fitness*. They can also carry a genetic adaptation that causes them to help others with the same adaptation, increasing their odds of survival. For example, a parent may invest extra effort in taking care of their offspring. This is called *indirect fitness. *The *inclusive fitness* of a genotype is the sum of its direct and indirect fitness.[^23b1jpur496]

Biological evolution can be defined as “inherited change in the properties of organisms over the course of generations” (Futuyama & Kirkpatrick, p. 7). *Evolution by natural selection* is when the relative frequencies of a genotype change across generations due to differences in fitness. Note that genotype frequencies can also change across generations for reasons other than natural selection, such as random drift or novel mutations.

Fitness as a measure of selection strength
==========================================

Let’s look at a case of intentional animal breeding. The details of the math that follows aren’t that important, but I wanted to run through them anyway, just to make it more concrete what “fitness” *actually means*. Still, you can just skim through them if you prefer.

Suppose that I happen to own a bunch of peppered moths of various colors and happen to like a light color, so I decide to breed them towards being lighter. Now I don’t know the details of how the genetics of peppered moth coloration works - I assume that it might very well be affected by multiple genes. But for the sake of simplicity, let’s just say that there is a single gene with a “light” allele and a “dark” allele.

Call the “light” allele B1 and the “dark” allele B2. B1B1 moths are light, B2B2 moths are dark, and B1B2 / B2B1 moths are somewhere in between (to further simplify things, I’ll use “B1B2” to refer to both B1B2 and B2B1 moths). 

Suppose that the initial population has 100 moths. I have been doing breeding for a little bit already, so we start from B1 having a frequency of 0.6, and B2 a frequency of 0.4. The moths have the following distribution of genotypes:

B1B1 = 36

B1B2 = 48

B2B2 = 16

To my eye, all of the moths with the B1B1 genotype look pleasantly light, so I choose to have them all breed. 75% of the moths with the B1B2 genotype look light enough to my eye, and so do 50% of the B2B2 ones (maybe their coloration is also affected by environmental factors or other genes). The rest don’t get to breed.

This gives us, on average, a frequency of 0.675 for the B1 alleles and 0.325 for the B2 alleles in the next generation[^h3ks5duler4]. Assuming that each of the moths contributed a hundred gametes to the next generation, we get the following fitnesses for the alleles:

B1: Went from 120 (36 + 36 + 48) to 5400 copies, so the fitness is 5400/120 = 45.

B2: Went from 80 (48 + 16 + 16) to 2600 copies, so the fitness is 2600/80 = 32.5.

As the proportion of B1 increases, the average fitness of the population will increase! This is because the more B1 alleles you carry, the more likely it is that you are selected to breed, so B1 carriers have a higher fitness… which means that B1 becomes more common… which increases the average fitness of the mouse population as a whole. So in this case, the rule that the average fitness of the population tends to increase over time does apply.

But now… wouldn’t it sound pretty weird to describe this process as *optimizing for the fitness of the moths? *

I am optimizing for *having light moths*; what the fitness calculation tells us is *how much of an advantage the lightness genes have -* in other words, *how much I am favoring the lightness genes* -  relative to the darkness genes. 

Because we were only modeling the effect of fitness and not e.g. random drift, all of the difference in gene frequencies came from the difference in fitness. This is tautological - it doesn’t matter *what* you are selecting (optimizing) for, *anything* that gets selected ends up having the highest fitness, by definition.

Rather than saying that we were optimizing for high fitness, it seems more natural to say that we were optimizing for the trait of *lightness* and that *lightness gave a fitness advantage. *The other way around doesn’t make much sense - we were optimizing for fitness and that gave an advantage to lightness? What?

This example used artificial selection because that makes it the most obvious what the actual selection target was. But the math works out the same regardless of whether we’re talking artificial or natural selection. If we say that instead of me deciding that some moths don’t get to breed, the birds and other factors in the external environment are doing it… well, nothing changes about the equations in question.

Was natural selection optimizing for the fitness of the moths? There's a sense in which you could say that since the dark-colored moths ended up having increased fitness compared to the light-colored ones. But it would also again feel a little off to describe it this way; it feels more informative and precise to say that the moths were *optimized for having dark color*, or to put it more abstractly, for having the kind of a color that fits their environment.

From coloration to behavior
===========================

I’ve just argued that if we look at the actual process of evolution, it looks more like optimizing for having specific traits (with fitness as a measure of how strongly they’re selected) rather than optimizing for fitness as such. This is so even though the process of selection *can* lead to the mean fitness of the population increasing - but as we can see from the math, this just means “if you select for something, then you get more of the thing that you are selecting for”.

In the sections before that, I argued that there’s no single thing that evolution selects for; rather, the thing that it’s selecting is constantly changing.

I think these arguments are sufficient to conclude that the claim “evolution optimized humans for fitness \[thus humans ought to be optimizing for fitness\]” is shaky.

So far, I have mostly been talking about relatively “static” traits such as coloration, rather than cognitive traits that are by themselves optimizers. So let's talk about cognition. While saying that “evolution optimized humans for genetic fitness, thus humans ought to be optimizing for fitness” seems shaky, the corresponding argument *does* work if we talk about specific cognitive behaviors that were selected for. 

For example, if we say that “humans were selected for caring about their offspring, thus humans should be optimizing for ensuring the survival of their offspring”, then this statement *does* generally speaking hold - a lot of humans do put quite a lot of cognitive effort into ensuring that their children survival. Or if we say that “humans were selected for exhibiting sexual jealousy in some circumstances, so in some circumstances, they will optimize for preventing their mates from having sex with other humans”, then clearly that statement does *also* hold.

This gets to my second part of the argument: while it’s claimed that we are now doing something that goes completely against what evolution selected for, contraception at least is a poor example of that. For the most part, we are still optimizing for exactly the things that evolution selected us to optimize for.

Humans still have the goals we were selected for
================================================

The desire to have sex was never sufficient for having babies by itself - or at least not for having ones that would survive long enough to reproduce themselves in turn. It was always only one component, with us having multiple different desires relating to children:

1.  A desire to have sex and to enjoy it for its own sake
2.  A desire to have children for its own sake
3.  A desire to take care of and protect children (including ones that are not your own) for its own sake

Eliezer wrote, in “AGI Ruin: A List of Lethalities” that

> 15\. \[...\] We didn't break alignment with the 'inclusive reproductive fitness' outer loss function, immediately after the introduction of farming - something like 40,000 years into a 50,000 year Cro-Magnon takeoff, as was itself running very quickly relative to the outer optimization loop of natural selection.  Instead, we got a lot of technology more advanced than was in the ancestral environment, including contraception, in one very fast burst relative to the speed of the outer optimization loop, late in the general intelligence game.  We started reflecting on ourselves a lot more, started being programmed a lot more by cultural evolution, and lots and lots of assumptions underlying our alignment in the ancestral training environment broke simultaneously.

This quote seems to imply that 

*   effective contraception is a relatively recent invention
*   it’s the desire for sex alone that’s the predominant driver for having children (and effective contraception breaks this assumption)
*   it’s a novel development that we prioritize things-other-than-children so much

All of these premises seem false to me. Here’s why:

*Effective contraception is a relatively recent innovation. *Even hunter-gatherers have access to effective “contraception” in the form of infanticide, which is commonly practiced among some modern hunter-gatherer societies. Particularly sensitive readers may want to skip the following paragraphs from [*The Anthropology of Childhood*](https://www.lesswrong.com/posts/vwM7hnT9ysE3suwfk/notes-on-the-anthropology-of-childhood):

> The Ache \[a Paraguyan foraging society\] are particularly direct in disposing of surplus children (approximately one-fifth) because their peripatetic, foraging lifestyle places an enormous burden on the parents. The father provides significant food resources, and the mother provides both food and the vigilant monitoring required by their dangerous jungle environment. Both men and women face significant health and safety hazards throughout their relatively short lives, and they place their own welfare over that of their offspring. A survey of several foraging societies shows a close association between the willingness to commit infanticide and the daunting challenge “to carry more than a single young child on the nomadic round” (Riches 1974: 356). 
> 
> Among other South American foragers, similar attitudes prevail. The Tapirapé from central Brazil allow only three children per family; all others must be left behind in the jungle. Seasonally scarce resources affecting the entire community dictate these measures (Wagley 1977). In fact, the availability of adequate resources is most commonly the criterion for determining whether an apparently healthy infant will be kept alive (Dickeman 1975). Among the Ayoreo foragers of Bolivia, it is customary for women to have several brief affairs, often resulting in childbirth, before settling into a stable relationship equaling marriage. “Illegitimate” offspring are often buried immediately after birth. During Bugos and McCarthy’s (1984) fieldwork, 54 of 141 births ended in infanticide. 

It takes years for a newborn to get to a point where they can take care of themselves, so a simple lack of active caretaking is enough to kill an infant, no modern-age contraceptive techniques required.

*It’s the desire for sex alone that’s the predominant driver for there being children. *Again, see infanticide, which doesn’t need to be an active act as much as a simple omission. One needs an active desire to keep children *alive*.

Also, even though the share of voluntarily childfree people is increasing, it’s still not the predominant choice. [One 2022 study](https://theconversation.com/more-than-1-in-5-us-adults-dont-want-children-187236) found that 22% of the people polled neither had nor wanted to have children - which is a significant amount, but still leaves 78% of people as ones who either have or want to have children. There’s still a strong drive to have children that’s separate from the drive to just have sex.

*It’s a novel cultural development that we prioritize things other-than-having-children so much.* Anthropology of Childhood spends significant time examining the various factors that affect the treatment of children in various cultures. It quite strongly argues that the value of children has always also been strongly contingent on various cultural and economic factors - meaning that it has always been just *one* of the things that people care about. (In fact, a desire to have lots of children may be more tied to agricultural and industrial societies, where the economic incentives for it are abnormally high.) 

> Adults are rewarded for having lots of offspring when certain conditions are met. First, mothers must be surrounded by supportive kin who relieve them of much of the burden of childrearing so they can concentrate their energy on bearing more children. Second, those additional offspring must be seen as “future workers,” on farm or in factory. They must be seen as having the potential to pay back the investment made in them as infants and toddlers, and pretty quickly, before they begin reproducing themselves. Failing either or both these conditions, humans will reduce their fertility (Turke 1989). Foragers, for whom children are more of a burden than a help, will have far fewer children than neighboring societies that depend on agriculture for subsistence (LeVine 1988). \[...\]
> 
> In foraging societies, where children are dependent and unproductive well into their teens, fewer children are preferred. In farming societies, such as the Beng, children may be welcomed as “little slaves” (Gottlieb 2000: 87). In pastoral and industrial societies, where young children can undertake shepherding a flock, or do repetitive machine-work, women are much more fertile. And, while the traditional culture of the village affords a plethora of customs and taboos for the protection of the pregnant mother and newborn, these coexist with customs that either dictate or at least quietly sanction abortion and infanticide. 

To me, the simplest story here looks something like “evolution selects humans for having various desires, from having sex to having children to creating art and lots of other things too; and all of these desires are then subject to complex learning and weighting processes that may emphasize some over others, depending on the culture and environment”. 

Some people will end up valuing children more, for complicated reasons; other people will end up valuing other things more, again for complicated reasons. This was the case in hunter-gatherer times and this is the case now. 

But it *doesn’t* look to me like evolution selected us to desire one thing, and then we developed an inner optimizer that ended up doing something completely different. Rather, it looks like we were selected to desire many different things, with a very complicated function choosing which things in that set of doings each individual ends up emphasizing. Today’s culture might have shifted that function to weigh our desires in a different manner than before, but everything that we do is still being selected from within that set of basic desires, with the weighting function operating the same as it always has. 

As I mentioned in the introduction, Robin Hanson [has suggested](https://www.overcomingbias.com/p/forager-v-farmer-elaboratedhtml) that modern society is more forager-like than farmer-like and that our increased wealth is causing us to revert to forager-like ways and psychology. This would then mean that our evolved weighting function is now exhibiting the kind of behavior that it was evolved to exhibit in a forager-like environment.

We do engage in novel activities like computer games today, but it seems to me like the motivation to play computer games is still rooted in the same kinds of basic desires as the first hunter-gatherers had - e.g. to pass the time, enjoy a good story, socialize, or experience a feeling of competence.

So what can we say about AI?
============================

Well, I would be cautious around reasoning by analogy. I’m not sure we can draw particularly strong claims about the connection to AI. I think that there are [more direct and relevant arguments](https://forum.effectivealtruism.org/posts/kLYD95SK8tQFRmw4T/ben-garfinkel-s-shortform?commentId=XMjAWSEgp9BXTDQBi) that one can make that do seem worrying, rather than trying to resort to evolutionary analogies.

But it does seem to me that e.g. the evolutionary history for the “sharp left turn” implies the *opposite* than previously argued. Something like “training an AI for recognizing pictures” or “training an AI for caring about human values” looks a lot more like “selecting humans to care about having offspring” than it looks like “optimizing humans for genetic fitness”. Caring about having offspring is a property that we still seem to pretty robustly carry; our alignment properties continued to generalize even as our capabilities increased. 

To the extent that we do not care about our offspring, or even choose to go childfree, it’s just because we were selected to *also *care about other things - if a process selects humans to care about a mix of many things, them sometimes weighing those other things more does not by itself represent a failure of alignment. This is again in sharp contrast to something like an AI that we tried to *exclusively* optimize for caring about human well-being. So there’s reason to expect that an AI’s alignment properties might generalize *even more* than those of existing humans.

*Thanks to Quintin Pope, Richard Ngo, and Steve Byrnes for commenting on previous drafts of this essay.*

[^ns0k7bysais]: Futuyama & Kirkpatrick, p. 60. 

[^23b1jpur496]: Futuyama & Kirkpatrick, p. 300. 

[^h3ks5duler4]: Each B1B1 moth has a 100% chance to “pick” a B1 allele for producing a gamete, each B1B2 moth has a 50% chance to pick a B1 gamete and a 50% chance to pick a B2 gamete, and each B2B2 moth has a 100% to pick a B2 allele for producing a gamete. Assuming that each moth that I’ve chosen to breed contributes 100 gametes to the next generation, we get an average of 3600 B1 gametes from the 36 B1B1 moths chosen to breed, 1800 B1 and 1800 B2 gametes from the 360 B1B2 moths chosen to breed, and 800 B2B2 gametes from the 8 B2B2 moths chosen to breed.This makes for 3600 + 1800 = 5400 B1 gametes and 1800 + 800 = 2600 B2 gametes, for a total of 8000 gametes. This makes for a frequency of 0.675 for B1 and 0.325 for B2.