summarize: One of my biggest frustrations in the area of nutrition—sorry, nutritional
biochemistry—has to do with how little we actually know about it for certain.
The problem is rooted in the poor quality of much nutrition research, which
leads to bad reporting in the media, lots of arguing on social media, and
rampant confusion among the public. What are we supposed to eat (and not
eat)? What is the right diet for you?
If all we have to go on is media reporting about the latest big study from
Harvard, or the wisdom of some self-appointed diet guru, then we will never
escape this state of hopeless confusion. So before we delve into the specifics,
it’s worth taking a step back to try to understand what we do and don’t know
about nutrition—what kinds of studies might be worth heeding and which
ones we can safely ignore. Understanding how to discern signal from noise is
an important first step in coming up with our own plan.
Our knowledge of nutrition comes primarily from two types of studies:
epidemiology and clinical trials. In epidemiology, researchers gather data on
the habits of large groups of people, looking for meaningful associations or
correlations with outcomes such as a cancer diagnosis, cardiovascular disease,
or mortality. These epidemiological studies generate much of the diet “news”
that pops up in our daily internet feed, about whether coffee is good for you
and bacon is bad, or vice versa.
Epidemiology has been a useful tool for sleuthing out the causes of
epidemics, including (famously) stopping a cholera outbreak in nineteenthcentury London, and (less famously) saving boy chimney sweeps from an
epidemic of scrotal cancer that turned out to be linked to their employment.
[*1] It has propelled some real public health triumphs, such as the advent of
smoking bans and widespread treatment of drinking water. But in nutrition it
has proved less insightful. Even at face value, the “associations” that
nutritional epidemiologists come up with are often absurd: Will eating twelve
hazelnuts every day really add two years to my lifespan, as one study
suggested?[*2] I wish.
The problem is that epidemiology is incapable of distinguishing between
correlation and causation. This, aided and abetted by bad journalism, creates
confusion. For example, multiple studies have found a strong association
between drinking diet sodas and abdominal fat, hyperinsulinemia, and
cardiovascular risk. Sounds like diet soda is bad stuff that causes obesity,
right? But that is not what those studies actually demonstrate, because they
fail to ask an important question: Who drinks diet soda?
People who are concerned about their weight or their diabetes risk, that’s
who. They may drink diet soda because they are heavy, or worried about
becoming heavy. The problem is that epidemiology is not equipped to
determine the direction of causality between a given behavior (e.g., drinking
diet soda) and a particular outcome (e.g., obesity) any more than one of my
chickens is able to scramble the egg she has just laid for me.
To understand why, we must consult (again) Sir Austin Bradford Hill, a
British scientist whom we met in chapter 11. Hill had helped sleuth out the
link between smoking and lung cancer in the early 1950s, and he came up
with nine criteria for evaluating the strength of epidemiological findings and
determining the likely direction of causality, which we also referenced in
regard to exercise.
[*3] The most important of these, and the one that can best
separate correlation from causation, is the trickiest to implement in nutrition:
experiment. Try proposing a study where you would test the effects of a
lifetime of eating fast food by randomizing young boys and girls either to Big
Macs or a non–fast-food diet. Even if you did somehow receive institutional
review board approval for this terrible idea, there are a bunch of different
ways in which even a simple experiment can go wrong. Some of the Big Mac
kids might secretly go vegetarian, while the controls might decide to frequent
the Golden Arches. The point is that humans are terrible study subjects for
nutrition (or just about anything else) because we are unruly, disobedient,
messy, forgetful, confounding, hungry, and complicated creatures.
This is why we rely on epidemiology, which derives data from observation
and often from the subjects themselves. As we saw earlier, the epidemiology
around exercise passes the Bradford Hill criteria with flying colors—but using
epidemiology to study nutrition often flunks those tests miserably, beginning
with the effect size, the power of the association, often expressed as a
percentage. While the epidemiology of smoking (like exercise) easily passes
the Bradford Hill tests because the effect size is so overwhelming, in nutrition
the effect sizes are typically so small that they could easily be the product of
other, confounding factors.
Case in point: The claim that eating red meats and processed meats
“causes” colorectal cancer. According to a very well-publicized 2017 study
from the Harvard School of Public Health and the World Health
Organization, eating those kinds of meats raises one’s risk of colon cancer by
17 percent (HR = 1.17). That does sound scary—but does it pass the
Bradford Hill tests? I don’t think so, because the association is so weak. For
comparison’s sake, someone who smokes cigarettes is at more like 1,000 to
2,500 percent (ten to twenty-five times) increased risk of lung cancer,
depending on the population being studied. This suggests that there might
actually be some sort of causation at work. Yet very few published
epidemiological studies show a risk increase of even 50 percent (HR = 1.50)
for any given type of food.
Second, and far more damning, is that the raw data on which these
conclusions are typically based are shaky at best. Many nutritional
epidemiological studies collect information on subjects via something called a
“food frequency questionnaire,” a lengthy checklist that asks users to recall
everything they ate over the last month, or even the last year, in minute detail.
I’ve tried filling these out and it’s almost impossible to recall exactly what I
ate two days ago, let alone three weeks.
[*4] So how reliable can studies based
on such data possibly be? How much confidence do we have in, say, the red
meat study?
So do red and processed meats actually cause cancer or not? We don’t
know, and we will probably never get a more definitive answer, because a
clinical trial testing this proposition is unlikely ever to be done. Confusion
reigns. Nevertheless, I’m going to stick my neck out and assert that a risk
ratio of 1.17 is so minimal that it might not matter that much whether you eat
red/processed meats versus some other protein source, like chicken. Clearly,
this particular study is very far from providing a definitive answer to the
question of whether red meat is “safe” to eat. Yet people have been fighting
about it for years.
This is another problem in the world of nutrition: too many people are
majoring in the minor and minoring in the major, focusing too much attention
on small questions while all but ignoring the bigger issues. Small variations in
what we eat probably matter a lot less than most people assume. But bad
epidemiology, aided and abetted by bad journalism, is happy to blow these
things way out of proportion.
—
Bad epidemiology so dominates our public discussion of nutrition that it has
inspired a backlash by skeptics such as John Ioannidis of the Stanford
Prevention Research Center, a crusader against bad science in all its forms.
His basic argument is that food is so complex, made up of thousands of
chemical compounds in millions of possible combinations that interact with
human physiology in so many ways—in other words, nutritional biochemistry
—that epidemiology is simply not up to the task of disentangling the effect of
any individual nutrient or food. In an interview with the CBC, the normally
soft-spoken Ioannidis was brutally direct: “Nutritional epidemiology is a
scandal,” he said. “It should just go into the waste bin.”
The true weakness of epidemiology, at least as a tool to extract reliable,
causal information about human nutrition, is that such studies are almost
always hopelessly confounded. The factors that determine our food choices
and eating habits are unfathomably complex. They include genetics, social
influences, economic factors, education, metabolic health, marketing, religion,
and everything in between—and they are almost impossible to disentangle
from the biochemical effects of the foods themselves.
A few years ago, a scientist and statistician named David Allison ran an
elegant experiment that illustrates how epidemiological methods can lead us
astray, even in the most tightly controlled research model possible: laboratory
mice, which are genetically identical and housed in identical conditions.
Allison created a randomized experiment using these mice, similar to the
caloric restriction experiments we discussed in chapter 5. He split them into
three groups, differing only in the quantity of food they were given: a lowcalorie group, a medium-calorie group, and a high-calorie, ad libitum group
of animals who were allowed to eat as much as they wanted. The low-calorie
mice were found to live the longest, followed by medium-calorie mice, and
the high-calorie mice lived the shortest, on average. This was the expected
result that had been well established in many previous studies.
But then Allison did something very clever. He looked more closely at the
high-calorie group, the mice with no maximum limit on food intake, and
analyzed this group separately, as its own nonrandomized epidemiological
cohort. Within this group, Allison found that some mice chose to eat more
than others—and that these hungrier mice actually lived longer than the high-
calorie mice who chose to eat less. This was exactly the opposite of the result
found in the larger, more reliable, and more widely repeated randomized trial.
There was a simple explanation for this: the mice that were strongest and
healthiest had the largest appetites, and thus they ate more. Because they were
healthiest to begin with, they also lived the longest. But if all we had to go on
was Allison’s epidemiological analysis of this particular subgroup, and not the
larger and better-designed clinical trial, we might conclude that eating more
calories causes all mice to live longer, which we are pretty certain is not the
case.
This experiment demonstrates how easy it is to be misled by
epidemiology. One reason is because general health is a massive confounder
in these kinds of studies. This is also known as healthy user bias, meaning
that study results sometimes reflect the baseline health of the subjects more
than the influence of whatever input is being studied—as was the case with
the “hungry” mice in this study.
[*5]
One classic example of this, I believe, lies in the vast, well-publicized
literature correlating “moderate” drinking with improved health outcomes.
This notion has become almost an article of faith in the popular media, but
these studies are also almost universally tainted by healthy user bias—that is,
the people who are still drinking in older age tend to do so because they are
healthy, and not the other way around. Similarly, people who drink zero
alcohol often have some health-related reason, or addiction-related reason, for
avoiding it. And such studies also obviously exclude those who have already
died of the consequences of alcoholism.
Epidemiology sees only a bunch of seemingly healthy older people who
all drink alcohol and concludes that alcohol is the cause of their good health.
But a recent study in JAMA, using the tool of Mendelian randomization we
discussed back in chapter 3, suggests that this might not be true. This study
found that once you remove the effects of other factors that may accompany
moderate drinking—such as lower BMI, affluence, and not smoking—any
observed benefit of alcohol consumption completely disappears. The authors
concluded that there is no dose of alcohol that is “healthy.”
—
Clinical trials would seem like a much better way to evaluate one diet against
another: One group of subjects eats diet X, the other group is on diet Y, and
you compare the results. (Or, to continue the alcohol example, one group
drinks moderately, one group drinks heavily, and the control group abstains
altogether.)