# Categorizing Selection Effects

*\[Author's note: LLMs were used to generate and sort many individual examples into their requisite categories, as well as find and summarize relevant papers, and extensive assistance with editing\]*

The earliest recording of a selection effect is likely the story of Diagoras regarding the "Votive Tablets." When shown paintings of sailors who had prayed to Poseidon and survived shipwrecks, implying that prayer saves lives, Diagoras asked: “Where are the pictures of those who prayed, and were then drowned?”

Selection effects are sometimes considered the most pernicious class of error in data science and policy-making because they do not merely obscure the truth, they often invert it. They create "phantom patterns" where the exact opposite of reality appears to be statistically significant.

Unlike measurement errors, which can often be averaged out with more data, selection effects are *structural*. Adding more data from a biased process only increases the statistical significance of your wrong conclusion.

The danger of selection effects lies in their ability to violate the fundamental assumption of almost all statistical intuition: randomness. When we look at a dataset, our brains (and standard statistical software) implicitly assume that what we are seeing is a fair representation of the world.

Standard taxonomies usually group these effects by domain, e.g. separating "Sampling Bias" (Statistics) from "Malmquist Bias" (Astronomy) from "Adverse Selection" (Economics). This is useful for specialists but it encourages memorizing specific named paradoxes rather than understanding the underlying gears of the generator. I have been personally frustrated by this because selection effects are already hard enough to think about without the additional recall burden. The following is my attempt to do something about that.

If we reorganize selection effects not by where they happen, but by the causal mechanism of the filter, we find that most reduce to six distinct variants.

Instrumental Selection
----------------------

Mechanism: Thresholding

This occurs when the bias is intrinsic to the sensitivity or calibration of the observation tool. The data exists in the territory, but the map-making device has a resolution floor or a spectral limit that acts as a hard filter.

Classic Case: Malmquist Bias. In astronomy, brightness-limited surveys preferentially detect intrinsically bright objects at large distances. We aren't seeing a universe of bright stars; we are using a telescope that effectively blindfolds us to dim ones.

Social Case: Publication Bias. The academic publishing system acts as a lens that is opaque to null results. The "file drawer" is not empty because reality is exciting; it is empty because the instrument of science is calibrated to detect only "significance."

Heuristic: Ask, “What is the minimum signal strength required for my sensor to register a ping?”

Ontological Selection
---------------------

Mechanism: Classification

Occurs inside the observer's mind before a single data point is collected. It does not filter physical reality; it filters the categories we use to describe reality.

While Instrumental Selection fails to catch a fish because the net has holes, Ontological Selection captures the fish but throws it back because the fisherman has defined it as "not a fish." It gerrymanders reality by defining inconvenient data out of existence.

Economic Case: The Unemployment Rate. How do you measure a recession? In the US, the standard "U-3" unemployment rate defines an unemployed person as someone currently looking for work. If a person gives up hope and stops looking, they are no longer "unemployed", they simply vanish from the denominator. During prolonged depressions, the unemployment rate can artificially drop simply because people have become too hopeless to count.

Military Case: Signature Strikes. In certain drone warfare protocols, the definition of "enemy combatant" was arguably expanded to include any military-age male in a strike zone. By redefining the category of "civilian" to exclude the people most likely to be hit, the official data could truthfully report "zero civilian casualties" despite a high body count. The selection effect wasn't in the sensor; it was in the dictionary.

Heuristic: Ask, “If the thing I am looking for appeared in a slightly different form, would my current definition classify it as 'noise' and discard it?”

Process Selection
-----------------

Mechanism: Attrition

This occurs when a dynamic pressure destroys, removes, or hides subjects before the observer arrives. The sample is representative not of the starting population, but of the population capable of withstanding the pressure.

Classic Case: Survivorship Bias. We study the bullet holes on returning planes, ignoring that the planes hit in critical areas never returned to be counted.

Digital Case: The Muted Evidence Effect. We analyze the history of the internet based only on content that is currently hosted. Deleted tweets, shadowbanned accounts, and defunct platforms are non-observable. We are analyzing a history written exclusively by the compliant and the solvent.

Heuristic: Ask, “What killed the data points that aren't here?”

Agentic Selection
-----------------

Mechanism: Self-Sorting

Here, the filter is driven by the internal preferences, private information, or incentives of the subject. The subject decides whether to be in the sample. This is distinct from Instrumental Selection because the bias is driven by the observed, not the observer.

Classic Case: Adverse Selection. If you offer health insurance at a flat rate, the only people who "select in" are those who know they are sick. The sample (insured people) becomes virtually perfectly negatively correlated with health.

Social Case: Homophily. People self-sort into networks of similar peers. If you try to sample public opinion by looking at your friends, you are sampling a dataset defined by its similarity to you.

Heuristic: Ask, “What private information or incentive does the subject have to enter (or avoid) my sample?”

\[Note: in practice, the distinction between process selection and agentic selection can rely on locus of control considerations, which are themselves slippery. Consider the case of student's leaving a charter school that is under study. Are they self selecting out (self-sorting) or being filtered out (attrition)?\]

  
Anthropic Selection
----------------------

Mechanism: Pre-requisite Existence

Occurs when the observation is conditional on the observer possessing specific traits to be present at the site of observation. The observer and the observed are entangled.

Classic Case: The Anthropic Principle. We observe physical constants compatible with life not because they are probable, but because we could not exist to observe incompatible ones.

Statistical Case: Berkson’s Paradox. Two independent diseases appear correlated in a hospital setting. Why? Because to enter the sample (the hospital), you usually need at least one severe symptom. If you don't have Disease A, you must have Disease B to be there. The "hospital" is a condition, not a location.

Occupational Case: The Healthy Worker Effect. Occupational mortality studies often show workers are healthier than the general public. This isn't because working makes you immortal; it's because being severely ill makes you unemployed.

Heuristic: Ask, “What conditions had to be true about me for me to be standing here seeing this?”

Cybernetic Selection
--------------------

Mechanism: Recursion

This acts as a "dynamic Goodhart effect." The selection mechanism learns from the previous selection, narrowing the filter in real-time. The variance of the sample collapses over time because the filter and the filtered are in a feedback loop.

Classic Case: Algorithmic Radicalization. You click a video. The algorithm offers more of that type. You click again. The algorithm removes everything else. The "reality" presented to you becomes a hyper-niche, distilled version of your initial impulse.

Market Case: Price Bubbles. High prices select for buyers who believe prices will go higher. Their buying drives prices up, which selects for even more optimistic buyers, while "selecting out" value investors.

Heuristic: Ask, “How did my previous interaction with this system change what the system is showing me now?”  
 

\[Note: Cybernetic selection is less a distinct category rather than itself a process, any of the others can fall into it via temporal compounding\]

Some major papers in the field:
-------------------------------

Heckman, J. J. (1979). *"Sample Selection Bias as a Specification Error."* 

Bowker, G. C., & Star, S. L. (1999). *"Invisible Mediators of Action: Classification and the Ubiquity of Standards."*

Bostrom, Nick. (2002). "*Anthropic Bias: Observation Selection Effects in Science and Philosophy*." 

Hernán, M. A., Hernández-Díaz, S., & Robins, J. M. (2004). "*A Structural Approach to Selection Bias.*"

Sculley, D., et al. (2015). *"Hidden Technical Debt in Machine Learning Systems."*