Probability, Justice, and the Risk of Wrongful Conviction

Background and ContextI am a professor of statistics, and most of my work is fairly technical and mathematical (see www.probability.ca/jeff/research.html). But one day I wrote a book, Struck by Lightning: The Curions World of Probabilities, for the general public, which did unexpectedly well, especially in Canada. I was then interviewed by the media about such diverse topics as lottery probabilities, public opinion polls, violent crime rates, sports statistics, and more, and was even involved in probing a major lottery retailer fraud scandal involving millions of dollars and criminal convictions (see, for example, www.probability.ca/lotteryscandal). I was also invited to give talks to all sorts of different groups, from insurance brokers to financial investors, from humour therapists to gambling addiction counselors.And then one day I was invited to speak to a prominent group of Canadian lawyers and judges. This invitation in turn led to others, and I ended up giving five different talks to five different groups of lawyers and judges (including the Irish Supreme Court and High Court justices) within a single year. This forced me to investigate the connection of probabilities and statistical analysis to the justice system, as I will now discuss.Probability and Justice?What is the connection of probability and statistics to justice issues? Well, both topics involve evaluating evidence, i.e., determining whether the available infonnation is sufficient to draw certain conclusions. This perspective was nicely summarised by the James Bond villain Auric Goldfinger when he stated, "Once is happenstance. Twice is coincidence. The third time it's enemy action."In probability and statistics, the possible conclusions might involve medical testing, or psychological analysis, or social science studies. We might try to determine if an observed difference is "statistically significant", or if a certain probability is above some threshold, or if a corresponding pvalue is less than some cutoff. In the justice system, the possible conclusions involve such questions as guilt or innocence. A court is then tasked with determining if a case has been proven "beyond a reasonable doubt", or (for civil cases) by a "balance of probabilities" or a "preponderance of the evidence". So what do these terms mean, and how are they related?The justice term "balance of probabilities" seems closest to the world of probability and statistics. It seems to mean that a certain conclusion is more likely than not. So perhaps that means simply that the probability has to be more than 50%? Unfortunately even this connection is not so clearcut. A classic example involves 1,000 people attending an event at which only 499 admission fares were collected. This means that a randomly-chosen attendee has probability 50.1% of not having paid admission. But surely no judge would award damages against a randomly-chosen attendee on that basis. Thus, even the simple-seeming "balance of probabilities" standard requires human judgement and cannot be interpreted in purely probabilistic terms.The phrase "beyond a reasonable doubt" is even more challenging. It is generally agreed to mean something weaker than "certainty", but something stronger than just "probably". (For example, the Ireland Director of Public Prosecutions web site states: "The judge or jury has to be convinced beyond a reasonable doubt that a person is guilty. It is not enough for them to think that the accused is probably guilty.") So does this mean the probability of guilt has to be more than 95%? more than 99%? more than 99.9%? Or that the corresponding p-value (i.e., the probability that we would have observed such evidence even if the accused were innocent) must be less than 5% or 1% or 0.1%? Once again, there is no clear standard, and the correspondence between probability/statistics and the standards of the justice system are hard to pin down.Nevertheless, despite these challenges, it does seem that justice issues should be somewhat analysable in tenus of probabilities and statistics. …