Australian Rugby Fan’s Study Claiming South African Referees’ Bias, Shown to Be False Red Flag
By: Nik Caine
Tags: Sports Analytics, Data Analytics, Sanzaar, Super Rugby

Earlier in the year, an Australian rugby fan, Rebels3, produced a study that presumes to show how South African referees favored South African teams playing at home in the Southern Hemisphere’s Super Rugby tournament. While it is a solid preliminary study and certainly deserving of a deeper look, the conclusions drawn from it are premature. Yet, sports reporters from across the Southern Hemisphere were only too keen to publish the fan’s data, as shown in these examples:

At least one brazenly worded open-letter was sent to the CEO of SANZAAR, the body that oversees Super Rugby. The hype around the unfounded conclusions prompted Australia’s rugby franchise executives to call for an official inquiry.

All of this for a fan’s unvetted spreadsheet.

For context, Super Rugby (at least until the coronavirus pandemic) is a tournament between up to five teams each from Australia, New Zealand, and South Africa. Recently one team each from Argentina and Japan joined the tournament. Each team’s schedule is split more-or-less evenly between home and away matches. The tournament’s professional referees are are not required to be neutral, a matter at the heart of the alleged problem. The term “neutral” is being used to describe a referee officiating a match in which neither the home nor away teams are from the same country as the ref.

Original Fan-study

The table below shows how the fan-study sampled different groups of matches.

The below bar-chart shows each group’s average penalty difference, the metric used in the fan-study. The penalty difference is simply the difference between the penalties awarded against the home and away teams.

A bar-chart depicting average penalty differences in favor of the home team, for all matches from 2017 to 2019, where the home team and referee are from the same country while the away team is from a different country.

The chart shows an Australian home team with an Australian referee, on average, was penalized 0.5 fewer times than their foreign opposition between 2017 to 2019. A New Zealand home team with a compatriot referee received 0.1 fewer penalties. These two figures are in stark contrast to the fact that South African home teams refereed by a South African were penalized 3.5 fewer times than their overseas rivals. This large difference is what drives the the idea that South African referees have a demonstrable bias toward South African teams.

The fan study only considered the years 2017 to 2019, but if we expand the range to 2009 to 2019 (there is no Super Rugby penalty count data before 2009) we see a similar picture:

A bar-chart depicting average penalty differences in favor of the home team, for all matches from 2009 to 2019, where the home team and referee are from the same country while the away team is from a different country.

Again, this is a sound preliminary study, or maybe the first step of a preliminary study, and Rebels3 is clearly onto something. But, to know what that something is, one needs to dig deeper.

Fan-study Ignores How Other Referees Treat South African Teams

This is where the misunderstanding happened. People saw the dissimilarity in average penalty differences and immediately jumped to the conclusion that South African referees unfairly favor South African teams.

However, the study only makes comparisons between Australian home matches officiated by Australians, New Zealand home matches officiated by New Zealanders, and South African home matches officiated by South Africans. What it doesn’t do, is compare South African home matches officiated by different groups of referees, for example. This is a huge oversight. Obviously, we cannot say South African referees favor South African home teams if we haven’t even looked at how other referees treat South African home teams. It’s completely possible that referees from other countries favor South African home teams to a similar or even greater extent (foreshadowing).

The problem can be more formally expressed in terms of the study’s variables.