Evaluation of inter-observer reliability in the case of trichotomous and four-level animal-based welfare indicators with two observers
Creators: name: Torsiello, Benedetta
name: Giammarino, Mauro
name: Quatto, Piero
name: Battini, Monica
name: Mattiello, Silvana
name: Battaglini, Luca
name: Renna, Manuela
Publisher: name: Taylor & Francis
Year: 2024
Subjects: subject: Biochemistry
subject: Environmental Sciences not elsewhere classified
subject: Ecology
subject: FOS: Biological sciences
subject: Sociology
subject: FOS: Sociology
subject: Inorganic Chemistry
subject: FOS: Chemical sciences
subject: Science Policy

description: This study focuses on assessing inter-observer reliability (IOR) between two observers in the case of trichotomous and four-level animal-based welfare indicators assessed at individual level. The Body Condition Score (BCS) and Knee calluses (KNC) were chosen as trichotomous indicators; data were collected in fourteen intensively managed dairy goat farms in Italy (ITF1 to ITF7) and Portugal (PTF1 to PTF7) and in extensively managed dairy goat farms exploiting three alpine pastures (AP1, AP2 and AP3) in Italy. The Ear posture (EP) and Eye white (EW) were chosen as four-level indicators; data were collected in three intensively managed dairy cattle farms (F1, F2 and F3) in Italy. The performance of the most documented agreement indices was compared. In the case of trichotomous indicators, Scott’s <i>π</i>, Cohen’s <i>K</i>, Cohen’s <i>K</i><sub>C</sub>, Cohen’s <i>weighted K</i> and Krippendorff’s <i>α</i> were affected by the paradox effect: when the concordance rate (P<sub>0</sub>) was high, they sometimes gave very low or even negative values (e.g. P<sub>0(BCS-ITF3)</sub> = 74%; Scott’s <i>π</i> = 0.05; Cohen’s <i>K</i> = 0.09; Krippendorff’s <i>α</i> = 0.06; P<sub>0(BCS-AP3)</sub> = 74%; Scott’s <i>π</i> = −0.12; Cohen’s <i>K</i> = Krippendorff’s <i>α</i> = −0.11). Bangdiwala’s <i>B</i>, Gwet’s <i>γ(AC<sub>1</sub>)</i> and Quatto’s <i>weighted S</i> were not affected by this phenomenon and provided values very close to P<sub>0</sub> (e.g. P<sub>0(KNC-PTF1)</sub> = 88%; Bangdiwala’s <i>B</i> = Gwet’s <i>γ(AC<sub>1</sub>)</i> = 0.85; P<sub>0(BCS-AP1)</sub> = 82%; Bangdiwala’s <i>B</i> = Gwet’s <i>γ(AC<sub>1</sub>)</i> = 0.79). In the case of four-level indicators, Cohen’s <i>K</i> and Krippendorff’s <i>α</i> were not affected by the paradox behaviour. However, Cohen’s <i>K</i><sub>C</sub> in some cases exceeded the observed P<sub>0</sub> (e.g. P<sub>0(EP-F3)</sub> = 78%; Cohen’s <i>K</i><sub>C</sub> = 1). Gwet’s <i>γ(AC<sub>1</sub>)</i> showed the best results for four-level indicators (e.g. P<sub>0(EP-F1)</sub> = 88%; Gwet’s <i>γ(AC<sub>1</sub>)</i> = 0.86), followed by Quatto’s <i>S</i> and Holley and Guilford’s <i>G</i> (e.g. P<sub>0(EP-F1)</sub> = 88%; Quatto’s <i>S</i> = Holley and Guilford’s <i>G</i> = 0.84). To evaluate IOR between two observers, Bangdiwala’s <i>B</i>, Gwet’s <i>γ(AC<sub>1</sub>)</i> and Quatto’s <i>weighted S</i> are suggested for trichotomous indicators, while Gwet’s <i>γ(AC<sub>1</sub>)</i>, Quatto’s <i>S</i> and Holley and Guilford’s <i>G</i> are suggested for four-level indicators. Scott’s <i>π</i>, Cohen’s <i>K</i>, Cohen’s <i>K</i><sub>C</sub>, Cohen’s <i>weighted K</i> and Krippendorff’s <i>α</i> can be affected by a paradox behaviour. Bangdiwala’s <i>B</i>, Gwet’s <i>γ(AC<sub>1</sub>)</i> and Quatto’s <i>weighted S</i> are suggested to evaluate IOR between two observers for trichotomous indicators. Gwet’s <i>γ(AC<sub>1</sub>)</i>, Quatto’s <i>S</i> and Holley and Guilford’s <i>G</i> are suggested to evaluate IOR between two observers for four-level indicators.