Different statistical techniques have been used for estimating interrater and test reliability. To get a better picture of the reliability of a performance assessment instrument, for example, researchers may need to obtain a variety of estimates. Unfortunately, available statistical packages either do not provide the different estimates or, if they do, it is extremely difficult and time consuming to obtain some of these indices. This is particularly true with interrater reliability estimates. The ITRS system, for the first time, makes it possible for users to obtain a variety of different reliability estimates for a given instrument.
Users can then compare the indices and select the most appropriate statistics for their case.
This part generates overall measures of rater agreement, and if needed, compares an individual rater with a group of raters for consistency.