When such differences are observed, it may be possible to provide feedback or improve instructions to make raters' marginal proportions more similar and improve agreement. Several statistical approaches to this problem are available. Before discussing formal statistical methods, non-statistical methods for comparing raters' marginal distributions should be briefly mentioned. Simple descriptive methods can be very useful. For example, a table might report each raters' rate of use for each category.
Graphical methods are especially helpful.
A histogram can show the distribution of each raters' ratings across categories. Vertical or horizontal stacked-bar histograms are good ways to summarize the data. With ordered-category ratings, a related type of figure shows the cumulative proportion of cases below each rating level for each rater. These are merely examples. Nonparametric tests The main nonparametric test for assessing marginal homogeneity is the McNemar test .