One way to represent success data can be as zeros and ones. Nielsen indicates that other frameworks for coding success are available. For example, the analyst may want to record partial successes for users who complete some a task. At WebWord, we certainly agree that data and quantitative information is a useful way to report usability test results. However, the more technically inclined consumer of usability data may wonder how success rate data might be subjected to formal statistical analyses.
Statistical analyses are important for assessing the reliability of a result.
However, since usability data often consist of a small number of users, formal statistical analyses are often not powerful enough to be useful. On the other hand, repeated measures designs, where each user generates data in several conditions, are statistically powerful and efficient techniques. One statistical technique is the Cochran's Q test, which is designed to assess differences across conditions in which there is a dichotomous outcome. For usability data, users might generate success data in multiple conditions or across multiple tasks. Cochran's Q can test whether the responding across the conditions are significantly different from each other.