For diagnostic testing, quantitative data may be less important than a qualitative lists of problems. Usability testing always includes qualitative notes. It almost always includes reporting some numbers, such as percentage of participants who had a particular problem or who took a particular path to completing a task. But it may or may not include the types of quantitative data that are traditional in psychology research, such as time and errors. That is, if you have selected representative participants, had them do realistic and representative tasks, and taken careful and objective notes on what they did, you should be able to tell with confidence what is working well on the Web site and what needs to be fixed.
Usability testing draws people from many different backgrounds.
Some people come to usability testing from doing academic psychology experiments in a very quantitative, hands-off approach. Others come to usability testing from anthropology, ethnography, sociology, and other fields that stress naturalistic, qualitative observations. In a comparison, you are usually collecting quantitative data to show the differences between two or more alternatives. Therefore, comparative testing is usually formal. For a fair comparison here, the people in the two groups must be matched on all the relevant characteristics.