Agreement Between Product Ratings Generated by Different Consumer Testing Organizations: A Statistical Comparison of Consumer Reports and Which? from 1957 to 1986

Agreement Between Product Ratings Generated by Different Consumer Testing Organizations: A Statistical Comparison of Consumer Reports and Which? from 1957 to 1986 This study constitutes a first attempt to determine the agreement existing between product quality ratings published in the leading consumer testing magazines in the United States and abroad. For many years such ratings have been guiding the purchase decisions of millions of consumers; yet, little, if anything, is known about whether the ratings provided by one magazine agree with the ratings provided by another. This question is of fundamental importance because a lack of agreement would suggest that the magazines' product ratings are not a solid foundation upon which consumers could or should base their purchase decisions. On the other hand, close agreement would suggest consistent measurement across magazines. To put the matter more precisely, this study attempts to assess the reliability of product quality ratings published in consumer testing magazines (or, in other words, the extent to which different magazines agree on their ratings of the same product). The validity of the ratings is not directly assessed (whether they do indeed measure product quality). However, neither are validity considerations completely ignored by the study in that high reliability is a necessary but not sufficient condition for high validity. Thus, an unreliable measure cannot be valid, but a reliable measure may or may not be valid. Before addressing the reliability question it is helpful to provide some background information for the study. For some years now, the two leaders in consumer product testing have been Consumers Union in the United States, which has published Consumer Reports since 1936, and the Consumers' Association in Britain, which has published Which? since 1957. Both organizations are nonprofit, and their publications, Consumer Reports and Which?, are monthly magazines featuring comparative test reports for a variety of consumer products. Both magazines rate products by brand and model, and both have large circulations (3,800,000 for Consumer Reports and 720,000 for Which?). (1) METHOD The task confronting the investigator in this study was somewhat akin to rummaging through dozens of used furniture stores in search of a few valuable antiques. Rare coincidences of circumstance were being sought--instances of tests reported in both Consumer Reports and Which? focusing on the same product category, and, more specifically, on the very same product brands and models. Also, to ensure data comparability, the tests reported had to use the same or similar evaluative dimensions to assess the common brands and models. Finally, to avoid the possibility of the evaluative data generated by one magazine affecting the evaluative data generated by the other, the tests reported in the two magazines had to be conducted independently of each other. To arrive at a study sample meeting these criteria, a series of gatekeeper steps was established. Each had the effect of reducing the potential pool of thousands of test report pairs (one member from Consumer Reports and one from Which?) to a much smaller subset of eligibles that would constitute the study sample. In 1987, copies of all issues of Consumer Reports and Which? for the years 1957 through 1986 were secured (the 30 years of overlap for the two magazines). Next, the copies were examined in an effort to find identical consumer product categories tested by the two magazines within a period of two calendar years (e.g., a test of toasters in Consumer Reports in 1974 and a test of toasters in Which? in 1976). After completing this step, each identified test pair meeting the two-year criterion for the same product category was scrutinized to see if the American and British tests had evaluated two or more of the same brand and model names. To illustrate this procedural step with the 1970s toaster example, this test pair would qualify if both its American and British members had evaluated at least two of the same brands and models, say, a Sunbeam Model 18A and a General Electric Model 6B. …