Implementing Quality Improvement Programs Designed to Enhance Customer Satisfaction : Quasi-experiments in the U . S . and Spain

We describe two related quasi-experiments, one in the United States and one in Spain, in which a sophisticated, high-technology firm designed and implemented customer-satisfaction improvement programs. Voice-of-the-customer measurements and “House-of-Quality” techniques were used to design intervention programs to improve satisfaction with five targeted customer needs. Although the interventions implemented in the two countries differed in some respects, both interventions were targeted at the same five needs and the same type of business-to-business customers. In each country, the programs were implemented in “treatment” regions, but not in “control” regions and the firm collected pretest and posttest satisfaction measures for targeted and non-targeted needs. An analysis of these measures reveals that the intervention had a significant impact on satisfaction with the targeted needs in both countries. The data also reveal a complex and, in many respects, surprising picture. While the interventions were able to effect significant, enduring improvements in satisfaction with the targeted needs, several natural assumptions failed. First, although the firm believed ex ante that the interventions were similar, seemingly inconsequential differences in empowerment between the Spanish and U.S. interventions appear ex post to be important. Second, despite the use of state-of-the-art methods to identify customer needs, overall satisfaction responded significantly to effects that were not captured by the measured needs. Third, despite the careful selection of control regions, there were unobserved ecological impacts on satisfaction which could only be accounted for with a nonequivalent-dependent-variables design. Such designs are rare in industry. The absence of such controls in typical industry studies may explain the growing concern among industry commentators that quality interventions do not yield their anticipated outcomes. Enhancing Customer Satisfaction with Quality Improvements Many literatures within marketing seek to understand and develop means to enhance customer satisfaction. The marketing research literature has developed a variety of methods to identify and prioritize customer needs to focus managerial effort; the product-quality literature proposes methods to link product or service improvements to customer needs in order to design interventions to enhance satisfaction with respect to those needs; the service-quality literature focuses on measuring changes in service quality; the customer-satisfaction literature seeks to establish the means by which customer satisfaction can be improved; and the return-on-quality literature seeks to determine whether such improvements lead to increased profits. While these literatures vary in their analyses and definitions of constructs, they generally agree that if a firm were to improve its products and services in order to fulfill important customer needs, then that firm would enhance its customers’ satisfaction and its future profits (Anderson and Sullivan 1993, Fornell 1992; Hauser, Simester and Wernerfelt 1994, 1996, 1997; Rust, Zahorik, and Keiningham 1995; Zeithaml, Parasuraman and Berry 1990). Such prescriptions are common in textbooks and monographs and are supported by several excellent laboratory studies and many compelling anecdotes of product and service quality successes. However, there are few published field experiments (or quasi-experiments) designed to test whether firms can implement quality improvement interventions that lead to measurable improvements in customer satisfaction. One notable exception is Bolton and Drew’s (1991) description of GTE’s attempt to improve telephone service. Widespread acceptance of the relationship is evident in the growing popular literature on quality, and the reliance on customer satisfaction measures in new product development and employee compensation (Anderson, Fornell and Lehman, 1994). In contrast, despite initially accepting the wisdom of implementing new quality interventions, firms are now beginning to demand explicit justification for their investments. Consulting firms and industry commentators are encouraging this trend with a series of Customer Satisfaction Quasi-experiments in the U.S. and Spain Page 2 studies questioning the benefits of quality interventions (Howe, Geaddert, and Howe 1995; Rust, Zahorik, and Keiningham 1995): “A study by the American Quality Foundation and Ernst & Young suggests that many companies are wasting their efforts in trying to improve quality. The consulting firms of AT Kerney and Arthur D. Little present equally disappointing findings in two separate studies: (1) 80% of more than 100 British firms reported ‘no significant impact as a result of TQM’ and (2) almost two-thirds of 500 U.S. companies saw ‘zero competitive gains’. ” Anderson, Fornell and Lehman (1994, p. 53) A scientific evaluation of an actual industrial program provides an important contribution to this debate. In this paper we describe two quasi-experiments undertaken by a technically sophisticated, $2 billion, international firm seeking to evaluate a new quality-and-customer-satisfaction intervention. For this paper we disguise the firm by calling it KemTek. The intervention began when a CEO-led task force determined that enhancing customer satisfaction was key to the firm’s survival. The task force commissioned state-of-the-art marketing research to determine important customer needs and adopted widely-used quality tools to design a customer-satisfaction improvement program to improve its products and services on these important customer needs. It hoped the improvements would enhance customer satisfaction and, hence, long-term profitability in its business-to-business market. The implementation is unique from the perspective of field research because this firm, with its science-oriented culture, sought to measure the impact of the program by (1) designing parallel quasiexperiments in two countries and (2) investing approximately two years and $500,000 in data collection to measure the impact of the program. Given the significant capital investment required to implement similar programs in all of its divisions, the firm sought to determine whether this investment was justified. By describing the development and implementation of the firm’s intervention and by providing data on the outcome of the quasi-experiments we seek to add insight to the relevant scientific literatures within marketing and to provide a window on the implementation of one large-scale, market-driven, customersatisfaction intervention in industry. Customer Satisfaction Quasi-experiments in the U.S. and Spain Page 3 The study confirms the basic premise: KemTek’s efforts led to measurable and enduring improvements in satisfaction with the targeted needs. Beyond this, however, the data contain a number of surprises. At least three commonly held beliefs are questioned. First, while we (and KemTek) believed ex ante that the treatments in the two countries, the U.S. and Spain, were equivalent, this appears not to have been the case. Seemingly minor and inconsequential differences in the manner in which employees were empowered appear to have been quite important. Second, while we used state-of-the-art methods to elicit customer attributes (and evidence suggests that the list of attributes was fairly exhaustive), some of the improvement in overall satisfaction was due to factors not captured in attribute satisfaction. This suggests the need for academic research to improve our measurement (and understanding) of the determinants of overall satisfaction. Third, in spite of relatively large sample sizes, the nonequivalent treatment and control groups were not sufficient to identify significant effects in both quasi-experiments. Fortunately, we had data available to incorporate a “nonequivalent-dependent-variables” design with the standard “pretest/posttest untreated control group” design to enhance the power of the analysis (Cook and Campbell, 1979, pp. 249, 261). Together these three lessons may help explain why industry commentators and firms themselves are starting to question the wisdom of quality interventions. If these three effects are not taken into account, industry might be led to erroneous conclusions that some quality and customer-satisfaction interventions do not work when a more complete analysis might suggest otherwise. Quasi-experimental Design and Measures KemTek implemented a quasi-experimental design that included an extensive array of controls, including use of: 1. Pretest and posttest measures from the same panel of customers 2. Nonequivalent control groups 3. Nonequivalent dependent variables 4. Replication in separate countries (chosen by KemTek to vary in the amount of competition) Using Cook and Campbell’s (1979) notation, we depict the design in each country as follows: Customer Satisfaction Quasi-experiments in the U.S. and Spain Page 4 O1A X O2A O1B O2B O1A O2A O1B O2B Measurement activities are denoted by O, treatment activities are denoted by X, and the dashed line distinguishes between treatment and control regions. That is, KemTek chose a set of treatment regions in each country (above the dashed line) and implemented the customer-satisfaction improvement programs to every customer that they could reach in the treatment regions. KemTek did not implement the program, nor undertake any special activities to improve customer satisfaction, in the control regions (below the dashed line). Identical measurement instruments (O’s) were fielded prior to (O1) and following the intervention (O2). The measurement instruments included two sets of dependent variable scales, one of which was expected to change because of the treatment (OA) and one which was not expected to change (OB). (For a detailed discussion of the use of two related but different sets of dependent variables see Cook and Campbell 1979, p. 261). The design was replicated