The Reliability of Subjective Well-Being Measures

Economists are increasingly analyzing data on subjective well-being (SWB). From 2000 to 2006, 157 papers and numerous books have been published in the economics literature using data on life satisfaction or subjective well-being, according to a search of Econ Lit.1 Data on life satisfaction or happiness have been used as outcome measures in studies of the tradeoff between inflation and unemployment, the effect of cigarette taxes on welfare, the effect of German reunification on well-being, and the effect of lottery winnings on well-being.2 In addition, life and work satisfaction measures have appeared as explanatory variables in studies of labor turnover, productivity and health.3 If it could be measured accurately, or even approximately, subjective well-being is a natural variable for economists to model and understand because utility maximization is a central idea in economics, from either a normative or positive perspective. Here we analyze the test-retest reliability of two types of measures of subjective well-being: a standard life satisfaction question and affective experience measures derived from the Day Reconstruction Method (Kahneman et al., 2004). Although economists have longstanding reservations about the feasibility of interpersonal comparisons of utility that we can only partially address here, another question concerns the persistence of subjective well-being measurements for the same set of individuals over time. Absent dramatic events, overall life satisfaction should not change much from week to week. Likewise, individuals who have similar routines from week to week should experience similar feelings over time. How persistent are individuals’ responses to subjective well-being questions? To anticipate our main findings, both measures of subjective well-being (life satisfaction and affective experience) display a serial correlation of about 0.60 when assessed two weeks apart, which is lower than the reliability ratios typically found for education, income and many other common micro economic variables (Bound, Brown, and Mathiowetz, 2001 and Angrist and Krueger, 1999). If measurement errors are white noise, a reliability ratio of 0.60 implies substantial attenuation if the variable is used as an explanatory variable in a regression. Measurement error when subjective well-being is used as a dependent variable would imply a loss of precision in resulting estimates. Nonetheless, the estimated degree of reliability of subjective well-being data is probably high enough to detect effects when they are present in most applications, especially if samples are large and the data are aggregated across people or activities. The life satisfaction question that we examine is nearly identical to that used in the World Values Survey, and similar to that used in many other well being surveys. There is reason to expect, however, that life satisfaction measures such as this may not be as stable from week to week as might be assumed. Rather, these judgments are the result of a complex thought experiment which is often partially dependent on transient influences (e.g. one’s mood at the time; see Schwarz and Strack, 1999). For measurements of the affective experience of daily life the gold standard is perhaps the Experience Sampling Method (ESM) (also called Ecological Momentary Assessment (EMA)), in which participants are prompted at random intervals to record their current circumstances and feelings (Csikszentmihalyi & Larsen, 1987; Stone, Shiffman & DeVries, 1999). This method of measuring affect minimizes the role of memory and interpretation, but it is expensive and difficult to implement in large samples. The Day Reconstruction Method (DRM) is a recent development in the measurement of affective experience, which reduces the cost of obtaining this information. Consequently, we use the DRM, in which participants are requested to think about the preceding day, break it up into episodes, and describe each episode by selecting from several menus (Kahneman, et al., 2004). The DRM involves memory, but is designed to increase the accuracy of emotional recall by inducing retrieval of the specifics of successive episodes (Robinson & Clore, 2002; Belli, 1998). Evidence that the two methods can be expected to yield similar results was presented earlier for subpopulation averages (Kahneman, et al., 2004). A critical advantage of the DRM is that it provides data on time-use – a valuable source of information in its own right, which has rarely been combined with the study of subjective well-being. In this paper we report reliability measures for a sample of 229 employed women who each filled out a DRM questionnaire for two Wednesdays, two weeks apart in 2005. We compare these reliability estimates to those of global well-being measures more typical in the literature, and we decompose the reliability of duration-weighted net affect into a component due to the similarity of activities across days and other factors. We also provide an application using the reliability estimates to correct observed correlations between self-reported well-being and other variables (e.g., income) for attenuation. We conclude with a discussion of the implications of measurement error for DRM studies and for well-being research more generally.