Benchmarking Sales Forecasting Performance Measures (1999)

Although different companies use different measures to evaluate the performance of forecasts, mean absolute % error (MAPE) is the most popular one ... by and large, error is the lowest on an industry level, and the highest, on a location level... satisfied companies are more likely to measure forecast accuracy than others. There has been a lot of interest lately in determining benchmarks for sales forecast accuracy. However, present benchmarks are mostly aggregated across industries and do not distinguish between the ways for calculating accuracy. Consequently, present benchmarks do not offer clear benchmarks for accuracy on an industry and/or performance measurement basis. In an attempt to clarify such benchmarks, a study was undertaken by Georgia Tech's Marketing Analysis Laboratory. Like previous studies, the study intended to determine what methods companies are using to calculate forecast accuracy, what degrees of forecast accuracy are being achieved, how forecast accuracy is being reported, and what factors appear to drive forecast accuracy. However, unlike previous studies which have mostly aggregated industry and forecast measurement statistics into one set of benchmark criteria, the present study considered sales forecast accuracy on a market-specific basis, e.g., consumer product companies, and forecast performance measurement basis, e.g., accuracy achieved using mean absolute percent error. METHODOLOGY Data for the study were collected using a mail survey with follow-up telephone interviews. Eighty sales forecasting managers were surveyed. Following two survey waves, 40 managers replied and were subsequently interviewed (50% response rate). Interviews conducted with responding managers, clarified responses, and asked additional open-ended questions. Responding firms were primarily consumer product firms (75% of respondents), the remaining firms (25%) primarily served industrial markets. Respondents indicated a high degree of competition, moderate degree of technological change, and high degree of promotions being used. Respondents also indicated that their products were made-to-forecast and that the average number of forecasts made per period was 4,877 forecasts to support on average 62 distribution outlets. The average length of delivery cycle was four weeks, with a nine-week average raw material lead time, and an eight week average production lead time. The average product shelf life was 100 weeks. RESULTS Analyses comprised descriptive statistics (average, standard deviation, minimum value, maximum value), correlations, and/or crosstabulation. Below are results for the four given research questions: How is forecast accuracy measured? What level of forecast accuracy is being achieved? How is forecast accuracy reported? And what factors appear to impact forecast accuracy? How Is Forecast Accuracy Measured? Forty percent of respondents measure sales forecast error, twenty-five percent measure accuracy, twenty percent measure both accuracy and error, and fifteen percent do not measure sales forecast performance at all. Of those that measure accuracy or error, mean absolute percent error is the most popular statistic, followed by mean absolute deviation (MAD) and standard deviation. Overwhelmingly, these respondents indicated that their sales forecast accuracy/error statistic was weighted by volume. (See Tables 1 and 2) Note that in accuracy measurement, forecast accuracy is expressed as a percentage; for example, the forecast is 90% accurate. Inventory statistics included line fill and order-fill statistics. Interestingly, half of the companies using MAPE employed a formula that used forecast as the denominator, while the other half of companies employed the textbook version of the formula, which prescribes the use of actual as the denominator. (See Table 3) It should be noted that each of these two formulas can introduce a bias into the performance measurement process. …