Benchmarking Forecasting Software and Systems (2002)

With increasing awareness about the importance of forecasting, more and more companies are looking into software and systems to improve as well as automate their forecasting process. Which forecasting software or forecasting system other companies are using can help in making the right choice. Before going further, however, it is important to know the difference between a forecasting software package and a forecasting system. Forecasting software is a standalone package which is used to generate forecasts using either a model selected by the forecaster or a model selected by the expert system built in the software. The forecasting system does much more than generating forecasts. Most of the forecasting software packages available in the market have an expert system. A package with such a system has an automatic feature that tests a number of forecasting models with the historical data and then prepares a forecast for the next period with a model that, on average, yields the best results. The criteria used for selecting the best model depend on the number of models included in the program to select from and the criteria used by the vendor for selecting the best one. One expert system may have more models to choose from than others. As mentioned in the article on Benchmarking Forecasting Models, there are three types of models: Time Series, Cause-and-Effect, and Judgmental. Each one has its own tool kit for evaluating a model. In Time Series models, you may select a model by the lowest Mean Error (ME), Mean Absolute Error (MAE), MeanAbsolutePercentError (MAPE), Mean Squared Error (MSE), or any other criterion. In Regression (which is a one of several Cause- and-Effect models), you may decide to use F value, R^sup 2^, or any other diagnostic available in its tool kit. The same applies to Judgmental models, though at present, I don't know of any vendor that has an expert system based on this type of model. In fact, most of the expert systems available in the market are based just on Time Series models; that is to say, their software selects the best model only from a series of Time-Series models. The forecasting system does much more than generating forecasts. It automates the forecasting process. It automates the process for acquiring data/information from various sources and, after the data are acquired, it cleanses them. If needed, it slices and dices data (breaking up monthly data into weekly or daily buckets) or aggregates them (aggregating SKU data into categories and category data into overall totals). Additionally, it looks for problems in the data, e.g., outliers (unusual values), structural change (a permanent change in the data pattern because of merger/acquisition or for any other reason), and seasonality. Once problems are identified, it treats them as specified in the system, prepares forecasts either by the model specified by the forecaster or by using the expert system. After statistical forecasts are generated, the system sends them to all the stakeholders for their feedback. After feedback from everyone is received and incorporated, the forecasts are finalized. Then final forecasts are aggregated, disaggregated, and converted into formats that different functions and planning modules need. For example, they may have to be expressed in units and dollars; by SKUs, categories, and aggregates (company as a whole); and by regions, channels of distributions, and customers. Furthermore, they may have to be broken down into weekly, monthly, and/or quarterly buckets. The system also feeds the final forecasts to various modules for preparing plans-to the production module for preparing a production plan, to the procurement module for preparing a procurement plan, to the sales module for preparing a sales plan, to the financial module for preparing a financial plan, and so on. WHY FORECASTING SOFTWARE PACKAGES AND SYSTEMS? With the explosion in the number of SKUs (Stock Keeping Units), expansion in markets here and abroad, and increasing competition, it has become more important than ever to prepare forecasts that are not only accurate but also are available when needed by decision makers. …