Silviase/jgov_v1.4_full_ann huggingface dataset:
To answer this question, this paper evaluates the rationality and usefulness of business cycle phases predictions by professional forecasters through directional analysis. Since the pioneering work of Henriksson and Merton (1981) and the adaptation into macroeconomic forecast evaluation by Schnader and Stekler (1990) and Stekler (1994), directional analysis has been adapted to several macroeconomic variables. As a recent example, Ash et al (1998) tested 20 variables $ ^{2} $ of 7 countries in the OECD Economic Outlook, Pons (2000) compared GDP growth forecasts of G7 by IMF and OECD, Ashiya (2003) tested 15-months-ahead forecasts made by the IMF, Sinclair et al (2010) jointly evaluated the direction of change predictions of GDP growth and inflation by the Federal Reserve. As for the Japanese economy, Ashiya (2006) tested 16-month-ahead forecasts by Japanese institutional forecasters, Tsuchiya (2013) compared GDP growth and Inflation forecasts by the Japanese government and the IMF. But studies of directional analyses have rarely focused on prediction of business cycle phases by professional forecaster.

Two possible reasons are apparent. One concerns the forecast data. Forecasts by government, international organizations and professional forecasters tend to be quantitative in nature, such as GDP growth and inflation. The second reason focuses on the actual data. There are not so many economic turning points. For example, there are only 29 economic turning points of Japan after World War II and forecasts are not necessarily available in each business cycle phases.

To address this gap, we use a coincident diffusion index(DI) as actual data in the Indexes of Business Conditions released monthly by ESRI, Cabinet Office, government of Japan. DI are used to determine turning points of the business cycle by counting changes in directions of selected indicators. Whether coincident DI exceeds 50 or not allegedly indicates whether the Japanese economy is in a phase of expansion or contraction.

As forecast data, we use coincident DI forecast by Japanese professional in the ESP Forecast Survey conducted by Japan Center for Economic Research(JCER). Through directional analysis, we examine whether business cycle phases predictions by professional forecasters are useful. Also, we adapt directional analysis to GDP growth forecast in the ESP Forecast Survey(ESPF) and leading DI(LDI) in the Indexes of Business Conditions (IBC) to compare the rationality and usefulness with DI forecast.

The next section explains the statistical methods of directional analysis. Section 3 describes the data used in this paper. We show the results in Section 4. Finally in Section 5, we make concluding remarks and statement on further works.
## 2 Statistical Method

### 2.1 $ 2 \times 2 $ Contingency Table

When the coincident DI forecasts and LDI are evaluated, they are grouped into two categories: a coincident DI over 50 or under 50. Because coincident DI over 50 indicates a business condition is positive (expansion), and a coincident DI under 50 indicates a business condition is negative (contraction). As for GDP growth forecast evaluation, they are grouped into whether growth rate is positive or negative (including zero growth).

A $ 2 \times 2 $ contingency table is created that compares the forecasts and actual data (Table1). Let $ x_{t} $ be the predictor of a macroeconomic variable $ y_{t} $ , where subscript t denotes the time period in which the outcome occurs. We have a total of N observations where for $ n_{1} $ of them both the actual and forecasts are positive and for $ n_{2} $ actual are negative (or flat) and forecasts are positive. We have n observations where forecasts are positive and N - n observations where forecasts are negative (or flat). We also have $ N_{1} $ observations where the actual is positive and $ N_{2} = N - N_{1} $ observations where the actual is negative (or flat).

Eqs. $ (1) $ defines the probability of a directionally correct forecast conditional upon the actual being positive. Eqs. $ (2) $ defines the probability of a directionally correct forecast conditional when the actual is negative(including flat).

$$ \begin{aligned}P_{1}\quad&=\quad Prob(x_{t}y_{t}>0|y_{t}>0)=\frac{n_{1}}{N_{1}}\end{aligned} $$

$$ \begin{aligned}P_{2}\quad&=\quad Prob(x_{t}y_{t}>0|y_{t}\leq0)=\frac{N_{2}-n_{2}}{N_{2}}\end{aligned} $$

### 2.2 Test statistics

We use the following three statistical methodologies to show whether coincident DI forecasts predict the direction of change in the actual coincident DI; Fisher's Exact test (FE test), Pesaran and Timmerman's (1992) test and Pesaran and Timmerman's (2009) test.

#### 2.2.1 Fisher's Exact Test

Merton(1981) shows that a necessary and sufficient condition for the forecast to be rational is that $ P_{1} + P_{2} \geq 1 $ . It also shows that a necessary and sufficient condition for the forecast to have no value is that $ P_{1} + P_{2} = 1 $ . It follows that the null hypothesis is $ H_{0}: P_{1} + P_{2} = 1 $ for non-parametric tests for forecast value. Alternative hypothesis is $ H_{1}: P_{1} + P_{2} > 1 $ . R. A. Fisher's Exact Test is identical to Henriksson and Merton's test for predictive value.

#### 2.2.2 Pesaran and Timmerman's(1992) Test

Pesaran and Timmerman (1992) test for a significant difference between the observed, sample estimate of the probability of a correctly signed forecast, and
the estimate of what that probability would be under the null of independence between forecasts and outcomes. Their test static is given by Eqs. $ (3) $

$$ S_{n}^{2}=\frac{(\hat{p}-\hat{p}_{*})^{2}}{var(\hat{p})-var(\hat{p}_{*})}\sim\chi^{2}(1) $$

where

$$ var(\hat{p})=\frac{1}{N}[\hat{p}_{*}(1-\hat{p}_{*})] $$

and

$$ \begin{aligned}var(\hat{p}_{*})&=\frac{1}{N}[(2\hat{p}_{y}-1)^{2}\hat{p}_{x}(1-\hat{p}_{x})\\&+(2\hat{p}_{x}-1)^{2}\hat{p}_{y}(1-\hat{p}_{y})\\&+4\hat{p}_{y}\hat{p}_{x}(1-\hat{p}_{y})(1-\hat{p}_{x})/N]\end{aligned} $$

$ \hat{p}_{x} = Prob(x_{t} > 0), \hat{p}_{y} = Prob(y_{t} > 0). \hat{p} = Prob(x_{t}y_{t} > 0) $ means that the sample estimate of the probability of a correctly signed forecast. $ \hat{p}_{*} = \hat{p}_{x}\hat{p}_{y} + (1 - \hat{p}_{x})(1 - \hat{p}_{y}) $ means that the probability of a correctly signed forecast under the null that forecasts and outcomes are independent.

#### 2.2.3 Pesaran and Timmerman's(2009) Test

Pesaran and Timmerman (2009) test for dependence among multi-category variables, allowing for serial dependency. Their test static is given by Eqs.(4)

$$ N e w P T=(T-1)\times(\mathbf{S}_{\mathbf{y}\mathbf{y},\mathbf{w}}^{-\mathbf{1}}\mathbf{S}_{\mathbf{y}\mathbf{x},\mathbf{w}}\mathbf{S}_{\mathbf{x}\mathbf{x},\mathbf{w}}^{-\mathbf{1}}\mathbf{S}_{\mathbf{x}\mathbf{y},\mathbf{w}})\sim\chi^{2}(1) $$

where

$$ \begin{aligned}&\mathbf{S}_{\mathbf{y}\mathbf{y},\mathbf{w}}=(T-1)^{-1}\mathbf{Y}^{\prime}\mathbf{M}_{\mathbf{w}}\mathbf{Y},\mathbf{S}_{\mathbf{y}\mathbf{x},\mathbf{w}}=(T-1)^{-1}\mathbf{Y}^{\prime}\mathbf{M}_{\mathbf{w}}\mathbf{X}\\&\mathbf{S}_{\mathbf{x}\mathbf{x},\mathbf{w}}=(T-1)^{-1}\mathbf{X}^{\prime}\mathbf{M}_{\mathbf{w}}\mathbf{X},\mathbf{S}_{\mathbf{x}\mathbf{y},\mathbf{w}}=(T-1)^{-1}\mathbf{X}^{\prime}\mathbf{M}_{\mathbf{w}}\mathbf{Y}\\&\mathbf{M}_{\mathbf{w}}=\mathbf{I}_{\mathbf{T}-\mathbf{1}}-\mathbf{W}(\mathbf{W}^{\prime}\mathbf{W})^{-1}\mathbf{W}^{\prime},\mathbf{W}=(\tau_{T-1},\mathbf{X}_{-\mathbf{1}},\mathbf{Y}_{-\mathbf{1}})\\ \end{aligned} $$

X is the actual data, Y is the forecast data. $ \mathbf{X} = (X_{2}, \cdots, X_{T})' $ , $ \mathbf{Y} = (Y_{2}, \cdots, Y_{T})' $ , $ \mathbf{X}_{-1} = (X_{1}, \cdots, X_{T-1})' $ , $ \mathbf{Y}_{-1} = (Y_{1}, \cdots, Y_{T-1})' $ , and $ \tau_{T-1} $ is a $ (T-1) \times 1 $ vector of ones.
## 3 Data

### 3.1 Actual Data

#### 3.1.1 Indexes of Business Condition

IBC are summary measures for aggregate economic activity. They are indexes that combine the behavior of key cyclical indicators that represent widely differing activities of the economy such as production, employment, and many more. There are two types of indicators: composite indexes (CI) and DI.

From May 2008, ESRI has changed its main indicator to CI instead of DI. CI are used to identify the volume of overall business activities by composing percentage changes of selected indicators. In this article, however, we use DI because it is comparable with the forecast data. Both CI and DI have leading, coincident and lagging indexes. We use coincident DI as the actual business condition. Coincident DI has been released monthly, but forecast data is quarterly. So we changed the actual monthly coincident DI into quarterly averaged data.

We use real-time data and historical data as regards the actual data. We collect actual data from the preliminary monthly issues via ESRI. For example, for the second quarter (2Q) of 2013, we see the preliminary issues of June 2013 published in August 2013 and coincident DI of April, May, and June in the issue are averaged. We collect historical data from the preliminary monthly issues published in August 2013.

We cover the actual data from 1Q of 2004 to 2Q of 2013, which consists of 38 observations.

#### 3.1.2 Quarterly Estimates of GDP

We also use real-time data and historical data of real GDP growth rate. We collect quarterly growth rate of GDP from issues of the first preliminary Quarterly Estimates of GDP (QE) published by ESRI. We collect historical data from issues of the first preliminary QE published in August 2013. We cover the actual data from 1Q of 2004 to 2Q of 2013, which consists of 38 observations.

### 3.2 Forecast Data

#### 3.2.1 The ESP Forecast Survey

We use data of the ESPF conducted by the Association for Economic Planning from April 2004 and taken over by the Japan Center for Economic Research from April 2012. The monthly survey aims to obtain consensus on perspectives of the Japanese economy, such as real GDP growth, Consumer Price Index (CPI), and unemployment rate. The respondents on average more than 40 forecasters deserved to be called "professional", since they make some forecasts on the Japanese economy at their occupation.
Each forecaster is required to submit the questionnaire response during the first week, and the compiled survey results are published around 10-th day every month on line (http://www.jcer.or.jp/esp/index.html). The summary is available for free, but the full text is limitedly accessible to dues paying members. All the information is only published in Japanese, as of today.

Table2 shows an example of data release schedule. At the first week of August of 2013, the ESPF target for 2Q of 2013 was released. At that time, the first preliminary GDP and IBC of 2Q of 2013 had not yet been released. However, Indices of Industrial Production(IIP) of 2Q of 2013 were released at the end of July of 2013. We can assume the direction of GDP growth rate and IBC from the direction of IIP growth rate. That is why we call this ESPF a "0 month ahead" forecast $ (f_{t}) $ .

From this survey, we collect consensus DI forecast data, sougou keiki handan DI in Japanese. In this survey, each forecaster is required to choose from three options regarding business condition DI forecasts.

1. Expansion means coincident DI is predicted to be over 50.

2. Flat means coincident DI is predicted to be around 50 or difficult to judge.

3. Contraction means coincident DI is predicted to be below 50.

Consensus DI forecast data is calculated using the total of the ratio of Expansion responses and the half of the ratio of Flat responses.

From 1Q of 2004 to 2Q of 2013, we collect calculated consensus data $ ^{3} $ from monthly issues of the ESP Forecast survey. The number of observations is 427 $ ^{4} $ . We also use GDP growth consensus forecasts from 1Q of 2004 to 2Q of 2013. The number of observations is 434.

#### 3.2.2 Stylized Facts of Forecasts

Consensus DI forecasts tend to indicate expansion. The number of observations that consensus coincident over-50 DI forecast is 357, or 83.6% of all samples(Table 3). This ratio is larger than those of the actual real-time data (25 in 38, 65.8% of all samples) and historical data (24 in 38, 63.2% of all samples).

Judgment of turning point prediction by consensus forecasts is earlier than the assessment of the actual coincident index. As for contraction from 1Q of 2008, consensus forecast for 1Q of 2008 became below 50 from $ f_{t-3} $ (released February 2008), and consensus forecast for 2Q of 2008 became below 50 from $ f_{t-4} $ (released April 2008). We can judge contraction if coincident DI is below 50 for two quarter terms in a row, so consensus forecast is regarded to indicate contraction from 1Q of 2008 at April 2008. This is four months earlier than the assessment of the actual coincident index $ ^{5} $ .
As for expansion from 2Q of 2009, consensus forecast for 2Q of 2009 became over 50 from $ f_{t-3} $ (released May 2009), and consensus forecast for 3Q of 2009 became over 50 from $ f_{t-6} $ (released May 2009). So consensus forecast is regarded to indicate expansion from 2Q of 2009 at May 2009. This is seven months earlier than the assessment of the actual coincident index $ ^{6} $ .

As for contraction from 2Q of 2012, consensus forecast for 2Q of 2012 had not become below 50 even $ f_{t} $ (released August 2012), consensus forecast for 3Q of 2012 became below 50 from $ f_{t-2} $ (released September 2012), and consensus forecast for 3Q of 2012 became below 50 from $ f_{t-4} $ (released October 2012). So consensus forecast is regarded to indicate contraction from 2Q of 2012 at October 2012. This is two months earlier than the assessment of the actual coincident index $ ^{7} $ .

Consensus GDP growth forecasts are tend to indicate expansion. The number of observations in which consensus GDP growth forecasts are positive is 390, 89.9% of all samples(Table4). This ratio is larger than that of the actual real-time data(24 in 38, 63.2% of all samples) and historical data(25 in 38, 65.8% of all samples).

#### 3.2.3 LDI of Indexes of Business Condition

To compare the usefulness with ESPF, we also use LDI in IBC. We collect historical data from the preliminary monthly issues published in August 2013 and process monthly data on a quarterly basis as below.

$$ f_{t-i}of TQYY=(LDI_{3T-i}+LDI_{3T-i-1}+LDI_{3T-i-2})/3 $$

For example

$$ f_{t-1}o f2Q13=(L D I_{M a y2013}+L D I_{A p r2013}+L D I_{M a r2013})/3 $$

## 4 Results

### 4.1 Performance of Coincident DI

First we investigate the directional accuracy of the consensus forecast of coincident DI. Table 5 shows the results with real-time data and Table 6 shows the results with historical data. Accuracy rates are calculated using the total of the ratio of both forecasts and actual are over 50 or below 50. There are no significant difference between results with real-time data and result with historical data. The test statistics are significant with horizons of up to four months. The consensus DI forecast with horizons of four months is a useful predictor of the actual coincident DI.

$ ^{6} $ IBC of October 2009 (released December 2009) changed the assessment of the coincident index from "Signaling a possible turning point" to "Improving."
### 4.2 Comparing GDP Growth Forecasts

We also test the directional accuracy of the GDP growth forecasts in ESPF. Table 7 shows the results with real-time data and Table 8 shows the results with historical data. The test statistics are significant with horizons of up to three months for the results with real-time data. As for the results with historical data, PT and New PT test statistics are significant with horizons of up to three months. However, the FE statistics are not significant with horizons of two months. The consensus forecast is more useful to predict economic turning points than to predict GDP growth rates. GDP growth forecasts are also vulnerable to data vintage.

### 4.3 Comparing with LDI

We now turn to the directional accuracy of the LDI in the Indexes of Business Conditions. Table 9 shows the results. FE and PT test statistics are significant with horizons of two months. New PT test statistics are significant with horizons of only one month. The consensus DI forecast is more useful than LDI.

## 5 Conclusions

We used directional analysis to investigate the accuracy of coincident DI forecasts using ESPF. Our findings indicate business cycle phases predictions with horizons of up to four months are useful. The prediction is more useful than LDI in IBC. This finding suggests predictability of LDI can be improved with the aid of coincident DI forecasts as Chua and Tsiaplias (2011) adapted to the US economy. The consensus forecast is more useful to predict economic turning points than to predict GDP growth rates.

On the other hand, we have room for improving the accuracy of consensus DI forecasts through analyzing individual forecaster's prediction. However, as we show in the appendix, we have some problem in handling their prediction because there are many flat answers. In this survey flat responds have two meanings: one is that coincident DI is predicted to be around 50 and the other it is difficult to judge. We must conduct further studies on flat answers.

## 6 Appendix

From 1Q of 2004 to 3Q of 2012, we collect responses from forecasters in ESPF's sougou keiki handan DI (in Japanese). Here we call forecasts of t with i months ahead by forecaster j, $ f_{t,t-i}^{j} $ . For example, forecasts of 3Q of 2012 are obtained from November 2012 ( $ f_{3Q12,t}^{j} $ ) to December 2011 ( $ f_{3Q12,t-11}^{j} $ ). From each forecasters, we can also obtain 393 responses. We exclude forecasters whose participation rates are less than 70% (i.e., whose number of forecast sets is less than 275), leaving 32 forecasters. The total number of forecast sets from $ f_{t,t}^{j} $ to $ f_{t,t-11}^{j} $ is 11682; the average number of observations per forecasters is 365.1.
Individual forecasts tend to respond flat. Average ratio of response by each forecasters is 20 – 30% and the Max ratio is around 50%(Table10).

We investigate the directional accuracy of individual forecasters responses with real-time data through the Pesaran and Timmerman (1992) test. Tables 11 and 12 show the results with all samples and the result with samples excluding flat. There are significant differences between them. The results with all samples are mixed and hard to interpret. Only 12 forecasters are useful, with horizons of 0 months, and still 14 forecasters are useful, with horizons of 11 months. This is partly because those flat answers are classified in terms of contraction. As noted above, flat means coincident DI to be around 50 or difficult to judge. It may be inappropriate that at the answers are classified into either contraction or expansion.

As for the results with samples excluding flat, most of forecasters are useful with horizons of up to three months and 78% of forecasters are useful with horizons of four months. Only one forecaster is useful with horizons of up to five months and exceeds the consensus DI forecast.

## References

[1] Ash, J.C.K., Smyth, D.J. and Heravi, S.M. 1998. Are OECD forecasts rational and useful?: a directional analysis, International Journal of Forecasting 14, 381-391.

[2] Ashiya, M. 2003. The directional accuracy of 15-months-ahead forecasts made by the IMF, Applied Economics Letters 10, 331-333.

[3] Ashiya, M. 2006. Are 16-month-ahead forecasts useful? A directional analysis of Japanese GDP forecasts. Journal of Forecasting 25, 201-207.

[4] Chua, C.L. and Tsiaplias, S. 2011. Predicting economic contractions and expansions with the aid of professional forecasts, International Journal of Forecasting 27, 438-451.

[5] Henriksson, R.D. and Merton, R.C. 1981. On market timing and investment performance. ii: Statistical procedures for evaluating forecasting skills, The Journal of Business 54, 513-533.

[6] Merton R.C, 1981. On market timing and investment performance i: An equilibrium theory of value for market forecasts, The Journal of Business 54, 363–406.

[7] Pesaran, M.H. and Timmermann, A. 1992. A simple nonparametric test of predictive performance, Journal of Business and Economic Statistics 10, 461-465.

[8] Pesaran, M.H. and Timmermann, A. 2009. “Testing Dependence Among Serially Correlated Multicategory Variables,” Journal of the American Statistical Association 485, 325–337.
[9] Pons, J. 2000. The accuracy of IMF and OECD forecasts for G7 countries, Journal of Forecasting 19, 53-63

[10] Schnader, M.H. and Stekler, H.O. 1990. Evaluating predictions of change, Journal of Business 63, 99-107.

[11] Sinclair, T.M., Stekler, H.O. and Kitzinger, L. 2010. Directional forecasts of GDP and inflation: a joint evaluation with an application to Federal Reserve predictions, Applied Economics, 42, 2289-2297.

[12] Stekler, H.O. 1994. Are economic forecasts valuable?, Journal of Forecasting 13, 495-505.

[13] Tsuchiya, Y. 2013. Are government and IMF forecasts useful? An application of a new market-timing test, Economics Letters 118, 118-120
Table 1: $ 2 \times 2 $ Contingency Table

Forecasts ( $ x_{t} $ )	Actual ( $ y_{t} $ )	Total
> 0	$ \leq 0 $
> 0	$ n_{1} $	$ n_{2} $	n
$ \leq 0 $	$ N_{1} - n_{1} $	$ N_{2} - n_{2} $	N - n
$ N_{1} $	$ N_{2} $	N

Table 2: Example of Data Release Schedule

Early part of the months	Mid	Late
Aug-13	$ \text{ESPF}(f_{2Q13,t}) $	1st preliminary GDP of 2Q 13	IBC of Jun 13
Jul-13	$ \text{ESPF}(f_{2Q13,t-1}) $	IBC of May 13
Jun-13	$ \text{ESPF}(f_{2Q13,t-2}) $	2nd preliminary GDP of 1Q 13	IBC of Apr 13
May-13	$ \text{ESPF}(f_{2Q13,t-3}) $	1st preliminary GDP of 1Q 13	IBC of Mar 13
Apr-13	$ \text{ESPF}(f_{2Q13,t-4}) $	IBC of Feb 13
Mar-13	$ \text{ESPF}(f_{2Q13,t-5}) $	2nd preliminary GDP of 4Q 12	IBC of Jan 13
Feb-13	$ \text{ESPF}(f_{2Q13,t-6}) $	1st preliminary GDP of 4Q 12	IBC of Dec 12
Jan-13	$ \text{ESPF}(f_{2Q13,t-7}) $	IBC of Nov 12
Dec-12	$ \text{ESPF}(f_{2Q13,t-8}) $	2nd preliminary GDP of 3Q 12	IBC of Oct 12
Nov-12	$ \text{ESPF}(f_{2Q13,t-9}) $	1st preliminary GDP of 3Q 12	IBC of Sep 12
Oct-12	$ \text{ESPF}(f_{2Q13,t-10}) $	IBC of Aug 12
Sep-12	$ \text{ESPF}(f_{2Q13,t-11}) $	2nd preliminary GDP of 2Q 12	IBC of Jul 12

Note: IBC and IIP denote Index of Business Condition, and Indices of Industrial Production.
Table 3: Summary of Forecasting Coincident DI

$ f_{t} $	$ f_{t-1} $	$ f_{t-2} $	$ f_{t-3} $	$ f_{t-4} $	$ f_{t-5} $
$ f > 50 $	23	26	26	28	28	31
$ f < 50 $	9	10	10	9	9	5
$ f = 50 $	0	2	0	0	0	0
Sum	32	38	36	37	37	36

Table 4: Summary of GDP Growth Forecast

$ f_{t} $	$ f_{t-1} $	$ f_{t-2} $	$ f_{t-3} $	$ f_{t-4} $	$ f_{t-5} $
$ f > 0 $	30	30	30	32	33	33
$ f < 0 $	8	8	7	5	4	3
$ f = 0 $	0	0	0	0	0	0
Sum	38	38	37	37	37	36