Electronic copy available at: http://ssrn.com/abstract=1793449

Investor Flows and the 2008 Boom/Bust in Oil Prices

Kenneth J. Singleton1

March 23, 2011

1Graduate School of Business, Stanford University, kenneths@stanford.edu. This research is the

outgrowth of a survey paper I prepared for the Air Transport Association of America. I am grateful

to Kristoffer Laursen for research assistance and to Kristoffer and Stefan Nagel for their comments.
Electronic copy available at: http://ssrn.com/abstract=1793449

Abstract

This paper explores the impact of investor flows and financial market conditions on returns

in crude-oil futures markets. I begin with a review of the economic mechanisms by which

informational frictions and the associated speculative activity may induce prices to drift away

from “fundamental” values and show increased volatility. This is followed by a discussion of

the interplay between imperfect information about real economic activity, including supply,

demand, and inventory accumulation, and speculative activity. Finally, I present new evidence

that there was an economically and statistically significant effect of investor flows on futures

prices, after controlling for returns in US and emerging-economy stock markets, a measure

of the balance-sheet flexibility of large financial institutions, open interest, the futures/spot

basis, and lagged returns on oil futures. The intermediate-term growth rates of index positions

and managed-money spread positions had the largest impacts on futures prices.
1 Introduction

The dramatic rise and subsequent sharp decline in crude oil prices during 2008 has been

a catalyst for extensive debate about the roles of speculative trading activity in price

determination in energy markets.1 Many attribute these swings to changes in fundamentals of

supply and demand with the price effects and volatility actually moderated by the participation

of non-user speculators and passive investors in oil futures markets and other energy-related

derivatives.2 At the same time there is mounting evidence that the “financialization” of

commodity markets and the associated flows of funds into these markets from various categories

of investors have had substantial impacts on the drifts and volatilities of commodity prices.3

This paper builds upon the latter literature and undertakes an in depth analysis of the impact

of investor flows and financial market conditions on returns in crude-oil futures markets.

The prototypical dynamic models referenced in discussions of the oil boom (e.g., Hamilton

(2009a), Pirrong (2009)) have representative agent-types (producer, storage operator, commercial consumer, etc.) and simplified forms of demand/supply uncertainty. Moreover, these

models, as well as the price-setting environment underlying Irwin and Sanders (2010)’s case

against a role for speculative trading, do not allow for learning under imperfect information,

heterogeneity of beliefs, and capital market and agency-related frictions that limit arbitrage

activity. As such, they abstract entirely from the consequent rational motives for many

categories of market participants to speculate in commodity markets based on their individual

circumstances and views about fundamental economic factors.

Detailed information about the origins of most of the open interest in OTC commodity

derivatives that could in principle shed light on the historical contributions of informationand learning-based speculative activity is not publicly available. However, indirect inferences

suggest that traders’ investment strategies did impact prices. Tang and Xiong (2009)

show that, after 2004, agricultural commodities that are part of the GSCI and DJ-AIG

indices became much more responsive to shocks to a world equity index, changes in the

U.S. dollar exchange rate, and oil prices. These trends are stronger for those commodities

that are part of a major index than for other commodities. Tang and Xiong attribute their

findings to “spillover effects brought on by the increasing presence of index investors to

individual commodities (page 17).” Using proprietary data from the Commodity Futures

1This debate is surely stimulated in part by the large costs that oil price booms and busts potentially

impose on the real economy. See, for example, Hooker (1996), Rotemberg and Woodford (1996), Hamilton

(2003), and the survey by Sauter and Awerbuch (2003).

2The conceptual arguments and empirical evidence favoring this view are summarized in a recent Organization of Economic Cooperation and Development report by Irwin and Sanders (2010).

3See, for example, Tang and Xiong (2009), Masters (2009), and Mou (2010).

1
250,000

350,000

450,000

550,000

650,000

750,000

850,000

$0.00

$25.00

$50.00

$75.00

$100.00

$125.00

$150.00

3-Jan-07

3-Mar-07

3-May-07

3-Jul-07

3-Sep-07

3-Nov-07

3-Jan-08

3-Mar-08

3-May-08

3-Jul-08

3-Sep-08

3-Nov-08

3-Jan-09

3-Mar-09

3-May-09

3-Jul-09

3-Sep-09

3-Nov-09

3-Jan-10

Contracts of 1000's of Barrels

WTI Price Per Barrel

Figure 1: Commodity index long positions inferred from the CIT reports (dashed line, right

sale) plotted against the front-month NYMEX WTI futures price (solid line, left scale).

Trading Commission (CFTC), Buyuksahin and Robe (2010a) link increased high-frequency

correlations among equity and commodity returns to trading patterns of hedge funds. Less

formally, Masters (2009) imputes flows into crude oil positions by index investors using the

CF T C’s commodity index trader (CIT) reports. The imputed index long positions based on

his methodology (Figure 1), displayed against the near-contract forward price of WTI crude

oil, shows a strikingly high degree of comovement. Additionally, Mou (2010) documents

substantial impacts on futures prices of the “roll strategies” employed by index funds, and

finds a link between the implicit transactions cost born by index investors and the level of

speculative capital deployed to “front run” these rolls.

To place these as well as my own empirical findings in an economic context, I begin in

Section 2 with a review of the economic mechanisms by which informational frictions, and

the associated speculative activity, may lead prices to drift away from “fundamental” values

and induce higher market volatility. Section 3 discusses the interplay between imperfect

information about real economic activity, including supply, demand, and inventory accumulation, and speculative activity. Section 4 presents new evidence that, even after controlling

2
for many of the other conditioning variables in recent students of price behavior and risk

premiums in oil futures markets, there were economically and statistically significant effects

of investor flows on futures prices. Concluding remarks are presented in Section 5.

2 Speculation and Booms/Busts in Commodity Prices

As background to the subsequent empirical analysis of the impact of investor flows on futures

prices I briefly review some of the potential consequences of heterogeneity of views, and

associated speculative trading, on commodity prices. Absent near stock-out conditions in a

commodity market, and (for simplicity) assuming a constant interest rate r, the current spot

commodity price is related to a market participant’s expected future price according to:4

St =

1

1 + r

E

Q

t

[St+1] + StCt, (1)

where Ct denotes the convenience yield net of storage costs, and E

Q

t denotes the expectation

under the risk-neutral pricing distribution conditional on date t information.

Much of the literature arguing for a “supply/demand” explanation of the oil price boom

focuses on representative producers and refiners and arrives at the similar expression

S

∗

t =

1

1 + r

E

P

t



S

∗

t+1 + Gt+1Ct+1

, (2)

where S

∗

t denotes the price of crude oil St adjusted for storage costs, Gt

is the price of

refined gasoline, and E

P

t denotes the expectation of market participants under the distribution

generating the historical data. The perfect-foresight model of Hamilton (2009a), for instance,

leads to a special case of (2) without the expectation operator (since there is no uncertainty

about future oil prices, inventory accumulations, or supply). The similarity between (1) and

(2) arises in extant supply/demand models when market participants are assumed to be

risk-neutral. If refiners and investors are risk averse, or if they face capital constraints that

lead them to behave effectively as if they are risk averse, then (1) continues to hold but (2) is

no longer valid. Accordingly, I henceforth focus on (1).

Implicit in (1) are the risk premiums that market participants demand when trading

commodities in futures and spot markets. In an arbitrage-free setting the futures price today

for delivery of a commodity τ periods in the future, F

τ

t

, is equal to the expected future spot

4See, for example, equation (4) of Casassus and Collin-Dufresne (2005).

3
price: E

Q

t

[St+τ ] = F

τ

t

. Therefore,

F

τ

t = E

P

t

[St+τ ] + E

Q

t

[St+τ ] − E

P

t

[St+τ ]



≡ E

P

t

[St+τ ] + RPτ

t

, (3)

where RPτ

t

is the risk premium associated with the economic forces that determine oil

prices over the horizon τ . More generally, RP also captures the consequences of any limits

to arbitrage, including financial market frictions that impinge on the flexibility of market

participants to finance their commodity positions. Combining expressions (1) and (3) gives

St =

1

1 + r

E

P

t

[St+1] + StCt +

1

1 + r

RP1

t

. (4)

An analogous expression holds for each investor who is participating in oil markets.

Expression (4) is not a structural relationship. Rather it summarizes the intertemporal

trade-offs of a market participant who is unconstrained in trading in the spot and futures

markets in circumstances where inventories are not near stock-out conditions. To sustain

this expression in equilibrium, it is not necessary that participants in the spot and futures

markets, or those refining or holding inventories of crude oil, be one and the same individual.5

Nor must one assume that investors hold the same beliefs about future market conditions

(i.e., that there is a representative investor).6

It follows that: (i) Spot prices are influenced not only by current oil market and macroeconomic conditions, but also by investors’ expectations about future economic activity. (ii)

Supply and demand pressures in the futures and commodity swap markets will in general

affect prices in the spot market. Indeed, these relationships are fully consistent with price

discovery taking place in either the futures, the cash, or the commodity swap markets, or in

all three. (iii) Risk premiums will typically change over time as investors’ willingness to bear

risk changes. As I discuss in more depth below, the capacity of financial institutions to bear

risk also changes over time, and this also may affect equilibrium futures and spot prices. (iv)

Higher-order moments of prices and yields in financial markets also affect spot, futures, and

swap prices through risk premiums and precautionary demands.

In addition these pricing relationships accommodate the possibility that investors hold

different beliefs about the future course of economic events that impinge on commodity prices,

and hence that there is not a representative investor in commodity markets. There is likely to

5

In particular, the claim that “index fund investors ... only participated in futures markets... In order to

impact the equilibrium price of commodities in the cash market, index investors would have to take delivery

and/or buy quantities in the cash market and hold these inventories off of the market. (ISOECD, page 8)” is

not true in the economic environment considered here.

6The same observations apply to the trading in and pricing of commodity swap contracts.

4
be some disagreement among market participants about virtually every source of fundamental

risk, including the future of global demands, the prospects for supply, future financing costs,

etc. Saporta, Trott, and Tudela (2009) document large errors in forecasting demand for oil,

typically on the side of under estimation of demand and mostly related to the non-OECD Asia

and the Middle East regions. Additionally, they document substantial revisions to forecasts

of market tightness, based on data reported by the U.S. Energy Information Administration

(EIA), especially during 2007.7 The International Energy Agency (IEA (2009)) points to

substantial revisions to their monthly estimates of demands for the U.S. and, regarding nonOECD inventories, IEA (2008b) observes that “detailed inventory data [for China] continues

to test observers’ powers of deduction. As we have repeatedly stressed in this report, these

data are key to any assessment of underlying demand trends... (page 15)” Sornette, Woodard,

and Zhou (2008) document significant differences in the total world supplies for liquid fuels

published by the IEA and the EIA, particularly from 2006 until 2008. The timeliness of

non-OECD data is highly variable (IEA), and OPEC quotas and measured production levels

are quite vague (Hamilton (2009b)).

Direct evidence on the extent of disagreement about future oil prices on the part of

professional market participants comes from comparing the patterns in the cross-sectional

standard deviations of the one-year ahead forecasts of oil prices by the professionals surveyed

by Consensus Economics.8 Larger values of this dispersion measure correspond to greater

disagreement among the professional forecasters surveyed. Figure 2 shows a strong positive

correlation between the degree of disagreement among forecasters and the level of the WTI

oil price. This comovement is consistent with the positive relation between price drift and

greater dispersion in investors beliefs found in theory and documented in equity markets.

How do heterogeneous beliefs get impounded into spot and futures commodity prices; and

what are the potential implications for booms and busts in commodity prices? Virtually all

classes of participants in commodity markets are, at one time or another, taking speculative

positions.9 Certainly in this category are the large financial institutions that make markets

7Market tightness is defined as total consumption (excluding stocks) minus the sum of non-OPEC and

OPEC production. After comparing news about, and revisions in forecasts of, supply and demand for oil

during 2008, these authors conclude that “Based on the news about the balance of demand and supply in

2008 ... it seems that one can justify neither the rise in prices in the first half of 2008, nor the fall in prices in

the second half (page 222).”

8Consensus Economics surveys over thirty of (in their words) “the world’s most prominent commodity

forecasters” and asks for their forecasts of oil prices in the future. The series plotted in Figure 2 is the

cross-forecaster standard deviation for each month of their reported forecasts. I am grateful to the IMF for

providing this series, as reported in their World Economic Forum.

9The primary exception would be participants that hold futures or options positions that precisely offset

their current spot exposures and who adjust their derivative positions frequently enough to rebalance as new

exposures arrive and old exposures dissipate.

5
5.00

7.00

9.00

11.00

13.00

15.00

17.00

60.00

80.00

100.00

120.00

140.00

160.00

o

n of

1‐Year Ahead Oil‐Price Forecasts

Price/Barrel ($)

WTI Price

Dispersion of Forecasts

‐1.00

1.00

3.00

5.00

7.00

9.00

11.00

13.00

15.00

17.00

0.00

20.00

40.00

60.00

80.00

100.00

120.00

140.00

160.00

9‐Jan‐07

9‐Mar‐07

9‐May‐07

9‐Jul‐07

9‐Sep‐07

9‐Nov‐07

9‐Jan‐08

9‐Mar‐08

9‐May‐08

9‐Jul‐08

9‐Sep‐08

9‐Nov‐08

9‐Jan‐09

9‐Mar‐09

9‐May‐09

9‐Jul‐09

9‐Sep‐09

9‐Nov‐09

9‐Jan‐10

Dispersion of

1‐Year Ahead Oil‐Price Forecasts

Price/Barrel ($)

WTI Price

Dispersion of Forecasts

a

Figure 2: The front-month NYMEX WTI futures price (solid line, left scale) plotted against

the cross-sectional dispersion of forecasts of oil prices one-year ahead by the professionals

surveyed by Consensus Economics (squares, right scale).

in commodity-related instruments; refiners and others who hold sizable inventories; hedge

funds and investment management companies; and commodity index investors.

How might this heterogeneity of beliefs impact oil prices? In a “rational expectations”

equilibrium (REE) the source of different views across investors is private information.

Investors share common priors and they do not disagree about public information. In

contrast, in a “differences of opinion” equilibrium (DOE) investors can disagree even when

their views are common knowledge. Accordingly, in a DOE investors can agree to disagree

even when they share common information– they disagree about the interpretation of public

information. Under a REE it is difficult to generate the volume of trade observed in

commodity markets, because investors share common beliefs (see the “no-trade” theorems

of Milgrom and Stokey (1982) and Tirole (1982)). In contrast in a DOE, because investors

may disagree about the interpretation of public information, it is possible to generate rich

patterns of comovement among asset returns, trading volume, and market price volatility

(e.g., Cao and Ou-Yang (2009) and Banerjee and Kremer (2010)).

When market participants have different information sets, behavior in the spirit of Keynes’

“beauty contest” may arise naturally. It is typically optimal for each participant to forecast the

6
forecasts of others (Townsend (1983), Singleton (1987)). That is, participants will try to guess

what other participants are thinking and to adjust their investment strategies accordingly.

Within present value models that share many of the same intertemporal considerations

involved in pricing commodities,10 Xiong and Yan (2009) and Nimark (2009) show that

groups of traders that hold different views will naturally engage in speculative activity with

each other. Indeed, Allen, Morris, and Shin (2006) show that this heterogeneity leads investors

to overweight public opinion and this, in turn, exacerbates volatility in financial markets.

In addition to excessive volatility, differences of opinion can give rise to drift in commodity

prices and momentum-like trading in response to public announcements.11 Conditional on

past performance, there may be periods when commodity prices tend to drift in the same

direction. Banerjee, Kaniel, and Kremer (2009) show that such price drift does not arise

naturally in a REE, but it typically symptomatic of a DOE in which investors disagree about

the interpretation of public information and in which they are uncertain about the views

held by other investors. Both of these suppositions seem plausible in commodity markets.

Adam and Marcet (2010a), taking a complementary approach, show how boom and

bust cycles in asset prices can result from Bayesian learning by investors. Investors in their

model are “internally” rational in the sense of Adam and Marcet (2010b)– they make fully

optimal dynamic decisions given their subjective beliefs about variables that impact prices

and are beyond their control. However investors may not agree on how public information

about fundamentals translate into a specific price level. Nor do investors know the utility

weights that other investors assign to specific economic events. For both of these reasons

internally rational investors try to infer from market prices information about fundamental

economic variables and the end result is not a REE. They show that a model of stock price

formation embodying these features produces boom/bust cycles in stock prices that match

those experienced historically.

Three implications of this literature, particularly as they relate to the roles of speculation

in commodity markets, warrant emphasis. First, it is not necessary for investors with heterogeneous beliefs to have private information in order for their actions to impact commodity

prices. Rather, so long as they have differences of opinion about the interpretation of public

information and find it useful to learn from past prices, then their actions can induce higher

volatility, price drift, and booms and busts in prices. Second, the documented comovement

among futures prices on commodities that are and are not in an index, or among spot prices

10These authors study bond marktes. As we have seen, analogous to the discounting in bond markets,

commodity markets involve present values tied to financing cost, convenience yields, and storage costs.

11There is extensive empirical evidence that announcements of public information lead post-announcement

drift and momentum in common stock markets; see, for instance, Zhang (2006) and Verardo (2009).

7
across markets with and without associated futures contracts, is not evidence against an

important role for speculation underlying this comovement.12 Participants in all commodity

markets should find it optimal to condition on prices in other markets when drawing inferences

about future spot prices, and this includes wholesalers and speculators.13

Third, the fact that investors are learning about both fundamentals and what other

investors know or believe about future commodity prices may mean that the release of a

seemingly small amount of new information about supply or demand has large effects on

prices. Indeed, it is possible that prices change owing to changes in investors perceptions or

risk appetite and absent the release of any new information.14

3 Demand/Supply, Inventories, and Speculation

Many of the arguments against a significant role for speculative trading in the recent

boom/bust in oil prices highlight the historical linkages between supply/demand and inventory

accumulation. Specifically, a widely held view is that speculative trading that distorts prices

on the upside must be accompanied by increases in inventories.15 This supposition has been

used by both sides of the speculation/fundamentals debate. Some arguing for fundamentals

have noted that we did not see large accumulations in inventories on the parts of refiners (e.g,

Hamilton (2009a)), while others (e.g., U.S. Senate Permanent Subcommittee on Investigations

(2006)) argue that the coincident increases in U.S. inventories and oil prices from 2004 to

2006 is evidence of speculative activity inducing higher spot prices. From Figure 3 it is seen

that prior to 2003 there was a strong negative relationship between the price of oil and the

amount of oil stored in the U.S. for commercial use (net of strategic petroleum reserves).

This price/inventory relationship turned significantly positive from 2004 to 2007. It weakened

in 2007 and turned negative, and then was weakly positive again during the first half of 2008.

12It follows that the presence of heterogeneous beliefs and learning could invalidate both of the following

claims in Irwin and Sanders (2010): (i) for index investors to have had a material affect on commodity prices

“would have required a large number of sophisticated and experienced traders in commodity futures markets

to reach a conclusion that index fund investors possessed valuable information that they themselves did not

possess (page 8).” and (ii) “if index buying drove commodity prices higher then markets without index fund

investment should not have seen prices advance (page 9).”

13The perception that there are links between flows into index funds and agricultural commodity prices

is evident from Corkery and Cui (2010) who cite concerns about pension fund investments in commodities

exacerbating fluctuation in food prices and, thereby, food shortages in poorer nations.

14Tang and Xiong (2009) conclude that “the price of an individual commodity is no longer simply determined

by its supply and demand. Instead, prices are also determined by ... the risk appetite for financial assets,

and investment behavior of diversified commodity index investors (page 30).”

15For instance, the IEA expresses the view that “if speculators are driving spot oil prices, an imbalance in

the form of higher stocks should be apparent (IEA (2008a)).”

8
$0

$15

$30

$45

$60

$75

$90

$105

$120

$135

$150

260 270 280 290 300 310 320 330 340 350 360 370 380

Weekly U.S. Crude Oil Inventory Excluding Strategic Petroleum Reserve (Millions of Barrels )

Crude Oil Spot Price vs. U.S. Stocks

4/19/02 – 10/16/09

Source: Energy Information Administration; Bloomberg

Note 1: Contango and Backwardation are defined using the spot price and the three-month futures price.

Price of U.S. Crude Oil

(U.S. Dollars a Barrel)

Regression Results of Spot Prices on Inventories

Time Period Term Structure[1]

Number of 

Observations Intercept Coefficient

T Stat for

Coefficient

Adjusted

R Squared

4/19/02 – 11/5/04 Backwardation 134 62.56 -0.10 -2.60 0.04

11/12/04 – 7/13/07 Contango 140 -57.95 0.36 11.97 0.51

7/20/07 – 5/16/08 Backwardation 44 180.29 -0.28 -1.98 0.06

5/23/08 – 10/16/09 Contango 74 369.09 -0.89 -8.28 0.48

Figure 3: U.S. commercial inventories of crude oil plotted against the spot price of oil, for

various recent subperiods.

Several caveats about the theoretically predicted price/inventory relationship and the

historical evidence warrant emphasis. First, the price of oil is set in global markets, so

it is potentially misleading to carry out a debate about inventory/price relationships by

focusing on U.S. inventory levels alone. As I discussed above, during this period several

major emerging economies where stockpiling crude oil in strategic reserves. These reserves

are omitted from Figure 3 and, even if one wanted to include them, the inventory data for

emerging economies has been much less reliable than for the G7.

Under the assumption that there is time-varying volatility (risk) related to either the

demand or supply of oil, those with storage capacity may also have a precautionary demand

for oil. An inherent feature of precautionary demand is that it increases with the degree of

risk. In a model of rational market participants in which there is time-varying economic

uncertainty about the future, but otherwise similar features to Hamilton’s framework, Pirrong

(2009) shows that there is not a stable relationship between inventories and prices and that

a positive inventory-price relationship may arise as a consequence of increased demand- or

supply-side uncertainty. Thus, there is not an unambiguously positive theoretical relationship

9
between changes in prices and inventories, even absent accommodation of important roles in

price setting of trading patterns induced by investor beliefs and learning.

Equally importantly, the impact of inventory adjustments on the volatility of prices

depends critically on what one assumes about the nature of uncertainty about supply and

demand. Many storage models (e.g., Deaton and Laroque (1996)) assume that, subsequent

to a surprise change in inventories induced by a shock to demand, inventories revert to a

long-run mean. It is this response pattern that led Verleger (2010), among others, to expect

inventory adjustments to have a stabilizing effect on oil prices. However, these models of

storage cannot simultaneously explain the high degree of persistence in oil prices and the

high level of oil price volatility over the past 30 years (Dvir and Rogoff (2009)).

Arbitrageurs (those who store to make a profit from price changes) are confronted with

two opposing implications of a positive income or demand shock. The price of oil increases

and there is a drop in effective availability, both of which encourage a reduction in optimal

storage. On the other hand, the persistent nature of aggregate demand means that both

income and prices are expected to be higher in the future. Dvir and Rogoff (2009) show

that when growth has a trend component, the expectation that prices will be higher in the

future encourages an increase in inventories and this effect dominates the reduction in storage

induced by the immediate post-shock increase in prices. On balance, storage (by arbitrageurs,

refiners or consumers) may amplify the effects of demand shocks on prices.16 Aguiar and

Gopinath (2007) argue that shocks to growth contribute more to variability in output in

emerging than in developed economies.

These observations, together with the inherent difficulty of accurately predicting future

growth, suggest that there were (i) differences of opinion about future growth in emerging

economies, and hence about demand for oil; (ii) market participants were, in part, drawing

inferences from market prices about the “consensus” view about economic growth; and (iii)

at least some subsets of participants were taking speculative (risky) positions in commodities

and emerging market equities, or both, based on their views. The literature summarized in

Section 2 shows that the resulting trading patterns could well have had destabilizing effects on

prices. Optimal inventory management, through the channels just discussed, can potentially

amplify the effects of differences of opinion and learning on commodity prices.

Figure 4 plots the level of non-strategic U.S. crude oil inventories against the spread

between the futures prices for two- and four-month contracts (M2 − M4, inverted scale).

Spreads that are above the zero line occur when the futures market is in contango, and spreads

16While this amplification mechanism has some characteristics of the precautionary demand studied by

Pirrong, the economic mechanism underlying it is not driven by uncertainty about demand, but rather by

expectations of rising prices.

10
200

250

300

350

400

450

Jan-04 Jan-05 Jan-06 Jan-07 Jan-08 Jan-09

-$8

-$6

-$4

-$2

$0

$2

$4

$6

U.S. Crude Oil Ending Stocks excl. SPR

M2 – M4

U.S. Crude Inventory Level excl. SPR vs. WTI Contango

1/5/04 – 10/23/09

Source: Energy Information Administration; Bloomberg (Millions of Barrels )

Figure 4: U.S. Commercial Inventories of Crude Oil Plotted Against the Spread Between

Two-Month and Four-Month Futures Prices

below this line indicate backwardation. There is a clear tendency throughout the period of

2004 through 2009 for inventories to increase when the futures market is in contango.17 The

positive correlation between inventory levels and the futures basis is consistent with the modern

theory of storage as developed by, for instance, Deaton and Laroque (1996) and Routledge,

Seppi, and Spatt (2000). However their models embody the “leaning against the wind” view

of inventory management and, hence, omit the possibility that expectations of higher prices

in the future may encourage inventory accumulation in response to a price increase today. A

notable feature of Figure 4 that seems consistent with the latter amplification effect, at least

from 2007 onwards, is that steepening and flattening of the forward curve preceded changes

in inventories: a steeper forward curve anticipated accumulations of inventories.

These theories of storage typically presume that market participants are risk neutral

and, hence, there is no risk premium embedded in futures returns. Gorton, Hayashi, and

Rouwenhorst (2007) extend the model of Deaton and Laroque (1996) to allow for risk averse

speculators (maintaining mean reverting demand) and show that inventories are negatively

related to expected excess returns in futures markets. They also establish a link between

the futures basis and inventories. These authors and Hong and Yogo (2010), among others,

17These patterns are even stronger when inventory levels from Cushing or Padd2 are used.

11
present empirical evidence that a high basis (high M2 − M4 in Figure 4) predicts a high

excess returns on futures positions, consistent with the theory of normal backwardation and

compatible with the theory of storage. I revisit these correlations for the recent period of the

oil boom as part of the following analysis of investor flows and oil prices.

4 Investor Flows and Oil Prices

Teasing out the relative contributions of the risks associated with fundamental factors in

demand and supply through the channels encompassed in models such those of Hamilton

(2009a) and Pirrong (2009) from the effects of price drift owing to learning and speculation

based on differences of opinion will require much richer structural models than have heretofore

been examined. In an attempt to provide some guidance to such endeavors, the remainder of

this paper explores the historical correlations between differences of opinion, trader flows,

and excess returns in oil markets, particularly for the 2008/09 boom and bust.

The comovement of the price of oil and the dispersion of forecasts of this price documented

in Figure 2 suggests that professional participants in this market held different views and

that these differences of opinion increased during this period. Of relevance to the subsequent

discussion is whether this increase in dispersion coincided with increased dispersion in forecasts

of world economic growth. Some evidence on this question is provided in Figure 5 which

plots the ratio of the forecast dispersion for the price of oil to the corresponding dispersion of

forecasts of growth for the world economy.18 At least relative to the dispersion in opinions

about world economic growth, there was something special about oil markets during 2008.

Dispersion in views about economic growth did not rise substantially from its mid-2008 value

until the spring of 2009 when the financial crisis was more pronounced.

4.1 What Is Known About Investor Flows and Commodity Prices?

A contentious issue related to the recent behavior of commodity prices is the degree to

which growth in index investing– exposure to commodities through index-linked products–

contributed to price volatility, a higher level of oil prices and greater disagreement among

market participants about the future course of oil prices. Surely the entry of index investors

as a new class of market participants affected the trading strategies of at least some other

large investors. In particular, Buyuksahin et al. (2008) argue that prior to the early 2000’s,

the prices of long- and short-dated futures contracts behaved as if these contracts were traded

18For the purpose of these calculations the world is considered to be the G7 plus Brazil, China, India,

Mexico, and Russia. I am grateful to the IMF for providing me with these dispersion measures.

12
5

10

15

20

25

30

35

40

45

Oil Std/GDP Std

0

5

10

15

20

25

30

35

40

45

February‐07

April‐07

June‐07

August‐07

October‐07

December‐07

February‐08

April‐08

June‐08

August‐08

October‐08

December‐08

February‐09

April‐09

June‐09

August‐09

October‐09

December‐09

February‐10

Oil Std/GDP Std

Figure 5: Ratio of the dispersions in forecasts for the price of oil and world economic growth

(real GDP growth).

in segmented markets. They find that, since the middle of 2004, the prices of one- and

two-year futures have been “cointegrated” with the nearby contract; that is, that all of these

prices trend together. This closer integration of futures along the maturity spectrum was

no doubt a consequence of several developments, including the increased trading activities

of hedge funds engaged in spread trades (Buyuksahin et al. (2008)) and the incentives for

index-fund managers to purchase longer-dated exposures through futures when the market

is in contango. Very little is known publicly about the degree to which different groups of

commodity investors were effectively trading against each other, either based on revealed

positions of classes of investors, observed order flow, or by following momentum strategies.19

Many have characterized index traders as “passive investors.”20 As noted by Stoll and

Whaley (2009), patterns similar to Figure 1 (in their case for agricultural commodities) reflect

the fact that a portion of the imputed position of index traders in any given commodity is

19Some information about positions was available from the CFTC and mutual funds, or was observed (by

traders) through financial institutions’ own trading operations. There is extensive empirical evidence that

order flow information in markets is a valuable input into the trading strategies of large financial institutions.

See, for example, the evidence on currency markets in Evans and Lyons (2009).

20For instance, Stoll and Whaley (2009) express the view that commodity index investors “do not take a

directional view on commodity prices. They simply buy-and-hold futures contracts to take advantage of the

risk-reducing properties they provide (Stoll and Whaley (2009), page 17).”

13
525,000%

560,000%

595,000%

630,000%

665,000%

700,000%

735,000%

$80.00%

$90.00%

$100.00%

$110.00%

$120.00%

$130.00%

$140.00%

8/Jan/08%

22/Jan/08%

5/Feb/08%

19/Feb/08%

4/Mar/08%

18/Mar/08%

1/Apr/08%

15/Apr/08%29/Apr/08%

13/May/08%27/May/08%

10/Jun/08%24/Jun/08%

Imputed%Index%PosiEons%in%Crude%Oil%%%

(thousands*of*barrels)*

Crude%Oil%Price%Per%Barrel%

375,000&

425,000&

475,000&

525,000&

575,000&

625,000&

675,000&

$35.00&

$55.00&

$75.00&

$95.00&

$115.00&

$135.00&

1.Jul.08&

15.Jul.08&29.Jul.08&

12.Aug.08&26.Aug.08&

9.Sep.08&

23.Sep.08&

7.Oct.08&

21.Oct.08&

4.Nov.08&

18.Nov.08&

2.Dec.08&

16.Dec.08&30.Dec.08&

Imputed&Index&PosiGons&in&Crude&Oil&&&

(thousands*of*barrels)*

Crude&Oil&Price&Per&Barrel&

Figure 6: Crude oil prices (near futures contract) and imputed positions of index investors

(barrels of oil) during the first (left) and second (right) halves of 2008.

driven by the movement in the underlying commodity price, as opposed to changes in the

sizes of the positions of index traders. Nevertheless, overall position sizes did change. Even

under the conservative estimates of position sizes by index investors in Stoll and Whaley, they

doubled between 2006 and the middle of 2008, and then declined rapidly by nearly one half

as of early 2009. Figure 6 overlays time paths of crude oil prices and the imputed positions of

index investors in crude oil during the first and second halves of 2008. This data also shows

a substantial increase and then decline in index positions, with medium-term patterns that

closely track those of oil prices during the “boom and bust.”

Moreover, the increased correlation between excess returns on commodities and global

equity returns during 2004 - 2009 documented in Tang and Xiong (2009) and Buyuksahin

and Robe (2010b) suggests that either index investors held positions in both asset classes

until the global economy weakened, at which point many simultaneously unwound their long

positions, or that different investors were engaged in correlated trading strategies induced by

similarly optimistic views about emerging economies.

More generally, changes in aggregate positions reflect purchases by new investors and

changes in existing positions of established investors. Even if the horizons of a majority of

index investors are relatively long (weeks and months, not days), their positions are surely

not immune to changes in their assessments of future economic growth, nor of their subjective

assessments of the reliability of their forecasts.

Another, complementary issue that naturally arises in discussions of the impact of any

given class of investors on commodity prices is whether large increases in desired long or short

14
positions can impact prices in the futures and spot markets. In any market setting where

there are limits to the amount of capital investors are willing to commit to an asset class– that

is, where there are limits to arbitrage– the answer is generally yes. Price increases in responses

to increased demands for long positions are typically necessary to induce other investors to

commit more capital to taking the opposite side of these transactions. Acharya, Lochstoer,

and Ramadorai (2009) and Etula (2010) document a significant connection between the

risk-bearing capacity of broker-dealers and risk premiums in commodity markets.

Though index traders have received much of the negative publicity in discussions of the

2008 boom/bust in oil prices, it is of interest to examine the impacts of the trading activities

of all large classes of investors on prices during this period. The CFTC is now making

available position reports on four categories of traders, back to 2006: traditional commercial

(commodity wholesalers, producers, etc.), managed money (e.g., hedge funds), commodity

swap dealers, and “other.” In addition, research staff at the CFTC have undertaken several

studies of trader positions using internal proprietary data that has a much finer breakdown

of market participants into categories of traders and is available daily.

Overall, most of the evidence from this literature suggests that position changes in futures

markets by managed money or commodity swap dealers either have weak or no (statistically

significant) impact on prices and there is some evidence that hedging activity tends to stabilize

prices (reduce price volatility).21 However, knowing whether price changes lead or lag position

changes over short horizons (a few days) is of limited value for assessing the price pressure

effects of flows into commodity derivatives markets. Of more relevance is whether flows affect

returns and risk premiums over weeks or months.22 The imputed flows of funds into index

positions displayed in Figure 1 suggests that such intermediate-term price-pressure effects

may well have been present.

Prior to 2009 the Commitment of Traders Report (COT) only reported information for

the broad categories of “commercial” and “non-commercial” traders. Figure 7 redisplays the

imputed long positions of index investors from the CIT reports that is in Figure 1,

23 along

with the “swap dealers and managed money” category from the COT report. The latter

is the data often used in empirical studies of the impact of index investor flows on futures

prices. Clearly these two series are very different, particularly from the fourth quarter of 2007

21See, for example, Boyd, Buyuksahin, Harris, and Haigh (2009), Buyuksahin and Robe (2009), Buyuksahin

and Harris (2009), and Brunetti and Buyuksahin (2009).

22Similarly, evidence that any particular group of investors acquires positions after say a price decline does

not contradict the view that this group is inducing systematic pressure for prices to move up or down.

23Implied CIT positions are calculated by dividing the imputed dollar amount of total index positions in

NYMEX WTI crude oil futures by the value of a contract, calculated as the front-month futures contract

price per barrel multiplied by 1000.

15
Figure 7: CIT-imputed commodity index long positions plotted against the swap dealer and

managed money positions as reported by the CFTC’s Commitment of Traders report.

through the third quarter of 2008, and then again through the second half of 2009. This

graph lends support to the view that the CFTC’s COT data does not give a reliable picture

of the overall demand for and supply of commodity risk exposure.24

Perhaps the most compelling evidence to date that index flows and “limits to arbitrage”

have, together, had economically important effects on futures prices is provided by Mou

(2010)’s analysis of excess returns around the dates of the rolls of the futures positions in

the GSCI index. He shows that, by taking certain spread positions in commodity futures

prior to the publicly known schedules for rolling the futures positions in commodity index

funds, speculators made substantial profits effectively at the expense of index investors. The

price-pressure effects were substantial, particularly for energy-related contracts. Moreover,

the profitability of the trading strategies Mou examines were decreasing in the amount of

arbitrage capital deployed in the futures markets and increasing in the proportion of futures

positions attributable to index fund investments. A striking aspect of Mou’s findings is that

simple and low-cost trade strategies could have been used to arbitrage away the large profits

24There is an extensive literature examining links between net positions of hedgers and the forecastability

of commodity returns– the “hedging pressure” hypothesis (Keynes (1930), Hicks (1939)). In two recent

explorations of this issue Gorton, Hayashi, and Rouwenhorst (2007) find no support for the hedging pressure

hypothesis, while Basu and Miffre (2010) argue that systematic hedging pressure is an important determinant

of risk premiums. Both use the aggregated CFTC data on commercial and non-commercial traders in futures

markets, a very course categorization that, as can be seen from Figure 7, is not reliably informative about

the trading activities of such classes of investors as index investors or hedge funds.

16
from positioning ahead of the Goldman roll. Yet, while the profitability of such positions

declined leading up to the boom of 2008, they remained positive suggesting that there were

limits to the amount of speculative capital investors were willing to deploy.

4.2 New Evidence on the Impact of Trader Flows on Oil Prices

In the light of this conflicting evidence on the impact of trader positions on futures prices,

I explored complementary statistical relationships using the imputed flows by index and

managed-money investors. Specifically, I computed weekly time-series of excess returns from

holding positions in futures at different maturity points along the yield curve. The maturities

included were the 1, 3, 6, 9, 12, 15, 18, 21, and 24 month contracts, and the sample period

was September 12, 2006 through January 12, 2010. Details of the excess return calculations

are presented in the Appendix.

I included the following list of predictor variables for excess returns:

RSP1 and REM1: the one-week returns on the S&P500 and the MSCI Emerging Asia

indices, respectively. Inclusion of these returns controls for the possibility that investors were

pursuing trading strategies in oil futures that conditioned on recent developments in global

equity markets.

REPO1: the one-week change in overnight repo positions on Treasury bonds by primary

dealers. Etula (2010) in the context of futures trading, and Adrian, Moench, and Shin (2010)

more generally, argue that the balance sheets of financial institutions affect their willingness

to commit capital to risky investments. This in turn implies that risk premiums may depend

on the costs to these institutions of financing their trading activities. The growth in overnight

repo positions is one indicator of balance-sheet flexibility.

IIP13: the thirteen-week change in the imputed positions of index investors in millions,

computed using the same algorithm as in Masters (2009). In contrast to most of the extant

literature, I focus on changes in index positions measured over three months (thirteen weeks)

rather than over a few days or a week.25

MMSPD13: the thirteen-week change in managed-money spread positions in millions, as

constructed by the CFTC. Erb and Harvey (2006) and Fuertes, Miffre, and Rallis (2008)

25The flows computed using the methodology in Masters (2009) is not without its limitations. However,

for analyzing forecasts of changes in futures prices, it is not necessary that IIP13 be a perfect measure of

the flow of funds into index positions. Some measurement errors seem inevitable. If the proportion of each

index made up of any one agricultural product is small, mismeasurement is likely to be amplified through

the scaling process. Further, valuation is done at the near-contract futures price (as was the case in Tang

and Xiong (2009)), and this might not have been how index traders positioned the actual fund flows in oil

markets. The evidence in Buyuksahin et al. (2008), based on proprietary CFTC data, suggests that the net

positions of commodity swap dealers were primarily in short-dated futures contracts (three months or under).

17
document that simple spread trades based on the term structure of futures prices led to large

historical returns. Spread positions were the largest component of open interest during my

sample period (Buyuksahin et al. (2008)), and the disaggregated COT reports show that

managed money accounts showed substantial growth in spread positions. Spread trades are

not signed: trades that are long or short the long-dated futures are treated symmetrically.

OI13: the thirteen-week change in aggregate open interest in millions, as constructed by the

CFTC. Hong and Yogo (2010) find that increases in open interest over an annual window

predict monthly excess returns on futures. One explanation for this finding is that investors

are learning about fundamental macroeconomic information from both past prices and open

interest.26 I account for this potential effect for weekly holding periods by conditioning on

the three-month change in aggregate open interest in oil futures.

AVBAS1: the one-week change in average basis. Defining the basis at time t of a futures

contract with maturity Ti(t) to be27

Bi(t) = 

F

Ti

t

St

1/(Ti(t)−t)

− 1, (5)

as in Hong and Yogo (2010), then AVBAS1 is the average of these values for maturities

i ∈ {1, 3, 6, 9, 12, 15, 18, 21, 24}. In computing (5) I account for the time-varying maturity of

the futures contracts. Hong and Yogo condition on their measure of basis to capture possible

effects of hedging pressures on subsequent returns on futures positions. It is also a proxy for

the net convenience yield in commodity markets.

Finally, I condition on the lagged value of the realized weekly excess return on oil futures

positions. Stoll and Whaley (2009) find that, once lagged returns on futures positions are

included in predictive regressions, there is no incremental predictive power for flows into

commodity index investment. Similar points related to lagged open interest have been made

by others. However, using data over a longer sample period and for a much broader set

of commodities, Hong and Yogo (2010) find a very strong predictive relationship between

current open interest and subsequent returns on futures positions. Moreover, when both open

interest and lagged returns are included in predictive regressions, open interest drives out the

forecasting power of returns.

I estimated the forecasting equations

ERmMt+1 = μm + ΠmXt + ΨmERmMt + εm,t+1, (6)

26Consistent with this interpretation, Hong and Yogo (2010) find that open interest also has predictive

content for future inflation and short-term bond yields.

27Note that this measure of the basis has the opposite sign of the basis in Figure 4.

18
Variable RSP1 REM1 REPO1 IIP13 MMSPD13 OI13 AVBAS1

Contemporaneous Predictors

ER1M 0.35 0.40 0.10 0.21 0.16 0.12 -0.43

ER3M 0.43 0.48 0.08 0.24 0.19 0.15 -0.26

ER6M 0.45 0.50 0.06 0.25 0.17 0.15 -0.21

ER12M 0.44 0.51 0.04 0.25 0.15 0.14 -0.17

ER24M 0.41 0.48 0.04 0.25 0.12 0.13 -0.12

Lagged Predictors

ER1M 0.03 -0.17 -0.21 0.25 0.18 0.12 -0.25

ER3M 0.11 -0.10 -0.20 0.26 0.19 0.13 -0.32

ER6M 0.13 -0.09 -0.19 0.26 0.18 0.13 -0.30

ER12M 0.16 -0.10 -0.19 0.26 0.16 0.12 -0.24

ER24M 0.15 -0.11 -0.17 0.25 0.13 0.11 -0.18

Table 1: Correlations among the one-week excess returns on futures positions and the

contemporaneous and lagged values of the predictor variables.

where ERmMtis the realized excess return for a one-week investment horizon on a futures

position that expires in m months, Xtis the set of predictor variables, and the data were

sampled at weekly intervals. The fitted values from these regressions are typically interpreted

as expected excess returns or, equivalently, as risk premiums in futures markets. This is

a natural interpretation when Xt represents information that was readily available to at

least some market participants at the time the forecasts were formed. The variables IIP13

and MMSP D13 were constructed (by the CFTC) based on information at the time of the

forecast. However this data was released by the CFTC starting in 2009 and, as such, was

not readily available to market participants during my sample period. Therefore, a finding

of economically important effects of these variables on ERmMt+1 represents evidence of

price pressure effects of flows by these investor classes on futures prices (controlling for other

variables in Xt), but not necessarily evidence of market participants adjusting their risk

premiums at the time in response to releases of information about these flows.

The correlations among the ERmM and both contemporaneous and first-lagged values of

the conditioning variables X are displayed in Table 1. All of the contemporaneous correlations

between the excess returns and the predictor variables have signs that are consistent with

previous findings in the literature. The correlations of the excess returns with emerging

market stock returns (REM1) and the growth in repo positions by primary dealers (REP O1)

change sign when these conditioning variables are lagged one period. Moreover, when investor

flows are measured over periods of weeks, rather than days as in much of the literature, they

have sizable correlations with excess returns. I elaborate on these findings below.

19
The correlations between changes in oil futures prices and both index and managedmoney flows are positive. For the signed index positions, this is consistent with positive

(momentum-type) price pressure effects. Notice also that the thirteen-week change in open

interest is positively correlated with oil price changes. This finding is consistent with the

strong positive correlation of these variables found by Hong and Yogo (2010) using monthly

data over a much longer sample period. They interpret these correlations as indicative of

open interest embodying information about future economic activity that investors find useful

for predicting future commodity prices. Such a role of open interest would naturally arise in

economic environments where investors learn from past prices and trading volumes as in the

models discussed in Section 2. Supporting such an informational role, Hong and Yogo also

find that open interest has predictive content for future bond returns and inflation in the U.S.

To explore these comovements more systematically and jointly, I estimated the parameters

in (6) using linear least-squares projection. The null hypotheses are that the elements of

Π are zero: excess returns on futures positions are not predictable by the variables in Xt,

after conditioning on lagged information about excess returns. The economic theories of

the dynamic properties of excess returns reviewed above allow for the possibility that other

transformations of the conditioning information (more lags or nonlinear transformations)

have incremental predictive content for excess returns. Accordingly, following Hansen (1982)

and Hansen and Singleton (1982), robust standard errors are computed allowing for serial

correlation and conditional heteroskedasticity in εt+1.

28 Estimates of Π along with their

asymptotic “t-statistics” are displayed in Table 2.

Note, first of all, that the adjusted R2’s in these projections provide compelling evidence

that excess returns on futures positions in oil markets had a significant predictable component

during this sample period. From Figure 8 it is seen that the volatilities of the excess returns

decline, and the mean excess returns are increasing, in the contract month. Thus, the low

adjusted R2’s for the longer maturity contracts imply that the predictor variables explain a

smaller percentages of relatively less volatile, but larger on average, returns.

The coefficients on most of the conditioning variables and for most of the contract months

are statistically different from zero at conventional significance levels. The two primary

exceptions are the coefficients on the lagged returns (second to last column) and the growth

in open interest (OI13). Interestingly, the coefficients on OI13 (partial correlations) switch

sign and shrink in absolute value relative to the correlations in Table 1, and they are small

relative to their estimated standard errors. After conditioning on the trading patterns of

index investors and hedge funds, at least for the sample period around the 2008 boom/bust,

open interest does not have significant predictive content for excess returns.

28Specifically, I use the Newey and West (1987) construction allowing for five lags.

20
Contract RSP1 REM1 REPO1 IIP13 MMSPD13 OI13 AVBAS1

RLag Adj

R2

1 0.332 -0.342 -0.201 0.272 0.357 -0.103 -4.165 -0.219 0.27

(1.44) (-2.44) (-2.89) (3.51) (4.36) (-2.17) (-6.26) (-2.05)

3 0.361 -0.242 -0.170 0.218 0.284 -0.082 -3.661 -0.152 0.27

(1.99) (-2.02) (-2.76) (3.71) (4.43) (-1.87) (-6.48) (-2.10)

6 0.391 -0.261 -0.150 0.197 0.245 -0.072 -3.022 -0.105 0.25

(2.35) (-2.27) (-2.64) (3.49) (4.14) (-1.74) (-5.59) (-1.62)

9 0.424 -0.275 -0.142 0.187 0.222 -0.067 -2.551 -0.090 0.24

(2.67) (-2.46) (-2.58) (3.45) (3.95) (-1.73) (-4.72) (-1.40)

12 0.437 -0.283 -0.133 0.179 0.202 -0.064 -2.141 -0.075 0.22

(2.84) (-2.60) (-2.49) (3.42) (3.83) (-1.73) (-3.97) (-1.14)

18 0.430 -0.286 -0.119 0.166 0.174 -0.058 -1.657 -0.054 0.20

(2.99) (-2.79) (-2.35) (3.42) (3.61) (-1.72) (-3.13) (-0.75)

24 0.412 -0.287 -0.107 0.157 0.159 -0.053 -1.329 -0.046 0.18

(2.98) (-2.87) (-2.21) (3.46) (3.40) (-1.67) (-2.60) (-0.59)

36 0.378 -0.294 -0.093 0.145 0.144 -0.048 -0.981 -0.033 0.16

(2.85) (-2.99) (-2.05) (3.52) (3.02) (-1.60) (-2.10) (-0.40)

Table 2: Estimates and robust test statistics for the futures excess return forecasting model.

21
0 10 20 30 40 50 60

−10

−5

0

5

10

15

20

25

Contract month

mean 1W rolling return (in bp)

(a) Sample Mean

0 10 20 30 40 50 60

3.5

4

4.5

5

5.5

6

6.5

Std Dev 1W rolling return (in %)

Contract month

(b) Sample Standard Deviation

Figure 8: Sample moments in basis points of the weekly excess returns on futures.

That intermediate-term changes in index positions largely drive out open interest as a

predictor of changes in oil prices suggests that at least a portion of the predictive content of

open interest found in previous studies was a consequence of it serving as a proxy for the

information in index and hedge fund positions. Consistent with this interpretation is the

sample correlation between IIP13 (MMSP D13) and OI13 of 0.56 (0.45). By conditioning

on order flow, absent information about IIP13 and MMSP R13, market participants may

well have captured a substantial part of the impact of index and manage-money flows on

prices, and this would show up in required risk premiums in oil markets.

There does appear to be a small remaining negative effect of open interest on futures

returns, particularly for the shortest maturity futures contracts. (The negative coefficients

on OI13 decline monotonically with the maturity of the futures contract.) A plausible

interpretation of these negative coefficients is that some market participants were taking

contrarian positions based on the view that oil prices had over-reacted to new information.

Some evidence that hedge funds played such a stabilizing role over very short horizons (much

shorter than what I am considering) is provided in Brunetti and Buyuksahin (2009). For the

intermediate term horizons investigated here, such (statistically weak) feedback effects are

dominated by flows from index and managed-money accounts.

The coefficients on the lagged futures returns for the one- and three-month contracts

are marginally significant, but for all other contracts they are statistically insignificant.

Additionally, the absolute values of the estimates decline rapidly with the maturity of the

futures contract. Thus, there is weak evidence of reversals in the prices of the short-dated

futures contracts, after conditioning on the information in the other components of Xt. More

22
generally, and importantly for interpreting the evidence regarding the boom and bust in

oil prices, these findings suggest that the significant predictive content of the conditioning

variables Xtis fully robust to inclusion of the lagged return (see also below). This stands in

contrast to the results from focusing on returns and conditioning variables over daily intervals

as, for instance, in Buyuksahin and Harris (2009) and Stoll and Whaley (2009).

The large positive correlation between returns on commodity futures positions and stocks

in emerging economies reported in Table 1 is often noted in discussions of investor flows. An

interesting aspect of both the correlations in Table 1 and the coefficients in Table 2 is that

the lagged returns on emerging market equity positions are negatively correlated with futures

returns. The negative (and statistically significant) partial correlation coefficients indicate

that, after for controlling for all of the other components of Xt, an increase in REM1 predicts

a decline in futures prices in the subsequent week. These findings suggest that positive news

about emerging economies leads to contemporaneous changes in oil futures and emerging

market equity prices in the same direction. However, REM1 predicts subsequent reversals in

futures prices. Limits to capital market intermediation and the consequent slow commitment

of capital to new OTC commodity derivatives positions is a plausible explanation for these

reversals (see Duffie (2010) and the references therein). Spot and futures prices respond

immediately to new information about emerging market growth, but broker-dealers take time

to adjust their own inventory and OTC derivatives positions.

Similarly, the negative and statistically significant effects of REP O1 on excess returns

are consistent with model of Etula (2010) in which risk limits and funding pressures faced by

broker-dealers impact risk premiums in commodity markets. The OT C commodity derivatives

market is substantially larger than the markets for exchange traded products and servicing

the OT C markets requires a substantial commitment of capital by broker-dealers. As funding

conditions improve– reflected here through an increase in the repo positions of primary dealers–

the effective risk aversion of broker-dealers declines and, hence, so should the expected excess

returns in commodity futures markets. This effect of funding liquidity on excess returns

declines (in absolute value) with contract maturity, while remaining statistically significant.

Perhaps the most striking findings in Table 2 are the statistically significant predictive

powers of changes in the index investor (IIP13) and managed money spread (MMSP D13)

positions on excess returns in crude oil futures markets. Increases in flows into index funds

over the preceding three months predict higher subsequent futures prices. These effects are

significant for contracts of all maturities, and this is after controlling for lagged futures returns

and all of the other conditioning variables in Xt. The flow variable IIP13 is capturing price

pressures associated with intermediate-term persistent flows of funds into index positions.

Elaborating, assuming that futures returns and the predictor variables are covariance

23
stationary, the null hypothesis that the coefficient on investor flows in projections of weekly

returns on intermediate-term growth rates in these flows has the same economic content

as the null hypothesis that short-term flows impact futures prices over intermediate-term

horizons (Hodrick (1992), Singleton (2006)). Consistent with most prior studies, including

weekly changes in index positions has little predictive content for the weekly excess returns.

These observations suggest that, if present, the price drift in futures markets related to

learning and speculative trade is manifested over return horizons of a few weeks or months.

Correlations between futures prices and flow variables sampled at high frequency are likely

to be dominated by noise that obscures the presence of this longer-horizon comovement.

There is also a significantly positive effect of flows into managed money spread positions

on future oil prices. The weekly excess returns embody the roll returns once per month.

Therefore, the predictive power of MMSP D13 might in part reflect the growth in spread

trading by hedge funds in anticipation of the Goldman roll for index funds (Mou (2010)).

Alternatively, Boyd, Buyuksahin, Harris, and Haigh (2010) present evidence of herding

behavior by hedge funds during this sample period. Whatever the motives of the professionals

categorized as “managed money” traders, their net effect on excess returns was positive:

increases in spread positions were associated with future increases in oil contract prices.

Ceterus paribus, the marginal effects of growth in index or managed-money positions on

excess returns were comparable: the hypothesis that the matching coefficients in columns

five and six of Table 2 are the same cannot be rejected for any of the contract months.

Finally, increases in the average basis (AV BAS1) are associated with declines in excess

returns. The coefficients on AV BAS1 are both more negative and statistically significant

for the short-maturity contracts. These statistically significant coefficients are in contrast to

those in studies of earlier sample periods (e.g., Fama and French (1987)), and also to those

in Hong and Yogo (2010) who examine monthly excess returns over the longer sample period

1987-2008. Additionally, AV BAS1 shows small bilateral correlations with the other conditioning variables. For instance, its correlations with (REP O1, IIP13, MMSP D13, OI13) are

(−0.15, −0.05, −0.05, −0.08) so the weekly average basis represents distinct information about

future returns. Hong and Yogo (2010) interpret a negative correlation between the basis and

returns on futures positions as arising out of hedging activities of producers. However, this

explanation appears to be based on the “leaning against the wind” view of hedging. Recall

from Figure 4 that changes in the shape of the futures curve tended to anticipate changes in

inventory positions during my sample period. Moreover, under plausible assumptions about

the persistence in aggregate demand for oil, price increases today can lead to increases in

inventories in anticipation of further price increases in the future.

An alternative possibility is that the trading strategies of investors– not necessarily

24
producers– led futures prices to move more than spot prices in response to commodityrelevant news. These reactions were then partially reversed in the subsequent week. The

impacts of AV BAS1 on excess returns decline (in absolute value) with contract maturity,

indicating that reversals were largest for the shorter maturity contracts. An interesting

question for future research is the relationship during this boom/bust period between the

convenience yields on futures contracts and excess returns.

4.3 Robustness of Excess Return Projections to the Inclusion of

Other Conditioning Information

The reported findings are robust to inclusion of several other conditioning variables. Specifically, as noted above, the growth rates in flows into index and managed-money accounts over

one-week intervals do not add significantly to the forecasts of excess returns.

In preliminary regressions I also included the one-week change in the Cushing, OK

inventory of crude oil in millions, as reported on Bloomberg, to check the robustness of the

results to the inclusion of inventory information. There is a statistically weak negative effect

of inventory information on the excess return for the one-month contract. Beyond one month

the coefficients are all small relative to their estimated standard errors.

Additionally, I estimated the predictive regressions with additional lags of excess returns

included as predictor variables and the pattern of results in Table 2 remained qualitatively the

same. The inclusion of past information about returns does materially affect the predictive

content of the investor flow variables.

Finally, some argue that the trading patterns of index and managed-money investors are

linked to speculation about global economic growth. A relevant question then is whether

measures of global economic growth also had predictive power for excess returns on futures.

As a proxy for aggregate demand, I follow Kilian (2009) and Pirrong (2009), as well as

many oil-market practitioners, and use shipping rates, namely, the Baltic Exchange Dry

Index (BEDI). The growth rate of the BEDI over the previous three months does explain

an additional 2 − 3% of the variation in excess returns, and its coefficients are marginally

statistically significant. However, BEDI has very little effect on the explanatory power of the

other predictors: they continue to explain most of the variation in futures returns.

5 Concluding Remarks

Investing while learning about economic fundamentals, both from public announcements

and market prices, may well induce excessive price volatility and drift in commodity prices.

25
These phenomena are entirely absent, essentially by assumption, from the models of oil price

determination that focus on representative suppliers, consumers, and hedgers. An implication

of the presence of “forecasting the forecasts” of others is that commodity prices can be more

volatile and, from a social welfare perspective, society can be worse off even though each

investor participating in this guesswork is small. That is, social welfare may be reduced

even though equilibrium prices do not depend directly on the degree to which any individual

investor incorrectly measures fundamental economic variables.

The welfare costs of trading based on imperfect information are potentially amplified by

the fact that the costs to individual investors of near-rational behavior – following slightly

suboptimal investment or consumption plans– is negligible and yet this behavior might be

quite costly for society as a whole (Lucas (1987) and Cochrane (1989)).29 When investors

make small correlated errors around their optimal investment policies, financial markets

amplify these errors and generate volatility in securities prices that is unrelated to fundamental

supply/demand information (Hassan and Mertens (2010)).

The particular economic mechanism through which social welfare is reduced in the model

of Hassan and Mertens (2010) is that higher volatility in capital markets raises risk premiums

and, as a consequence, the cost of capital to firms. This, in turn, affects firms’ investment

plans and impacts overall output in an economy. The same issues arise, for example, in

an economy in which commercial users purchase commodities as intermediate inputs into

production. Furthermore, such additional frictions as multi-period contracting over labor

and physical capital will likely exacerbate the social costs of excessive volatility.

Much of the recent debate about “excessive” speculation in commodity markets has

focused on the flows into index funds. I have found that these flows are positively correlated

with future changes in commodity prices, and these findings complement the evidence in Tang

and Xiong (2009) on the financialization of commodity markets. Assessing the social costs of

these price-pressure effects requires additional economic structure. If index investors are just

slightly too optimistic (in market rallies) or pessimistic (in market downturns) relative to the

true state of the world then their errors, while inconsequential for their own welfare, may be

material for society as a whole.30

29Such suboptimal plans may arise out of misinterpretations of public information say about future

economic growth in developing countries, because of small costs to sorting through the complexity of global

economic developments and their implications for commodity prices, or because of over-confidence about

future economic growth as in Dumas, Kurshev, and Uppal (2006).

30Recent research by Qiu and Wang (2010) shows that when market participants have heterogeneous

information, and so asset prices depend on the expectations of the expectations of others, prices tend to

be more volatile and the overall welfare of society is lowered. Additionally, if index traders impart noise to

market prices through their trading activities, then this could also reduce the efficiency with which futures

and spot markets perform their roles in price discovery.

26
More broadly, it is the dynamic interactions of the trading activities of index investors,

hedge funds, broker/dealers in commodity markets, and commercial hedgers that ultimately

set prices in commodity spot and futures markets. Just as index investors are, in part,

adjusting their positions based on their views about global supplies and demands, other

market participants are doing likewise and they are positioning based on their views about

what index and other classes of investors are doing. This may well explain the significant

effects of hedge fund spread positions on excess returns in oil markets documented here.

Finally, much of the literature on commodity pricing abstracts from the impact of the

extensive array of derivatives contracts in commodity markets (e.g., commodity swaps) on

market-price dynamics. Adding derivatives markets will typically improve price discovery

and mitigate some of the informational problems highlighted above. However enhanced

price discovery is only one facet of the complex effects of imperfect information and incomplete financial markets on commodity price setting. In addition to their affects on price

discovery, derivatives markets alter participants’ access to hedging vehicles and, thereby,

affect allocational efficiency. Society can be worse off when information is asymmetric and

participants are not able to hedge against all of their business or income risks (Huang and

Wang (1997)). A key step towards a better understanding of the effects of interactions among

various market participants on price behavior is the collection and dissemination of more

detailed information about the trading patterns in OT C commodity derivatives, as well as

exchange traded futures.

27
Appendix: Construction of Excess Returns

Let F

Ti(t)

t denote the futures contract with expiration Ti(t). The futures-price-term-structure

consists of points F

T1(t)

t

, ..., FTN (t)

t

. Let D(s) > s denote the first time after s that the generic

futures curve switches contracts. Then, for all i = 1, ..., N − 1, and all s,

Ti+1(D(s) − 1) = Ti(D(s))

The excess rolling return in generic contract i, between s and t is given by

F

Ti(t)

t

F

Ti(s)

s

− 1 if t < D(s)

F

Ti(D(s)−1)

D(s)−1

F

Ti(s)

s

·

F

Ti(t)

t

F

Ti+1(D(s)−1)

D(s)−1

− 1 if D(s) ≤ t < D(2)(s)

F

Ti(D(s)−1)

D(s)−1

F

Ti(s)

s

·

F

Ti(D(2)(s)−1)

D(2)(s)−1

F

Ti+1(D(s)−1)

D(s)−1

·

F

Ti(t)

t

F

Ti+1(D(2)(s)−1)

D(2)(s)−1

− 1 if D(2)(s) ≤ t < D(3)(s)

and so forth.

By construction these are the net returns from holding one long position in the generic imonth contract, liquidating the position the day before the generic curve ’moves the contracts

one month down’, and going long one unit in the following month i + 1 (which the day after,

by definition will be generic contract i). This strategy is followed from s until t.

The risk free rate does not enter these calculations. The rational is (following, for instance,

Etula (2010)) that investing in a futures position, does not require an initial capital injection.

In practice, however, the futures trading strategies are met with margin calls. For this reason

Hong and Yogo (2010) consider a fully collateralized return of the form (say if t < D(s))

F

Ti(t)

t

F

Ti(s)

s

R

f

s,t

My calculations omit the multiplying factor R

f

s,t from the construction of excess returns.

28
References

V. Acharya, L. Lochstoer, and T. Ramadorai. Limits to arbitrage and hedging: Evidence

from commodity markets. Technical report, New York University, 2009.

K. Adam and A. Marcet. Booms and busts in asset markets. Technical report, IMES

Discussion Paper 2010-E-2, Bank of Japan, 2010a.

K. Adam and A. Marcet. Internal rationality, imperfect market knowledge and asset prices.

Technical report, forthcoming, Journal of Economic Theory, 2010b.

T. Adrian, E. Moench, and H. Shin. Financial intermediation, asset prices, and macroeconomic

dynamics. Technical report, Federal Reserve Bank of New York Staff Report 422, 2010.

M. Aguiar and G. Gopinath. Emerging market business cycles: The cycle is the trend.

Journal of Political Economy, 115:69–102, 2007.

F. Allen, S. Morris, and H. Shin. Beauty contests and iterated expectations in asset markets.

Review of Financial Studies, 19:719–752, 2006.

S. Banerjee and I. Kremer. Disagreement and learning: Dynamic patterns of trade. Journal

of Finance, LXV:1269–1302, 2010.

S. Banerjee, R. Kaniel, and Il Kremer. Price drift as an outcome of differences in higher-order

beliefs. Review of Financial Studies, 22:3707–3734, 2009.

D. Basu and J. Miffre. Capturing the risk premium of commodity futures: The role of hedging

pressure. Technical report, 2010.

N. Boyd, B. Buyuksahin, J. Harris, and M. Haigh. The impact of hedging on futures prices.

Technical report, U.S. Commodity Futures Trading Commission, 2009.

N. Boyd, B. Buyuksahin, J. Harris, and M. Haigh. The prevelence, sources and effects of

herding. Technical report, U.S. Commodity Futures Trading Commission, 2010.

C. Brunetti and B. Buyuksahin. Is speculation destabilizing? Technical report, Commodity

Futures Trading Commission, 2009.

B. Buyuksahin and J. Harris. The role of speculators in the crude oil futures market. Technical

report, Commodity Futures Trading Commission, 2009.

29
B. Buyuksahin and M. Robe. Commodity traders’ positions and energy prices: Evidence

from the recent boom-bust cycle. Technical report, U.S. Commodity Futures Trading

Commission, 2009.

B. Buyuksahin and M. Robe. Speculators, commodities and cross-market linkages. Technical

report, Commodity Futures Trading Commission, 2010a.

B. Buyuksahin and M. Robe. Hedge funds, stress, and cross-market linkages. Technical

report, Commodity Futures Trading Commission, 2010b.

B. Buyuksahin, M. Haigh, J. Harris, J. Overdahl, and M. Robe. Fundamentals, trader activity

and derivative pricing. Technical report, Commodity Futures Trading Commission, 2008.

H. Cao and Ou-Yang. Differences in opinion of public information and speculative trading in

stocks and options. Review of Financial Studies, 22:299–335, 2009.

J. Casassus and P. Collin-Dufresne. Stochastic convenience yield implied from commodity

futures and interest rates. Journal of Finance, LX:2283–2331, 2005.

J. Cochrane. The sensitivity of tests of the intertemporal allocation of consumption to

near-rational alternatives. American Economic Review, 79:319–337, 1989.

M. Corkery and C. Cui. Calstrs reins in plans for a big bet. Wall Street Journal, November

12, 2010, 2010.

A. Deaton and G. Laroque. Competitive storage and commodity price dynamics. Journal of

Political Economy, 104:896–923, 1996.

D. Duffie. Asset price dynamics with slow-moving capital. Journal of Finance, LXV:1237–1267,

2010.

B. Dumas, A. Kurshev, and R. Uppal. What can rational investors do about excessive

volatility and sentiment fluctuations? Technical report, Swiss Finance Institute Research

Paper No. 06-19, 2006.

E. Dvir and K. Rogoff. The three epochs of oil. Technical report, Harvard University, 2009.

C. Erb and C. Harvey. The strategic and tactical value of commodity futures. Financial

Analysts Journal, 62:69–97, 2006.

E. Etula. Broker-dealer risk appetite and commodity returns. Technical report, Federal

Reserve Bank of New York, 2010.

30
M. Evans and R. Lyons. Forecasting exchange rate fundamentals with order flow. Technical

report, University of California, Berkeley, 2009.

E. Fama and K. French. Commodity futures prices: Some evidence on forecast power,

premiums, and the theory of storage. Journal of Business, 60:55–73, 1987.

A. Fuertes, J. Miffre, and G. Rallis. Tactical allocation in commodity futures markets:

Combining momentum and term structure signals. Technical report, EDHEC, 2008.

G. Gorton, F. Hayashi, and K. Rouwenhorst. The fundamentals of commodity futures returns.

Technical report, Yale University, 2007.

J. Hamilton. What is an oil shock? Journal of Econometrics, 113:363–398, 2003.

J. Hamilton. Causes and consequences of the oil shock of 2007-08. Brookings Papers on

Economic Activity, 2009a.

J. Hamilton. Understanding crude oil prices. Energy Journal, 30:179–206, 2009b.

L. Hansen. Large sample properties of generalized method of moments estimators. Econometrica, 50:1029–1054, 1982.

L. Hansen and K. Singleton. Generalized instrumental variables estimation of nonlinear

rational expectations models. Econometrica, 50:1269–1286, 1982.

T. Hassan and T. Mertens. The social cost of near-rational investment: Why we should worry

about volatile stock markets. Technical report, University of Chicago, 2010.

J. Hicks. Value and Capital. Oxford University Press, 1939.

R. Hodrick. Dividend yields and expected stock returns: Alternative procedures for inference

and measurement. Review of Financial Studies, 5:357–386, 1992.

H. Hong and M. Yogo. Commodity market interest and asset return predictability. Technical

report, Princeton University, 2010.

M. Hooker. What happened to the oil price-macroeconomy relationship? Journal of Monetary

Economics, 38:195–213, 1996.

J. Huang and J. Wang. Market structure, security prices, and informational efficiency.

Macroeconomic Dynamics, 1:169–205, 1997.

IEA. Medium-term oil market report. July:International Energy Agency, 2008a.

31
IEA. Medium-term oil market report. October:International Energy Agency, 2008b.

IEA. Oil market report. December:International Energy Agency, 2009.

S. Irwin and D. Sanders. The impact of index swap funds on commodity futures markets.

Technical report, OECD Food, Agriculture and Fisheries Working Papers No. 27, 2010.

J. Keynes. Treatise on Money. Macmillan, London, 1930.

L. Kilian. Not all oil price shocks are alike: Disentangling demand and supply shocks in the

crude oil market. American Economic Review, 99:1053–1069, 2009.

R. Lucas. Models of Business Cycles. Basil Blackwell, 1987.

M. Masters. Testimony before the commodity futures trading commission. Technical report,

Commodity Futures Trading Commission, August 2009.

P. Milgrom and N. Stokey. Information, trade, and common knowledge. Journal of Economic

Theory, 26:11–21, 1982.

Y. Mou. Limits to arbitrage and commodity index investments: Front-running the goldman

roll. Technical report, Columbia Business School, 2010.

W. Newey and K. D. West. Hypothesis testing with efficient method of moment estimation.

International Economic Review, 28:777–787, 1987.

K. Nimark. Speculative dynamics in the term structure of interest rates. Technical report,

Universitat Pompeu Fabra, 2009.

C. Pirrong. Stochastic fundamental volatility, speculation, and commodity storage. Technical

report, University of Houston, 2009.

W. Qiu and J. Wang. Asset pricing under heterogeneous information. Technical report, MIT,

2010.

J. Rotemberg and M. Woodford. Imperfect competition and the effects of energy price

increases on economic activity. Journal of Money, Credit, and Banking, 28:550–577, 1996.

B. Routledge, D. Seppi, and C. Spatt. Equilibrium forward curves for commodities. Journal

of Finance, 55:1297–1338, 2000.

V. Saporta, M. Trott, and M. Tudela. What can be said about the rise and fall in oil prices?

Bank of England Quarterly Bulletin, 49:215–225, 2009.

32
R. Sauter and S. Awerbuch. Oil price volatility and economic activity: A survey and literature

review. Technical report, IEA Research Paper, 2003.

K. Singleton. Asset prices in a time series model with disparately informed, competitive

traders. In W. Burnett and K. Singleton, editors, New Approaches to Monetary Economics.

Cambridge University Press, 1987.

K. Singleton. Empirical Dynamic Asset Pricing. Princeton University Press, 2006.

D. Sornette, R. Woodard, and W. Zhou. The 2006-2008 oil bubble and beyond. Technical

report, ETH Zurich, 2008.

H. Stoll and R. Whaley. Commodity index investing and commodity futures prices. Technical

report, Vanderbilt University, 2009.

K. Tang and W. Xiong. Index investing and the financialization of commodities. Technical

report, Princeton University, 2009.

J. Tirole. On the possibility of speculation under rational expectations. Econometrica, 50:

1163–1181, 1982.

R. Townsend. Forecasting the forecasts of others. Journal of Political Economy, 91:546–588,

1983.

U.S. Senate Permanent Subcommittee on Investigations. The role of market speculation in

rising oil and gas prices: A need to put the cop back on the beat. Technical report, United

States Senate, 2006.

M. Verardo. Heterogeneous beliefs and momentum profits. Journal of Financial and Quantitative Analysis, 44:795–822, 2009.

P. Verleger. Comments on federal speculative position limits for referenced energy contracts

and associated regulations. Federal Register 4144, 2010.

W. Xiong and H. Yan. Heterogeneous expectations and bond markets. Review of Financial

Studies, 2009.

F. Zhang. Information uncertainty and stock returns. Journal of Finance, LXI:105–36, 2006.

33