Scaling the Critics
Uncovering the Latent Dimensions of Movie Criticism
with An Item Response Approach∗Michael Peress†

Arthur Spirling‡

June 1, 2009∗

Excellent research assistance from Edward Laird and Chris Tice is gratefully acknowledged. We thank
Brett Gordon and Keith Poole for useful comments. This work was originally presented as a poster at
the Summer Political Methodology Meeting (2008) and we thank participants for feedback, especially Chris
Achen and Alastair Smith. Peress thanks the Institute of Quantitative Social Science for hospitality. We arevery grateful for comments from two anonymous referees and the AE at JASA that helped us improve thecontent and structure of our paper.
†
Department of Political Science, University of Rochester. mperess@mail.rochester.edu‡
Department of Government and Institute of Quantitative Social Science, Harvard University.
aspirling@gov.harvard.edu

1Abstract
We study the critical opinions of expert movie reviewers as an item response problem.
Building on earlier ‘unfolding’ models, we develop a framework that models an individual’s decision to approve or disapprove of an item. Using this approach, we are able torecover the locations of movies and ideal points of critics in the same multi-dimensional
space. We demonstrate that a three dimensional model captures much of the variation in critical opinions. The first dimension signifies movie ‘quality’ while the othertwo connote the nature and subject matter of the films. We then demonstrate thatthe dimensions uncovered from our ‘utility threshold model’ are statistically significant
predictors of a movie’s success, and are particularly useful in predicting the success of‘independent’ films.

Key words:

utility threshold model

2

film

ideal points

1

Introduction

For the year 2006, the Motion Picture Association reported that international revenues generated by its composite companies totaled some $42.6 billion (Hollinger, 2007). This sum
is on a par with the gross domestic product of Kenya for the same period. Clearly then,the movie industry is an important economic force both in the United States ($24.3 billion
revenue for 2006) and elsewhere ($18.3 billion). Fulfilling a consumer-advisory rôle within
this massive sector, movie critics are ubiquitous: reviews and recommendations for films canbe found in many journalistic outlets like newspapers, magazines and online websites. Major
studios apparently accord substantial influence to such critics, as do film historians: Smith(1998), for example, names the critics Gene Siskel and Roger Ebert in his top 100 rankingof the most influential people in movie history. Critics are fêted with press kits, advancescreenings and other perks, and then using (selected, positive) reviewers’ opinions directlyin the marketing of their product. Indeed, Sony Pictures went so far as to create a fictional critic—named David Manning—whose enthusiastic (and entirely fabricated) ‘quotes’
appeared on several of the studio’s movie adverts circa 2001 (Elsworthin, 2005).
Quite apart from their significance to large film-making firms and the news media devoted to the entertainment-industry, there is considerable academic interest in critics’ choices
and decision-making processes. First, within the marketing literature, assessing and quantifying the influence of critical reception on the commercial success of film media has been
an ongoing concern (Ainslie, Drèze and Zufryden, 2005; Eliashberg and Shugan, 1997; Neelamegham and Chintagunta, 1999). Modeling the behavior of critics directly would thus
paint a more complete picture of the interrelationship between film characteristics and market performance. Second, film criticism—particularly when practiced by those versed in film
theory —is an important element of cultural studies, a discipline that seeks to systematicallyunderstand cultural phenomena in terms of their social, political and psychological causes

3and consequences. Hence, there is motive to explore the ways in which audiences ‘receive’the motion picture medium (Blumer, 1933; Kracauer, 1957; Mulvey, 1975; Riesman, Denny
and Glazer, 1968). By analyzing new data on hundreds of critical reviews this paper seeks
to contribute to both these scientific endeavors.
As we will describe in more detail below, the data are examples of item responses (Lord,
1980; Hambleton, Swaminathan and Rogers, 1991). Our central concern is using psychometric measurement techniques—especially those derived from item response theory (IRT)—to
uncover the latent traits that characterize movie critics and the movies that they review.
The data differ from traditional applications since the subjects here choose whether to ‘approve’ or ‘disapprove’ of a single item. Hence, our theoretical framework of actor behavior
leads us to employ a statistical approach that differs somewhat from the cumulative models
commonly seen in social science applications like educational testing (Rasch, 1961; Lord,
1980; Bock and Aitken, 1981), marketing research (Kamakura and Srivastava, 1986; Goettler and Shachar, 2001; Anand and Byzalov, 2008), and legislator ideal point estimation(Poole and Rosenthal, 1997; Martin and Quinn, 2001; Clinton, Jackman and Rivers, 2004).
Specifically when uncovering legislative ideal points, notice that the spatial locations of twoalternatives are of interest: the status quo and the proposal; this is a sharp contrast to thecritic case, where only one alternative is reviewed. Our framework—the ‘utility thresholdmodel’—applies to movie criticism, and more generally, to approval or ordinal rating data.
Our paper generalizes existing models for approval data in a number of ways. First, ourframework is multidimensional. This is important because we expect critics to differ in theirpreferred movie characteristics. Second, we allow for a non-diagonal proximity metric in ourestimation. As we show in the paper, this is necessary for preserving rotational invariancein the model. Third, we allow critics to differ in their approval thresholds. This featureis necessary to account for the fact that some critics are stingier with their praise thanothers. Fourth, we can recover the ideal points of critics and the locations of movies in4the same multidimensional space. This differentiates our procedure from scaling procedures
developed for dichotomous and polytomous choice data. Finally, when applied to ratings
data, our procedure allows us to control for a type of selection bias which may be present inindices of movie quality. Specifically, critics may choose to review movies that they expect
to enjoy. Our procedure can control for this type of selection bias, if the critics’ choices ofwhich movies to review are based on the spatial characteristics of the movies.
Intriguingly, we find that the ‘expert’ critics in our data set—and the movies themselves—
are almost fully described by three latent dimensions: they pertain to ‘quality’, followed bya division of space between ‘nerds’, ‘jocks’ and ‘art-house’. These latter labels refer to typesof consumers who might enjoy predominantly science-fiction, action adventure and deep
(potentially disturbing) emotional movies, respectively. We demonstrate that such reviews
are good predictors of financial success for movie makers, especially for independent films
with relatively narrow audiences.
Outside of movie criticism, our estimator applies to a number of other important problems. Legislators choose whether or not to cosponsor legislation. In marketing, a panel ofconsumers may be given a set of products to rate, and latent characteristics of these products
could be deduced from these ratings. In admissions processes to universities, officers decide
whether or not to allow a potential students entry based on their qualities. More generally,
our framework extends existing models for approval data in a way necessary for analyzing
the diversity of choice present in many applications.

2

Data and Background

2.1

Data

Until relatively recently, data on critic responses to movies was both widely-scattered and in
no standard form: different media recorded reviews in multiple ways, from long discursive
5

articles with implicit judgments, to spoken television or radio reports to summary ‘star’system recommendations. It was thus extremely costly to collate critical opinions. Moreover,
the analyst was typically required to either use a few ‘key’ reviewers as indicative of a larger
audience, or laboriously recode responses in order to make them comparable. The adventof the internet, however, has changed matters. Rotten Tomatoes, a website situated at
http://www.rottentomatoes.com, collates both multiple reviews for any given movie, andcodes each review—in terms of how positive or negative it was towards the film—using acommon rating system. In particular, Rotten Tomatoes considers each film review by eachdifferent critic (of which more than 100 may exist for recent movies) and then denotes theopinion as ‘fresh’ (i.e. the critic recommends the film) or ‘rotten’ (i.e. the critic does notrecommend the film). This information is available to the public.
To see how this information might be used, first let c = 1, . . . , C index the critics and let
m = 1, . . . , M index the movies. The data to be modeled is then a C × M matrix of observed
ratings (coded by Rotten Tomatoes) by the C critics on the M movies. Let Y denote thismatrix and let yc,m denote the rating critic c gives movie m. We will code yc,m = 2 if thecritic recommended the movie (i.e. it is ‘fresh’), yc,m = 1 if the critic did not recommend it(i.e. it is ‘rotten’), and yc,m = 0 if the critic did not review the movie.
Our database uses a very expansive definition of what it is to be a film critic. Individuals
who submit only a handful of film reviews to online mailing lists are considered critics. To
focus on the population of interest—expert reviewers—we restrict Y to all critics who aremembers of the National Society of Film Critics. This organization holds a prestigious place
within the movie reviewing world and consists of approximately 60 respected individuals, allof whom are elected to their positions. In addition, these critics typically write and turn intheir reports for publication at approximately the same time. Thus there is little danger, forexample, that critics respond to each other’s opinions rather than their viewing experience.
We included all such critics who reviewed at least 20 films and all films that received at least6

50 reviews on Rotten Tomatoes. The resulting dataset has approximately 50 critics and 1000
movies. The minimum number of reviews a movie received among the NSFC critics was 16,
while the median number of movies each NSFC critic reviewed was 336.

2.2

Cumulative Models

As should be clear, the matrix Y contains rows of ‘individuals’ responding in a dichotomous
way to ‘items’ in its columns. If we wish to understand the latent traits possessed by bothcritics and movies, IRT seems a reasonable way to proceed. It is quite common to consider
the following model,Pr(yc,m = 2) = F (am (θc − bm )).

(1)

where F represents a strictly increasing cumulative distribution function (cdf). When F ischosen to be the Gaussian cdf, we have the normal ogive model. When F is chosen to be thelogistic distribution (i.e. F (x) = 1/(1 − e−x )), we have Birnbaum’s two-parameter logistic
model. When F is chosen to be the logistic distribution and am = 1 for all m, we have the
Rasch model as a special case.
These approaches are collectively referred to as cumulative models. When applied toeducational testing, θc is interpreted as the intelligence of individual c, bm is interpreted as
the difficulty of item m, and am determines the discrimination power of item m. Variantsof these models allow for more than two responses, multiple dimensions of intelligence, anonzero probability of guessing a correct answer, and various other features. Such models
share the property that the probability of observing a ‘correct’ response of yc,m = 2 is strictlyincreasing in intelligence θc . This is reasonable when applied to education testing, but may
not be appropriate in some other applications.

7

2.3

Unfolding Models

An alternative to the cumulative model is the unfolding model, pioneered by Coombs (1964,
esp. Ch 15). The unfolding model differs from the cumulative model in that the probabilityof a positive response is strictly decreasing in the distance between an individual’s ideal
point and the spatial location of the item. The probability of observing a positive response
is maximized at the individual’s ideal point, denoted by αc .
It is this framework that we build upon in our model of movie criticism. The unfolding
model often takes the form,
Pr(yc,m = 2) = F (−(αc − δm )2 ).

(2)Here, F would typically be selected to the logistic or normal distribution. Examples of unfolding models include DeSarbo and Hoffman (1987), Andrich (1988), Hoijtink (1990; 1991),
Andrich and Luo (1993), Takane (1996), Leenen and Mechelen (2004) and Maydeu-Olivares,
Hernandez and McDonald (2006). These models differ in the exact set of assumptions they
employ, including whether the characteristic space is allowed to be multidimensional, whether
the ideal points are treated as fixed or random effects, and so on.

3Model and Estimation Procedure

3.1

The Utility Threshold Model

Our model should have a number of features. First, it should be multidimensional because
we expect critics to differ in their preferred movie characteristics. Second, the model should
be of an unfolding variety. This will allow critics to prefer movies that offer a combinationof some action and some romance, for example. Cumulative models, by contrast, would
require critics to have preferences that are strictly increasing (or decreasing) in ‘action-ness’.
8Third, we should allow for a non-diagonal weighting matrix. This is mostly a technicalrequirement, but is necessary to ensure that the resulting likelihood function is invariant tolinear transformations of the characteristic space. A fourth requirement is that critics withsimilar ideal points should be allowed to differ in the probability that they assign a given
movie a positive review. Some critics may simply be ‘stingier’ with their praise, and wewould like to be able to capture this in our framework.
We begin by assuming that the ideal points of critics and the locations of movies canbe represented in the same D-dimensional space. We let αc ∈ RD denote the ideal point ofcritic c and we let δ m ∈ RD denote the location of movie m. For example, there might bethree dimensions (i.e. D = 3) in which all movies and critics can be situated: perhaps thefirst dimension corresponds to ‘action-ness’, the second to ‘romance-ness’ and the third to‘drama-ness’. A romantic-comedy would have a ‘low’ score on the first dimension, but be‘high’ on the other two. It seems sensible to suppose that critics are most likely to approveof a movie that is close to their ideal point, and we assume the utility critic c gets frommovie m is given by,
uc,m = −(αc − δ m )0 W (αc − δ m ) + ²c,m .

(3)Here, ²c,m are independent and identically distributed shocks from the standard normal
distribution, and W is a symmetric positive definite weighting matrix. A critic who likes
romantic-comedies over all other types of films would have an ideal point which is low on thefirst dimension and high on the other two. We assume that critic c gives a positive review
to movie m if his utility exceeds his approval threshold. Hence, we observe a fresh rating if
uc,m ≥ ūc , or equivalently,²c,m ≥ ūc + (αc − δ m )0 W (αc − δ m ).

(4)

One may worry that critics choose to review movies that they expect to like (because they9

enjoy seeing good movies) or movies that they expect to dislike (to allow for entertaining
reviews). These facts are accounted for in our framework, to the extent that such selection
operates on the estimated critic ideal points and movie locations. This is true because we
explicitly model the process by which critics decide whether to like or dislike a movie interms of movie locations and critic ideal points. In this sense, we control for many of the
aspects that determine whether a critic likes or dislikes a movie.
Returning to the derivation, we have,Pr(yc,m = 1) = Φ(ūc + (αc − δ m )0 W (αc − δ m ))(5)

Pr(yc,m = 2) = 1 − Φ(ūc + (αc − δ m )0 W (αc − δ m )).

(6)

Note that the ‘zero’-dimensional model is of interest as well. In this case, there are no spatial
locations of either critics or movies to be estimated: movies are treated as homogenous
entities and the only source of heterogeneity comes from the fact that some critics arestingier with their praise.
As a way to interpret the model in Equations (5) and (6), consider the ‘trace line’ in Figure
1. Notice that for any particular utility threshold, ū, the critic’s probability of approving
the film is decreasing quadratically as the movie’s location (δ̄) moves away from his spatial
preference (α)—he is most likely to approve when their locations coincide (when α − δ̄ = 0).
For any particular spatial distance between movie and critic, notice that increasing ū (i.e.
making the critic generally harder to please) will decrease the probability that he approvesof the movie.
[Figure 1 about here.]10

We can write the log-likelihood function as follows,

LC,M (α, ū, δ, W ) =

C X
M
X

[1{yc,m = 1} log Φ (ūc + (αc − δ m )0 W (αc − δ m ))c=1 m=1+1{yc,m = 2} log {1 − Φ (ūc + (αc − δ m )0 W (αc − δ m ))}] . (7)Estimating the parameters of the model can be accomplished by maximizing (7). This isstraightforward in principle, but a number of complication arise, which we describe later inthis section.

3.2

Relationship to Applied IRT

As noted above, one of the simplest educational models (see Lord, 1980, for example) has
00a test taker with latent trait zm
determining her performance on item m. The trait zm
isa composite of the examinee’s ability θ and an error component for item m. Typically, we
assume those errors are normally distributed, and that they have equal variance regardlessof the ability of the students concerned. Before going further, notice that for our model,that assumption about disturbances yields the representation in Figure 2. Here, there arethree individuals h, j, k with different spatial preferences (α parameters) but the same utility
threshold ū. They are confronted with the same movie which we will assume has δ = 3. Recall
that utility is quadratically decreasing in the movie’s distance from the reviewer. Critic hhas a spatial preference for α ≈ 1 so he is likely to disapprove of the movie. By contrast, j
is most likely to approve—and more likely to do so than k. Now suppose that we abandon
the assumption of a common ū, such that k is more difficult to please (though her spatial
preferences are similar to before). The shift up from ū to ūk will make k more likely todisapprove of the movie: a larger portion of her error term is now shaded.

11

[Figure 2 about here.]
The estimator we propose is closely related, but is it not isomorphic, to the estimators
commonly used for item response theory. Our estimator can be written as,
Pr(yc,m = 2) = F (ūc + αc 0 W αc − 2αc 0 W δ m + δ m 0 W δ m )(8)while the multidimensional normal ogive model can be written as,
Pr(yc,m = 2) = F (am + b0m θ c )

(9)
1

1

We can set up a relationship between the two models by letting θ c = ([W 2 αc ]1 , . . . , [W 2 αc ]D , ūc +1

1

αc 0 W αc ), am = δ m 0 W δ m , and bm = (−2[W 2 δ m ]1 , . . . , −2[W 2 δ m ]D , 1). We now have thatthe D-dimensional utility threshold model is isomorphic to a D +1-dimensional normal ogive
model where the last component of bm is restricted to be equal to 1. Otherwise put, we canalways find a D + 1-dimensional normal ogive model which summarizes the data at least as
well as the D-dimensional utility threshold model, and we can always find a D + 1 utility
threshold model which summarizes the data at least as well as a D-dimensional normal ogive
model. This arrangement suggests that we cannot differentiate between the utility thresholdand normal ogive models on the basis of model-fit alone (and hence, we don’t try to).
Instead then, the advantage of the utility threshold model is that it posits an appropriate
structural model for the data, which allows us to correctly interpret the estimated parameterswhen applied to movie criticism data (and approval or ordinal rating data more generally).
If the true data generating process were a D-dimensional utility threshold model, we would
be able to successfully fit a D + 1-dimensional normal ogive model. The difficulty would
come in interpreting θ c and (am , bm ). Note that θ c would contain the same information as(αc,1 , . . . , αc,D , ūc ), but the estimates would not reveal which components of θ c characterize

12the ideal points and which components characterize heterogeneity in the thresholds. This
problem occurs because of the rotational invariance present in item response models, meaningthat ūc need not appear as the last element of θ c .
A second advantage of our technique is that we can recover critic and movie locationsin the same multidimensional space, something which would be impossible if we applied
the traditional item response estimator to approval data. Cumulative models are closely
related to the dichotomous choice models considered in the political science, economics, andmarketing literatures. In these dichotomous choice models, individuals choose between two
items located in a multi-dimensional space. Each individual has an ideal point located in
the same-multidimensional space. This framework has a reduced form that is isomorphic
to the multi-dimensional cumulative model. In applications of the normal ogive model todichotomous choice data, we can recover ideal points and cutting planes in the same multidimensional space, but we cannot recover item locations because we cannot separately identify
the distance between the items and the variance of the disturbance term for that item (Poole,2005). Our setup is different because individuals rate a single item at a time. This is the
key difference that allows us to recover movie locations in our framework.

3.3

Identification

As is usual with such models we must impose some restrictions on the parameters in order toensure identification. In the case of the standard multi-dimensional item response problem,
it is well known that θ c must be constrained for D+1 individuals. A similar solution emerges
here.
Throughout, we use zero subscripts to denote the parameters of the data generatingprocess, i.e. the “true” parameter values. That is, α0 = (α1,0 , . . . , αC,0 ) denote the true critic
ideal points, δ 0 = (δ1,0 , . . . , δM,0 ) denote the true movie characteristics, etc. Unsurprisingly,
the parameters of the utility threshold model are only identified up to location and scale.
13Specifically, consider the reparametrization,

αc = Aαc,0 + b,ūc = ūc,0 ,δ m = Aδ m,0 + b,

W = (A0 )−1 W 0 A−1(10)where A has full rank. It is straightforward to show that
F (ūc + (αc − δ m )0 W (αc − δ m )) = F (ūc,0 + (αc,0 − δ m,0 )0 W 0 (αc,0 − δ m,0 ))for all c, m. This indicates that we can apply a linear transformation to the critic ideal points
without changing the value of the log-likelihood function, provided we can alter the otherparameters in the model. To achieve point identification, we can normalize any D + 1 ideal
points. Without loss of generality, we can constrain αD+1 = 0 and αc = ec for c ∈ {1, . . . , D}
where ec is a unit vector. These constraints allow us to pin down the location and scale ofthe critic ideal points and movie locations. Otherwise put, the estimated parameter vector
uniquely gives rise to the data seen in practice: there exists no other vector that could
possibly be responsible for the data. In the Appendix, we prove that the utility thresholdmodel is identified under these conditions. We effectively show that once we constrain theideal points of D + 1 critics, we cannot alter the parameter space leaving the value of the
log-likelihood intact, with any transformation (linear or nonlinear).

3.4

Implementation

The utility threshold model bears a strong resemblance to the item response models popularin the psychometric, marketing, and political science literatures. The estimation approaches
used fall into three broad categories. Fixed effects estimators treat both the item characteristics and individual characteristics as parameters to estimate (Lord, 1980; Poole andRosenthal, 1997). Random effects integrate out the item (or individual) characteristics (Bock14

and Lieberman, 1970; Bock and Aitken, 1981). Conditional fixed effect estimators concentrate out the item parameters (Rasch, 1961). The fixed effects estimators have the advantage
of producing additional information, which in our case includes both the individual (critic)
and item (movie) specific parameters. Hence we take this approach. In other applications,
we may observe a large number of raters rating a small number of items. In these situations,a random effects model would be more appropriate if the goal is to recover only the item
characteristics.
A second choice we must make is whether to employ a maximum likelihood or Bayesianestimator. Both maximum likelihood (Lord, 1980; Poole and Rosenthal, 1997) and Bayesian(Albert, 1992; Beguin and Glas, 2001; Martin and Quinn, 2001) versions of the fixed effects
estimator have been applied in the social science literature. Programs for implementing theseestimators are widely available but they cannot be directly applied here since, as noted, theinformation we wish to garner is not forthcoming from a standard item-response model.
The Bayesian estimator is easier to implement efficiently, and modifying the existing code
would not be very difficult. Experience indicates that the maximum likelihood estimator ismore difficult to implement, yet it is computationally more efficient, particularly when thedimensionality is large. Because computational efficiency was a chief concern, we choose to
implement the latter.
While maximizing the likelihood defined in equation (7) is straightforward in principle, anumber of complications arise. First, this model involves a very large number of parameters—
K = C(D + 1) + M D + D(D + 1)/2. For example, in a four dimensional model, there
are more than 6,000 parameters to estimate. This optimization problem would usually be
infeasible, but the special form of the objective function makes it tractable. In particular,
we can compute the objective function, the gradient, and the Hessian in O(CM ) operations,which is significantly less than the O(C 2 M 2 ) and O(C 3 M 3 ) operations that would usually berequired to compute them, respectively. Our implementation relies on the Zig-Zag algorithm
15that has been applied to estimate nonlinear fixed effects models (Heckman, 1981) and item
response models (Lord, 1980; Poole and Rosenthal, 1991, 1997).
A second concern is that despite our restriction to the NSFC critics there is still somesparseness in the data: some movies have few reviews while some critics opine on few films.
There is thus potential perfect-separation in the data. For these reasons, we use a penalizedlikelihood approach (in the sense of Firth, 1993). Here, we follow the spirit rather than theletter of Firth’s suggestions: we do not use a penalization based on Jeffrey’s priors and weare not per se interested in asymptotic refinements.
That objective function takes the following form:

L̃C,M (α, ū, δ, W ) = LC,M (α, ū, δ, W ) +CX

λu (ū2c ) +

c=1

C
X
c=1

λα (αc 0 αc ) +MXλδ (δ m0 δ m ) (11)m=1where LC,M is as given in Equation (7) and λu > 0, λα > 0 and λδ > 0 are penalty terms.
An equivalent formulation is to think our approach as finding the mode of the posterior distribution where independent normal priors are placed on (α, ū, δ) and a degenerate uniform
prior is placed on W . Notice that the contribution of the penalty terms in the objectivefunction approaches zero as the sample size increases: this is because the likelihood termfrom Equation (7) involves a double sum while each component of the penalty involves asingle term.

4

Results

We estimated a series of models, from zero through eight possible dimensions. Our firsttask was to choose between these models. We chose not to rely on purely statistical measures of model fit (e.g. a likelihood ratio test) because such measures tend to favor very
high-dimensional models in large data sets—far more dimensions than we will be able to16

successfully interpret (van der Linden and Hambleton, 1997; Ostini and Nering, 2006). We
instead considered the geometric mean probability (the average probability of a correct prediction). Relying solely on in-sample measures of model fit can lead to over-fitting, so we
also computed the geometric mean probability just on a holdout sample. In computing
the out-of-sample fit, we relied on a 20 percent holdout sample and computed the geometricmean probability among all movies that were reviewed by at least 12 critics. Table 1 displaysthese measures for the various models.
[Table 1 about here.]
Our choice of dimensionality was based primarily on out of sample fit, but we also considered
our ability to interpret the estimated dimensions and the usefulness of the estimated dimensions for subsequent analysis. Using the out of sample geometric mean probability, we foundthat the three dimensional model was best—it had a geometric mean probability of 64.6%.
The baseline model with no spatial dimensions provided a geometric mean probability of
54.3%. Among the models that we estimated, moreover, the dimensions generated by the 3
dimensional model proved easiest to interpret. In addition, we found that the results weremost useful for subsequent analysis (such as the regressions we consider in Section 5). Given
that these three criteria lead us to the same model choice, we are fairly confident that thethree dimensional model is most appropriate for this data.
The model we estimated located the movies and critics in three dimensions while also
estimating the individual-level utility thresholds for the critics. Recall that a lower ū implies
a more permissive critic who ceteris paribus is more willing to return a recommendation forthe movie. After plotting the density of the thresholds, there is evidence of a slight negative
skew: otherwise put, while the majority of critics are symmetrically located, there are a few‘easily pleased’ individuals to the far left (see Figure 3). Interestingly, the most generous
critic is Roger Ebert (of the Chicago Sun-Times) who gives a ‘fresh’ rating 64% of the time.
17

It is, by contrast, hard work to impress Amy Taubin, who writes columns for The Village
Voice—she likes just 39% of the movies she reviews.
[Figure 3 about here.]
In Figure 4 we present a plot of the three spatial dimensions. For the moment, we do notlabel the points, but they can be demarcated by their shape: the movies appear as roundpoints, while the critics are triangles. A feature of Figure 4 is that the point clouds for critics
and movies overlap, but not to the same extent in all dimensions. In the top and middle
panels, the movies and critics overlap much less than in the bottom panel. Otherwise put,the δ1 , α1 dimension appears to discriminate between the groups in space. In particular, thecritics generally appear to right of the movies: the critics have higher estimated positions on
this dimension. To be clear here, under our original normalization, we discovered a dimension
with a very high level of discrimination between critic and movie locations. We identified
this as a quality dimension and rotated the data (exploiting rotational invariance) such that
this dimension appeared as δ1 , to aid in our interpretations.
[Figure 4 about here.]
We contend that this dimension represents a movie’s ‘quality’ and, as we noted earlier, all else
equal, critics prefer higher-quality movies to lower-quality ones. In our understanding, ‘highquality’ movies have a combination of two elements—artistic pretension and production
values. Both refer to the craft and ingenuity of movie-making and we would expect ‘low
quality’ movies to include so-called ‘B-movies’, pornographic and ‘exploitation’ films. To
verify this notion, we conducted the probit regression reported in Table 2. Here, the response
is ordered in three categories: ‘winner’, ‘nominated’ and ‘not nominated’ for ‘Best Picture’and ‘Best Director’ at the Academy Awards. The predictor is the movie’s estimated δ1 score,
which is significant for both regressions at the p < 0.01 level. We obtain similarly significant
results when we use the Golden Globe ‘Best Motion Picture: Drama’ and ‘Best Director.’
18

[Table 2 about here.]In our conception, for ‘expert’ critics, quality is associated with the ‘high-mindedness’ of themovie as art, so small independent films could certainly be included within the rubric. Highquality films might well be over-represented in certain genres such as romances, dramas andthrillers rather than, say, horror or action movies. We comment on this below. In Figure
5 we plot the density (and provide a histogram) of both the critics and movie estimates inδ1 , α1 space—the dimension we claim is quality.
[Figure 5 about here.]
Notice that there is some variance in the estimates for the critics; in our interpretation, thisis due to sampling error rather than differing tastes for quality: ceteris paribus critics preferhigh quality movies, but this does not mean that, say, a higher quality comedy is preferred
to a lower quality drama.
Since we are sometimes dealing with relatively small numbers of reviews (e.g. The Skeleton Key of 2005 was reviewed by just four NSFC critics), there are reasonably large variances
associated with our estimated movie qualities too. To avoid potentially misleading inferences
then, in Table 3 we give some ranking information for the films in our sample at the 0.05,
0.5 (i.e. median) and 0.95 quantiles of their empirical cdf of the estimates for δ1 . We also
report the rottentomatoes.com aggregate (‘percent fresh’) rating for the movies and, in the
final column, the genre description words given for the movies on the site. Notice that ourδ1 dimension estimates seem to agree with the aggregate ratings from the website; moreover,the genres seem fairly uniformly spread throughout the quality distribution, suggesting that
this first dimension is indeed quality.
[Table 3 about here.]
From an initial inspection of the movies in the other dimensions δ2 and δ3 , it was not
immediately obvious what these aspects of movie criticism actually were. For example, The
19

Dreamers, a French movie that deals with the sexual awakening of three teenagers during
the strife of the 1968 Paris riots seems somewhat different in nature to Alexander, a big
budget historical epic starring Colin Farrell. Nonetheless these movies inhabit practicallythe same locations in space. We suspect an explanation lies in the nature of the first, ‘quality’,dimension of movie review. Put broadly, we would contend that ‘bad’ movies are actually
very similar to one another: a bad comedy is not funny, a bad drama is not very dramatic,
and a bad thriller does not leave one on the edge of the seat. Once these defining elements
are removed, the movies appear almost identical, whatever one’s initial spatial preferences
might have been. As an analogy, suppose one restaurant critic enjoys seafood, while anotherenjoys pasta-based meals. Also suppose that both are served multiple dishes of each typethat are heavily over-salted. We suspect that the original (latent) preferences will be nonobservable, because the critics will dislike everything they receive. Here then, we suspectthat the failure to select on (high) quality movies tends to disguise any spatial patterns in
the data.
[Figure 6 about here.]
In Figure 6 we attempt to ameliorate this problem by presenting only those movies (withat least 15 reviews) that are ‘high’ quality. For present purposes this refers to those films thatreceived a δ1 score above the 80th percentile of all values of δ1 . In the figure, we also denotethe (first) genre description of the movie as provided by Rotten Tomatoes, using different
colors and plotting characters.
We now note several patterns that were unapparent before. First, movies of a similar
genre appear in groups, running broadly north-west to south-east across the plot. In particular, in the right, bottom corner, foreign films (open triangles) cluster. North west of thesecome the dramas (filled circles). Running in a north-south band to the west of the dramas are the comedies, interspersed with the action/adventure pictures. The science-fiction
fantasy movies (filled diamonds) appear to the west of the other movie types. In general,
20

drama movies score relatively highly on δ3 (and this is also true of foreign films), and havehigher δ2 values also. By contrast, science-fiction fantasy films are low on δ2 while comedies
are somewhere between the two. Comedies though, tend to have lower δ3 scores. Action
adventure movies are similar to comedies in this regard
To construct Figure 7, we took a different tack: here, the movies are colored and demarcated by their Motion Picture Association of America rating. As can be seen from thefigure, the bulk of the ratings are either R, which denotes that any viewer under 17 years ofage requires an accompanying parent or guardian, or PG-13 which denotes movies for which“Parents [are] Strongly Cautioned” and that might be inappropriate for children under 13
years of age. Broadly speaking, the R rated movies lie predominantly to the north and eastof the PG and PG-13 movies which themselves run in a broad band from the west to theeast and south of the graphic. As a result, the more family-friendly pictures tend to scorelower on the δ3 axis, and although they are somewhat similar regarding δ2 . The ‘unrated’
movies help confirm this idea: generally lying to the north and east of the PG and PG-13
films, they include Born into Brothels which deals with the realities of child prostitution andCapturing the Friedmans which is a documentary concerning a father and son charged withchild abuse. Presumably, neither of these films is suitable for minors.
[Figure 7 about here.]
Based on our assessment of Figure 6 and Figure 7, we present a combined graphic with ourinterpretation of the dimensions in Figure 8.
[Figure 8 about here.]
We label the west of the graphic as ‘nerds’, denoting that movies in this area are popularamong sci-fi fans. To the north-east of the plot, we denote the area as ‘art-house’ to capture
the fact that movies in this zone of the graphic might appeal to fans of (possibly pretentious,‘deep’ and emotional) ‘art-house’ style pictures: The Dreamers, In the Bedroom and Spider
21

all reside in this general direction. By contrast, to the south of the plot, we denote the area
as ‘jocks’ and the movies here are predominantly action-adventure/comedy combinations:
we think Gladiator and Anger Management would appeal to such fans. Overlayed on thisplot are two descriptors that refer to the ratings of the movies: ‘adult entertainment’ refers(broadly) to films that receive at least an R rating, while ‘family fun’ refers to all othermovies. Now that we have gone some way to establishing the dimensions of movie criticism,the next section analyzes the effects of these judgements on movie success.

5

The Effect of Movie Reviews

We believe that movie critics, via their reviews, have a perceptible effect on the successof movie performance. In this section we measure that performance as ‘profit’ which we
define as the difference between (the log of) a film’s gross in the United States and the(log of) a film’s production budget. We used data obtained from The Numbers website
http://www.the-numbers.com/.

The general theoretical assumption is that that film-makers seek to maximize revenue minus costs. In the subsequent section, we will report
our findings on the relationship between movie reviews and opening revenues.
In addition to the reviews which are operationalized via our estimated δ̂, we have several
other predictors to act as ‘controls’: rating, which is a dummy for the MPAA rating themovie received; create, which is a dummy denoting the creative type of the movie: ‘Contemporary Fiction’, ‘Factual’ and so on. We use a production type dummy (prod.dum) which
includes categories like ‘live action’ or ‘stop motion animation’; a genre dummy (genre.dum)
which denotes the movie’s primary genre, such as ‘drama’ or ‘romance’. We also record themovie’s initial release in terms of the number of screens it was shown at when opening(init.theat) and its ‘maximum’ release in terms of the total number of screens it showedon during its entire theater run (max.theat) as well as using a dummy (holiday) to account22

for possible profit variation due to the film’s opening falling on a holiday. By including thesevariables in the estimation, some of which are surely contributing to the rating δs, we provide
a more stringent test of any hypothesized relationship between reviews and box office success;that is, we are attempting to convince the skeptical reader that the δ scores are not simply
proxies for more easily available, and better theoretically justified predictors. We thus hope
to partially rule out the possibility that spurious correlations are driving any association we
see in practice.
In Table 4 (on the left hand side) we report OLS results for our first model that includes
all movies for which (complete) data is available; since the coefficients and other details on
the controls ar not of current interest, we drop them, though readers can contact us directly
if they wish to view them.
[Table 4 about here.]Interestingly, δ1 is the only significant predictor for movie success. Recall that δ1 is essentiallymovie quality, so a positive coefficient makes sense: the better the critics thought the movie
was, the better it does at the box-office.
We were surprised to see that neither δ2 (which we think is related to ‘nerdiness’) andδ3 (which we think connotes ‘jockness’ and/or ‘art-houseness’) is significant. We suspectedthough, that NSFC critics are not to everyone’s tastes: they might not reflect the ‘general’
intended audiences for all the films. We thus split our sample into two parts: ‘wide-release’movies that (by our definition) showed on at least 600 screens at the peak of their theater
run, and ‘independent’ films that showed on less than 600 screens. To clarify, note thatthe industry standard defines a ‘wide-release’ as any film receiving an initial release of atleast 600 screens. Problematically, some studios might release films for an initially ‘limited’
number of theaters to either (a) ensure their movie is eligible for Academy Awards (which
requires it be released in a particular time frame for a given year) or to (b) ‘test the waters’
for a movie that might do poorly. We wanted to avoid counting such films as ‘independent’.
23

The second column of Table 4 reports the wide-release regression: in practice, δ1 has
an increased p-value, and is no longer a predictor at the same significance level as before.
This makes some sense if we regard the NSFC critics as being particular indicative of niche
appeal.
The third column of Table 4 confirms these ideas: we now see that all the componentsof the δ̂ estimate are significant at conventional levels for independent movies. Interestingly,‘nerdiness’ (a low δ2 value) is associated with more profitable films, and in fact, the coefficientis larger than previously. Now too, δ3 is a significant predictor, although we note that more‘jock’ movies tend to do better at the box office (relative to ‘art-house’ movies).
Broadly speaking, our results imply that the NSFC critical reviews are either disproportionately influential in convincing independent movie fans, or disproportionately representative of them. Neither is particularly surprising: these critics are known for their expertiseand presumably more ‘refined’ tastes (in the same sense that a restaurant critic will probably
not recommend a fast food joint as his top choice), so we expect their views to resonate withmore selective audiences.

5.1

Movie Reviews and Opening Weekend Revenues

Independent movies—those which have a relatively small theater circulation as defined
above—typically spend much less on advertising their film product than large-scale ‘Hollywood’ wide-releases. In part, this is a necessary feature of low budgets. A consequence isthat we expect wide-release ‘blockbuster’ pictures to have much larger ‘opening weekends’
than independent movies, as audiences flock to theaters to see the latest release having been
influenced by heavy publicity campaigns. We might also anticipate a different relationshipbetween movie reviews and this opening revenue.
We defined our dependent variable as (the log of) the revenue made by movies between
their opening Thursday (we look only at movies which did indeed open on a Thursday)24and the following Sunday. Again, we had a battery of controls as described above. In thebottom portion of Table 4 we report the regression coefficients for the δ̂ we estimated forthe movies. As can be seen, the movie quality dimension (δ1 ) is not a helpful predictor for
opening weekends of ‘blockbusters’ (column 4), yet the ‘jock’ dimension (δ3 ) appears to bestatistically significant.
In the fifth column of Table 4 we look at the more narrowly released ‘independent’ movies.
Notice from the table that, now, the movie quality predictor δ1 is a significant predictor of
opening revenue, but that the other two, more substantive dimensions, are not.
All in all, it seems that opening weekends are differently structured across movie types:
independent audiences need to believe the movie is high quality, whereas those seeing widerelease pictures are much less concerned. In part, we suspect this is due to the independent
producers inability to advertise and generate ‘buzz’ for the films before the first weekend ofviewing: instead, they must rely on solid reviews and helpful word-of-mouth.

6

Discussion

This paper developed a new ‘utility threshold model’ for estimating item response parametersof interest for movie critics and the films they review. We argued that a three dimensional
spatial model was most appropriate and that the most important dimension represented
movie ‘quality’, for which, universally, ‘more’ is preferred to ‘less’. We presented evidence
that such movie reviews are predictors of the financial success of movies, and that this effectis particularly strong for independent films.
In some IRT applications, notably educational testing, it makes sense to think of subjects
and items in the same one-dimensional space: a test question has a particular ‘difficulty’
and a test-taker has an ‘ability’ on the same measurement line. In multi -dimensional spatial models where individuals make a binary choice—such as ‘ideal point estimation’ in25

legislatures—items and subjects cannot usually be placed in the same space. Such models
typically have micro-foundations in which actors make pairwise comparisons between two
available alternatives (say, the ‘status quo’ and a legislative proposal) and select their preferred option. This is clearly not the case for critics: they choose to recommend a movie ornot, without any attendant ‘default’ outcome. In light of this, we designed an approach withhybrid qualities: critics and movies can be located in similar (multidimensional) spaces and
we are able to estimate individual ‘quality’ thresholds for the critics.
There are several avenues for further research. Clearly, most consumer-advice critics operate in similar ways to our movie-reviewers: restaurants, books, paintings, exhibits and soon are ‘experienced’ and then a judgement passed. More broadly, most ‘satisfaction survey’type exercises in marketing would yield data amenable to such analysis. We note that ourframework can easily be extended to the case where individuals report multiple levels of satisfaction by incorporating more than one utility threshold. This would allow applications of
our estimator to Likert scale data. In contrast to approaches relying on principal component
analysis and related techniques, our estimator will produce estimates of product characteristics and rater ideal points in the same multidimensional space. In political science, promising
applications include legislative cosponsorship and approval voting. Both of these have beenstudied to some degree using existing scaling techniques (Talbert and Potoski, 2002; Laslier,2005), but we believe our approach can improve on these results by differentiating between
spatial dimensions and heterogeneity in utility thresholds (following our argument in Section 3.2), and by providing estimates of the locations of bills and legislators, and voters andcandidates, in the same multidimensional space.

A

Identification of the Utility Threshold ModelIn this section we provide conditions that ensure that the utility threshold model is identified.

26

Proposition 1 Suppose that αc = ec where ec is a unit vector for c ∈ {1, . . . , D} andαD+1 = 0 and W 0 is a symmetric and positive definite matrix. Suppose that F is strictlyincreasing, that the vectors {δ m,0 − δ m0 ,0 }m,m0 span RD , and for any ω ∈ RD ,[(δ m,0 + δ m0 ,0 )0 (W 0 W −1 W 0 − W 0 ) + 2ω 0 W −1 W 0 ](δ m,0 − δ m0 ,0 ) = 0 for all m, m0 (12)holds if and only if W = W 0 . Then there does not exist a parameter vector (α, ū, δ, W ) for
which (α, ū, δ, W ) 6= (α0 , ū0 , δ 0 , W 0 ) with αc = ec for c = 1, . . . , D and αD+1 = 0 suchthat
F (ūc + (αc − δ m )0 W (αc − δ m )) = F (ūc,0 + (αc,0 − δ m,0 )0 W 0 (αc,0 − δ m,0 ))(13)

for all c, m holds.
The restrictiveness of (12) is not immediately apparent, but the one-dimensional case is instructive. When D = 1, we have, (δ m,0 + δ m0 ,0 )(W 0 − W ) + 2ω = 0 for m, m0 such that
δ m,0 6= δ m0 ,0 . If there are at least two distinct values of δ m,0 + δ m0 ,0 , then it follows thatW 0 = W is the only possible solution to this system. Clearly, this is a very weak condition.
In the multidimensional case, it is harder to reduce the condition in this way, but the condition is nonetheless likely to hold since we have a large number of equations (DM (M + 1)/2)
and very few free variables (D(D + 1)/2).

Proof of Proposition 1: Consider any (α, ū, δ, W ) with αc = ec for c ∈ {1, . . . , D}and αD+1 = 0, where (13) holds. We show that such a point must satisfy (α, ū, δ, W ) =(α0 , ū0 , δ 0 , W 0 ). Since F is strictly increasing, Equation (13) is equivalent to:ūc + (αc − δ m )0 W (αc − δ m ) = ūc,0 + (αc,0 − δ m,0 )0 W 0 (αc,0 − δ m,0 ) ∀ c, m.

27

(14)

Factoring out (14), we obtain00ūc +αc0 W αc −2αc 0 W δ m +δ m 0 W δ m = ūc,0 +αc,0W 0 αc,0 −2αc,0W 0 δ m,0 +δ m0 W 0 δ m,0 ∀ c, m

(15)
0
0
ūc +αc0 W αc −2αc 0 W δ m 0 +δ m 0 W δ m 0 = ūc,0 +αc,0W 0 αc,0 −2αc,0W 0 δ m 0 ,0 +δ m0 0 W 0 δ m 0 ,0 ∀ c, m.

(16)
Subtracting (16) from (15) yields, ∀ m, m0 ,−2αc 0 W δ m + 2αc 0 W δ m0 + δ m 0 W δ m − δ m0 0 W δ m00
= −2αc 0 ,0 W 0 δ m,0 + 2αc,0W 0 δ m0 ,0 + δ 0m,0 W 0 , δ m,0 − δ m0 0 ,0 W 0 δ m0 ,0 . (17)When c = D + 1, we obtain0
δ m0 W δ m − δ m0 0 W δ m0 = δ m,0W 0 δ m,0 − δ m0 0 ,0 W 0 δ m0 ,0 ∀ m, m0 .

(18)

Plugging (18) into (17), we obtain
α0c W (δ m − δ m0 ) = α0c,0 W 0 (δ m,0 − δ m0 ,0 ) ∀ c, m, m0 .

(19)

When c ∈ {1, . . . , D}, Equation (19) yields
e0c W (δ m0 − δ m0 ) = e0c W 0 (δ m,0 − δ m0 ,0 ) ∀ c ∈ {1, . . . , D} and m, m0 .

(20)

Stacking these by column, we obtain
W (δ m0 − δ m ) = W 0 (δ m,0 − δ m0 ,0 ) ∀ m, m0 .

28

(21)

Plugging Equation (21) into (19), we have0
α0c W 0 (δ m0 ,0 − δ m,0 ) = αc,0W 0 (δ m,0 − δ m0 ,0 ) ∀ c, m, m0 .

(22)

Since this must hold for all m, W 0 has full rank, and the vectors {δ m,0 − δ m0 ,0 }m,m0 span
RD we have thatαc = αc,0 ∀ c.

(23)

Now plug Equation (23) into Equation (15) to obtainūc +α0c,0 W αc,0 −2α0c,0 W δ m +δ 0m W δ m = ūc,0 +α0c,0 W 0 αc,0 −2α0c,0 W 0 δ m,0 +δ 0m,0 W 0 δ m,0 ∀ c, m.
(24)
Using c = D + 1 we obtain,ūD+1 + δ 0m W δ m = ūD+1,0 + δ 0m,0 W 0 δ m,0 ∀ m .

(25)

We can subtract (25) from (24) to obtainūc − ūD+1 +α0c,0 W αc,0 −2α0c,0 W δ m = ūc,0 − ūD+1,0 +α0c,0 W 0 αc,0 −2α0c,0 W 0 δ m,0 ∀ c, m. (26)When c ∈ {1, . . . D}, we obtain,1e0c W δ m = (ūc −ūc,0 −ūD+1 +ūD+1,0 +[W ]c,c −[W 0 ]c,c )+e0c W 0 δ m,0 ∀ c ∈ {1, . . . , D} and m, m0 ,2(27)
where [A]i,j denotes the element in the ith row of the jth column of the matrix A. Stacking
these by column, we obtain,11
δ m = W −1 ω + W −1 diag{W } − W −1 diag{W 0 } + W −1 W 0 δ m,0 ∀ m,
2
2
29

(28)

where,1ωc = (ūc − ūc,0 − ūD+1 + ūD+1,0 ) for c ∈ {1, . . . , D}.
2(29)

We can plug (28) into (18) to obtain[(δ m,0 + δ m0 ,0 )0 (W 0 W −1 W 0 − W 0 ) + 2ω 0 W −1 W 0 ](δ m,0 − δ m0 ,0 ) = 0 ∀ m, m0 .

(30)

By assumption, this is uniquely solved with,

W = W 0.

(31)

ω 0 (δm,0 − δm0 ,0 ) = 0 ∀ m, m0 .

(32)

We can plug (31) into (30) to obtain,

Since this must hold for all m, m0 and the vectors {δ m,0 − δ m0 ,0 }m,m0 span RD , we must have
ω = 0. This implies thatδ m = δ m,0 ∀ m.
Plugging (31) and (33) into (24), we obtain ūc = ūc,0 , thus proving the result.

(33)¥

References
Ainslie, Andrew, Xavier Drèze and Fred Zufryden. 2005. “Modeling Movie Life Cycles andMarket Share.” Marketing Science 24(3):508–17.
Albert, James H. 1992. “Bayesian Estimation of Normal Ogive Item Response Curves Using
Gibbs Sampling.” Journal of Educational and Behavioral Statistics 17(3):251–269.
30

Anand, Bharat N and Dmitri Byzalov. 2008. “Spatial Competition in Cable News: Where
are Larry King and OReilly Located in Latent Attribute Space?” Working Paper Harvard
University.
Andrich, D. 1988. “The Application of an Unfolding Model of the PIRT Type to the Measurement of Attitude.” Applied Psychological Measurement 12:33–51.
Andrich, David and Guanzhong Luo. 1993. “A Hyperbolic Cosine Latent Trait Model for
Unfolding Dichotomous Single-Stimulus Responses.” Applied Psychological Measurement
17:253–276.
Beguin, A.A. and C.A.W. Glas. 2001. “MCMC estimation and some fit analysis of multidimensional IRT models.” Psychometrika 66:471–488.
Blumer, Herbert. 1933. Movies and Conduct. New York: Macmillan.
Bock, R. D. and M. Aitken. 1981. “Marginal Maximum Likelihood Estimation of Item
Parameters: An Application of the EM Algorithm.” Psychometrica 46:443–459.
Bock, R and M Lieberman. 1970. “Fitting a response curve model for dichotomously scoreditems.” Psychometrika 35:179–198.
Clinton, Joshua, Simon Jackman and Douglas Rivers. 2004. “The Statistical Analysis of Roll
Call Data.” American Political Science Review 98(2).
Coombs, Clyde. 1964. A Theory of Data. New York: Wiley.
DeSarbo, Wayne S. and Donna L. Hoffman. 1987. “Constructing MDS Joint Spaces from
Binary Choice Data: A Multidimensional Unfolding Threshold Model for Marketing Research.” Journal of Marketing Research 24:40–54.

31

Eliashberg, Jehoshua and Steven M. Shugan. 1997. “Film Critics: Influencers or Predictors?”
Journal of Marketing 61(2):68–78.
Elsworthin, Catherine. 2005. “Sony to pay $1.5m for film hoax.” (Dublin) Independent
August 5.
Firth, David. 1993. “Bias reduction of maximum likelihood estimates.” Biometrika 80:27–38.
Goettler, Ronald L. and Ron Shachar. 2001. “Spatial Competition in the Network Television
Industry.” RAND Journal of Economics 32:624–656.
Hambleton, Ronald K., H. Swaminathan and H. Jane Rogers. 1991. Fundamentals of Item
Response Theory. Newbury Park, CA: Sage Press.
Heckman, James. 1981. In Structural Analysis of Discrete Data With Econometric Applications, ed. C. Manski and D. McFadden. Cambridge, MA: MIT Press chapter The
Incidental Parameters Problem and the Problem of Initial Conditions in Estimating a
Discrete Time-Discrete Data Stochastic Process and Some Monte Carlo Evidence.
Hoijtink, H. 1990. “A Latent Trait Model for Dichotomous Choice Data.” Pychometrika
55:641–656.
Hoijtink, H. 1991. “The measurement of latent traits by proximity items.” Applied Psychological Measurement 15:153–170.
Hollinger, Hy. 2007. “MPA study: Brighter Picture for Movie Industry.” Hollywood Reporter
June 15.
Kamakura, Wagner A. and Rajendra K. Srivastava. 1986. “An Ideal-Point Probabilistic
Choice Model for Heterogeneous Preferences.” Marketing Science 5:199–218.

32

Kracauer, Stanley. 1957. From Caligari to Hitler: A Psychological History of the German
Film. Princeton, NJ: Princeton University Press.
Laslier, Jean-Francois. 2005. “Spatial Approval Voting.” Political Analysis 14(2):160–185.
Leenen, Iwin and Iven Van Mechelen. 2004. “A Conjunctive Parallelogram Model for Pick
Any N Data.” Psychometrika 69:401–420.
Lord, Frederic M. 1980. Applications of Item Response Theory To Practical Testing Problems.
Mahwah NJ: Lawrence Erlbaum Associates.
Martin, Andrew and Kevin Quinn. 2001. “Dynamic Ideal Point Estimation via Markov
Chain Monte Carlo for the US Supreme Court, 1953–1999.” Political Analysis 10(2).
Maydeu-Olivares, Albert, Adolfo Hernandez and Roderick P. McDonald. 2006. “A Multidimensional Ideal Point Item Response Theory Model for Binary Data.” Multivariate
Behavioral Research 41:445–471.
Mulvey, Laura. 1975. “Visual Pleasure and Narrative Cinema.” Screen 16(3):6–18.
Neelamegham, Ramya and Pradeep Chintagunta. 1999. “A Bayesian Model to Forecast
New Product Performance in Domestic and International Markets.” Marketing Science18(2):115–136.
Ostini, Remo and Michael Nering. 2006. Polytomous Item Response Theory Models (Quantitative Applications in the Social Sciences). Thousand Oaks, CA: Sage Publications, Inc.
Poole, Keith. 2005. Spatial Models of Parliamentary Voting. Cambridge: Cambridge University Press.
Poole, Keith and Howard Rosenthal. 1991. “Patterns of Congressional Voting.” American
Journal of Political Science 35:228–278.
33

Poole, Keith and Howard Rosenthal. 1997. Congress: A Political Economic History. New
York: Oxford University Press.
Rasch, Georg. 1961. Probabilistic Models for Some Intelligence and Attainment Tests. Copenhagen: Danish Institute for Educational Research.
Riesman, David, Revel Denny and Nathan Glazer. 1968. The Lonely Crowd. New Haven,
CT: Yale University Press.
Smith, Scott. 1998. The Film 100: A Ranking of the Most Influential People in the Historyof the Movies. Yucca Valley, CA: Citadel.
Takane, Yohsio. 1996. “An Item Response Model for Multidimensional Analysis of MultipleChoice Data.” Behaviormetrika 23:153–167.
Talbert, Jeffery C. and Matthew Potoski. 2002. “Setting the Legislative Agenda: The Dimensional Structure of Bill Cosponsoring and Floor Voting.” Journal of Politics 64(3):864–891.
van der Linden, Wim J and Ronald K. Hambleton. 1997. Handbook of Modern Item Response
Theory. New York: Springer chapter Item Response Theory: Brief History, Common
Models, and Extensions.

34

0.6
0.3

u = 0.5

0.2

Probability

0.4

0.5

u=0

0.0

0.1

u=1

−2

−1

0

1

2

α −δ

Figure 1: The ‘trace line’ from keeping the characteristics of the movie fixed (at δ̄) while (1) varying the
spatial preference of the critic (α) and (2) varying the critic’s utility threshold (ū).

35z = 2 (’approve’)

j
uk

k

z’m

u

h

z = 1 (’disapprove’)

0

1

2

3

4

5

6

α

Figure 2: Critics with normal, homoscedastic error terms—and different spatial preferences (α)—

0.20
0.15
0.00

0.05

0.10

Density

0.25

0.30

0.35

contemplate the same movie: shaded areas correspond to disapproval.

−6

−5

−4

−3

−2

−1

0

u

Figure 3: Density of estimated critic threshold utilities (ū)
36

1
0
−1
−2
−3

δ 2α 2

−1

0

1

2

3

2

3

0
−1
−2

δ 3α 31δ 1α 1

−1

0

1

0
−1
−2

δ 3α 31δ 1α 1

−3

−2

−1

0

1

δ 2α 2

Figure 4: Scatter-plots for each of the three dimensions against the others. Movies are circular points,
critics are dark triangles. Notice that the two groups show least overlap along the δ1 , α1 axis.

37

300
200

Frequency

100
0
−4

−2

0

2

4

6

2

4

6

0.2
0.0

Density0.4

δ 1α 1−4−20δ 1α 1Figure 5: Histogram of movies (light color) and critics (dark color) in first dimension of model. We contend
that this dimension is movie quality.

38

1

2

Full Frontal

The Dreamers
Spider0

Apocalypto
Charlie and the Chocolate Factory
A Scanner Darkly
In the Bedroom
I Heart Huckabees
American Splendor
Vera Drake
Born Into Brothels
Capturing the Friedmans

δ3Girl with a Pearl Earring
The Brothers Grimm

Morvern Callar

Minority Report
Lost in Translation
Harry Potter and the Prisoner of Azkaban−1

Gladiator

An Inconvenient Truth
The Quiet American

House of Flying Daggers
Harry Potter and the Goblet of FireThe Fountain
Windtalkers−2

The Dancer Upstairs
Paradise Now
Thriller
Action/Adventure
Comedies
Education/General Interest
DramasChildrens
Science−Fiction/Fantasy
Foreign Films

Anger Management−3

The Family Stone

−5

−4

−3

−2

−1

0

1

δ2
Figure 6: Scatterplot of movies in δ2 and δ3 space, plotting character and shade denote genres. Movies
have 15 reviews or more, and are ‘high quality’.

39

2

Full Frontal

1

The Dreamers
SpiderCharlie and the Chocolate Factory
I Heart Huckabees
Vera Drake

Apocalypto
A Scanner Darkly
American Splendor

0

Born Into Brothels

δ3

Pieces of April
The Brothers Grimm
Minority Report

Capturing the Friedmans
SolarisMonsoon Wedding
A History of Violence
Sideways−1

Gladiator

SarabandThe Incredibles

The Fountain

The Illusionist

The Dancer Upstairs
Paradise Now−2Windtalkers

R
PG
PG−13
no rating
NC−17

Anger Management−3

The Family Stone

−5

−4

−3

−2

−1

0

1

δ2
Figure 7: Scatterplot of movies in δ2 and δ3 space, plotting character and shade denote MPAA rating.
Movies have 15 reviews or more, and are ‘high quality’.

40

2
1

art−housers

te

ul

ad

0

ta

er

nt
m

δ3

in
en
t

m
fa

nerds

−2

n

fu

−1

ily

−3

jocks

−5

−4

−3

−2

−1

0

1

δ2

Figure 8: Scatterplot of movies in δ2 and δ3 space, with summary description. Movies have 15 reviews or
more, and are ‘high quality’.

41

Geo Mean Prob(in sample)
Geo Mean Prob
(out of sample)

D=0
53.0%

D=1
66.2%

D=2
71.1%

D=3
75.2%

D=4
79.1%

D=5
82.4%

D=6
84.7%

D=7
86.6%

D=8
87.9%

54.3%

63.8%

63.9%

64.6%

62.4%

64.1%

63.8%

63.3%

64.0%

Table 1: Goodness-of-fit statistics for each model (dimensions 0 through 8).

δ1Best Picture (AA)Best Director (AA)Best Drama (GG)

Best Director (GG)

0.672 [0.144]

0.714 [0.141]

0.645 [0.141]

0.645 [0.152]

Table 2: Predicting ‘Best Director’ and ‘Best Picture’ Academy Award (AA) and Golden Globe (GG)
winners and nominees with ordered probit. Predictor is δ1 [standard error]. Emboldened coefficients are
significant at p < 0.01 level.

42Quantile

Title

Year

δb1

% ‘fresh’

0.95

Lost in Translation
Kontroll
Primer
The Last King of Scotland
This Film is Not Yet Rated

2003
2005
2004
2006
2006

1.23
1.223
1.22
1.22
1.208

95
81
72
88
84

DramasForeign Films
DramasDramasComedyCaptain Corelli’s Mandolin
Blood Work
Veronica Guerin
Hearts in Atlantis
The Low Down
Birth

2001
2002
2003
2001
2001
2004

−0.08
−0.08
−0.08
−0.08
−0.07
−0.07

28
56
52
48
60
39

DramasDramasDramasDramasComedies
DramasJuwanna Mann
Bulletproof Monk
First Daughter
Jungle Book 2
Greenfingers
Dragonfly

2002
2003
2004
2003
2001
2002

−1.57
−1.58
−1.58
−1.58
−1.58
−1.58

9
22
9
20
47
7

Comedies
Action/Adventure
Comedies
Childrens
DramasDramasMedian

0.05

Genre

Table 3: Movies at and around the 0.05, median and 0.95 quantiles of the empirical CDF of δb1 . Finalcolumns are Rotten Tomatoes aggregate rating and genre description from Rotten Tomatoes.

ProfitOpening Weekend

All Movies
Est[SE]‘Wide release’
Est[SE]‘Independent’Est[SE]‘Wide release’
Est[SE]‘Independent’Est[SE]

δ10.154 [0.057]

0.179 [0.100]

0.140 [0.071]−0.058 [0.131]−0.271 [0.094]

δ2−0.057 [0.055]

0.042 [0.092]−0.116 [0.069]−0.088 [0.125]−0.017 [0.091]

δ3−0.066 [0.052]

0.019 [0.086]−0.125 [0.066]

0.263 [0.121]

0.081 [0.087]

Table 4: OLS results: top table are coefficients [Standard Errors] predicting profit (logged movie revenue
minus logged movie cost). Dependent variable in right-side portion refers is opening weekend receipts.
Emboldened coefficients are significant as p < 0.10 level.

43