How to make the best Movie?
By: M Humza Awan
Tags: Causal Inference, Movies Prediction, Linear Regression, Towards Data Science

Photo by Jakob Owens on Unsplash

There are several thousand movies produced in the World every year. Some of them go to win the Oscars and others are lost in the abyss of movie charts. Almost all of the movies are produced on the basis of subjective assessment of what is best for a good movie. There are some wide known actors and producers whose presence in the credits make the movie shine in the market. All these factors are subjective, and no one can say for sure how this movie will perform on the box office.

We felt like there should be a method to this madness. Continuing our work from “3 decades of cinema : what makes a movie successful?”, we tried to predict what type of a movie will be the best scoring in the industry.

Summer Blockbusters?

After doing EDA on our dataset, we saw in our dataset that movies released in “Summer” are making much revenue than movies released in other seasons but ‘Other movies revenue are closer to revenue of movies released in ‘Summer’. So, we decided to do the hypothesis testing on our observation to see which season movies make greater revenue.

Our dataset had the season column and we decided to divide it into “Summer” and “Other” season categories and added the TF column that shows True for Summer and False for ‘Other. We took the mean of ‘Gross’ between groups of ‘True/False’.

Now, let’s look at the distribution of ‘Gross’ between Movies released in ‘Summers’ and ‘Other seasons. The distribution of gross for movies that released in ‘Summer’ are slightly right to distribution corresponding to ‘Other’ movies.

So the question raises of whether the difference reflects just chance variation or a difference in distributions in greater number of movies. So, our main goal is to see whether our analysis is correct or is it just because of the movies selected in our dataset are of ‘Summer’ having greater revenue.

Hypothesis:

That our dataset represents any number of movies.

Null hypothesis

In the dataset, movies released in ‘Summer’ earn more revenue than movies released in other seasons.

Alternative Hypothesis:

In our dataset, movies released in ‘Summer’ do not earn more revenue. So, our gross will not depend on any season.

Test Statistic:

The alternative hypothesis compares the average gross of 2 groups and says that the movies released in ‘Summer’ earn more revenue.

The observed value is calculated by taking the difference in average gross of 2 groups.

Predicting the statistic under the Null Hypothesis

We have used random permutation for predicting the statistic under Null Hypothesis. We have drawn the random sample without replacement and sample include all rows of dataframe in random order. We have simplified the dataframe to just ‘Gross’ and True/False representing Summer/Other.

After taking samples, we merged our sampled and original dataframe.

Now the shuffled dataframe has average gross for both groups and their difference is 6805250.25

We decided to take samples 3000 times and see the difference in average gross for both groups and plotted their histogram.

Conclusion: The histogram shows the distribution of 3000 values. It is the empirical distribution of the test statistic simulated under the null hypothesis. This is a prediction about the test statistic, based on the null hypothesis.

Our P value is 0 and so this shows that Null Hypothesis is wrong and Alternate Hypothesis is Correct so we concluded that Movies gross doesn’t depend on which season movies are released.

Does Budget guarantee a high score?

We decided to see if budget of a movie impacts it’s score

Let’s do some basic EDA:

We divided the movies in two groups based on score

Good movies: score 8 or above

Not-so-good movies: score below 8

Average budget for good movies: $59.8 Million

Average budget for Not-so-good movies: $36.8 Million

This clearly shows that high scoring movies generally have a higher budget, but does that mean high budget will yield a high score?

We use statistical inference to determine the answer.

Null hypothesis:

Budget does not affect movie score.

Alternative hypothesis:

Budget has an impact on movie score.

Our test statistic was the difference of mean budgets in both groups.

Budget being in millions and score in range 1–10, we had to use a normalization function of log10() to ensure proper statistical comparison.

If the null hypothesis is to be true, there shouldn’t be much difference for a distribution if we shuffle the scores.

So we form another column in the dataframe “shuffled” which contains all the scores, but shuffled.

We then use 1000 bootstrap samples and extract a distribution :

It appears that all our values are below the observed test statistic from the original data, larger values would support the alternative hypothesis however smaller means that our null hypothesis holds True! Whoops

Gods of data have bestowed their wisdom: A huge budget simply cannot guarantee you a great movie; there’s more to it than film locations and all-star casts!

Aspiring filmmakers would sure be happy to hear this.

Dataset

Our dataset contained 3726 rows and 17 features. We checked the dataset for uniqueness and if there were any nulls in the dataset. We made sure there were no nulls or non-unique items.

After that, we dropped columns which we did not need such as release date, country, name and year. In order to understand the current state of our data, as to how each of the variables is related to each other we drew a pairplot and we came to know that none of the variables are related linearly to each other. It is a good sign because that gives us a better chance of running linear regression.

There were some outliers visible in the data. These outliers may affect the ability of our model to predict correctly, so we need to eliminate those. We eliminated the movies which had less than 10,000 votes, budget less than 20,000, gross profit of less than 50,000 and runtime of less than 80 minutes. Most of the good movies do not lie in the given criteria. We left with 3011 rows even after dropping the rows by given criteria. It is 80% of our initial data which is sufficient to say that our criteria do not affect most of the data in our initial data set.

Feature Engineering

Feature Engineering was done to make our dataset conform to the requirements of linear regression. We dropped all non-numerical columns to make our data sleeker and test it by running linear regression on it. We will add these columns one by one again to see how performance of our model increases.

We needed to make dummy variables for qualitative features as linear regression only takes numerical values. Handling this much data with dummy variables may get tricky. When there are too many columns and there are not enough rows, an issue of cardinality arises. Although it still gives a high value of R² on training data but that’s flying too close to the sun re: it results in overfitting when we test. We needed to limit our number of columns to avoid these issues.

We decided to create another variable named “others”. Every column which had more than a specific number of values were classified as others. For example there were more than 50 company names. We decided to put the Pareto principle here and select only 20% of the company names and the rest of them were replaced by “other”. It helped us to restrict our number of columns.

Modeling

We were left with only 4 columns after dropping the other ones. Running regression on this data gives us an R-square of 35.8%. This is a pretty sweet R² value but we wanted better.

We added variables one by one to the model to see how the performance of our model increases. We were able to reach a maximum R Square of 46%.

The coefficient R², basically is (1 — u/v), where u is the residual sum of squares ((y_true — y_pred) ** 2).sum() and v is the total sum of squares ((y_true — y_true.mean()) ** 2).sum(). In simple words it means that the best possible score is 1.0 and it can be negative (because the model can be arbitrarily worse) but a constant model that always predicts the expected value of y, disregarding the input features, would get a R² score of 0.0.

Next we split the data into train and test data and tried linear and polynomial regression. Degree 3 polynomial regression did not perform well as well as linear regression. So, we stuck to linear regression.

It gave us a negative value with a magnitude of more than one which is clearly a very bad result. To understand how our data and our predicted results look like we drew a plot:

Our values were a little spread out as is visible in this graph. So, we decided to try different forms of linear regression like ridge and lasso.

Even though simple and ridge regression had the same mean of R-squared but Ridge was giving us better values overall and so we decided to hop onto Ridge regression for the rest of our work.

The magic of regularization! Ridge regression regularized our values which is very noticeable with the outliers. Regularization is essentially a technique used to reduce error by adding an additional penalty term in the error function. It helps avoid overfitting as well.

These are some decent plots. Our regression did a good fit. Our residual plot is mostly centered around 0 without any specific trend. And our Q-Q plot also shows that our theoretical quantities lie on or near the line for the most part. Our model is only affected by a few outliers.

This was a pretty good regression model. But we tried to do even better. We removed outliers that were affecting our model by eliminating all the movies with a score less than or equal to 4 and movies with a score greater than or equal to 8. Our dataset went from 3011 rows to 2885 rows. Not a big loss. What was next was to tune the alpha parameter for an even better R² value with Ridge Regression. We let sklearn’s RidgeCV do the job for us. RidgeCV uses cross validation to pick the best alpha from the list provided.

The ideal value of alpha that we stuck to was 18 which gave us a final R² value of 0.4608.

The final plots weren’t perfect but they were big improvements over our previous ones as is visible in the extremities and good enough to give us a final mean absolute error of 0.48003.

We then saw which features affected our score the most. The first one is very obvious because votes are directly related to score on IMDb. But the next find is very interesting because it means that a high budget does not improve the movie ratings. Runtime has a positive effect on score whereas gross revenue does not. The genre of animation has a positive affect whereas horror does not. Biography and drama are also well-liked but action movies tend to not improve the score.

But what do we know? This is just our model speaking.

So we can conclude that budget does not matter if you want to predict the best movie, but there are good chances of your movie being the best on the box office if you create and animation movie. Runtime is also critical to decide before starting off a movie that contributes to your overall IMDB Score.

There are several subjective factors still unmeasurable which our model may not incorporate but at least now we know what objective factors can influence your IMDB Score.

This blog would not have been possible without the help of Hamd Jalil, Ahmed Qazi, Abdullah, Awais Hassan, and Furqan Athar.

22100083@lums.edu.pk — Hamd Jalil

22100125@lums.edu.pk — Abdullah

22100234@lums.edu.pk — Ahmed Qazi

22100249@lums.edu.pk — Awais Hassan

22100070@lums.edu.pk — Furqan Athar