Collecting Movie Data. Using a bunch of APIs to assemble a... | by Zachary Ang | Towards Data Science

I started out with the initial premise that I could try and predict the price of an iTunes movie given its characteristics. I knew that we could get information on movies in iTunes with a well documented public API, so let’s take a look.
There are two main ways you can get movie information from the iTunes store on the iTunes API — via a keyword call or a lookup by ID. Obviously, using keywords to extract specific movies is generally not a good idea. Some keywords give you as many as 50 matches (try searching for “Harry Potter”).
A lookup by ID is not much of an improvement either. The iTunes ID trackId string is at least 10 digits long, meaning that there could be as many as 10 billion possible IDs I would have to look up one at a time. Not a good idea.
So I used the iTunes RSS Feed to narrow down my search. The snippet below defines a function collect_rss_movies() that pings the iTunes RSS Feed. I decided to rely on the RSS iTunes feed to get more up-to-date movies with complete price information. After getting the iTunes IDs from the iTunes RSS feed, I turned to the iTunes API to get the movie information from the iTunes store using the get_info_by_id() function.
I’ve also defined a settings.py file which defines the collection parameters. Here, I’m directing the function to collect data from 9 countries and across 2 main genres.
I eventually dropped the documentary titles as they looked a little obscure relative to the mainstream movies, but here’s a snippet of what a movie record looks like:
{'artistName': 'Christopher McQuarrie', 'contentAdvisoryRating': 'PG-13', 'country': 'USA', 'currency': 'USD', 'primaryGenreName': 'Action & Adventure', 'releaseDate': '2018-07-27T07:00:00Z', 'shortDescription': 'On a dangerous assignment to recover stolen plutonium, Ethan Hunt (Tom Cruise) chooses to save his', 'trackHdPrice': 19.99, 'trackHdRentalPrice': 5.99, 'trackId': 1406515547, 'trackName': 'Mission: Impossible - Fallout', 'trackNumber': 6, 'trackPrice': 14.99, 'trackRentalPrice': 5.99, 'trackTimeMillis': 8873322, 'wrapperType': 'track'}
That’s the gist of it — there are some features that I haven’t included here, but the data description is available at the official iTunes API documentation, or you can check out the data description at my Github.
Great! We’re all set to do a new exciting machine learning project. Right? But there’s so much more to a movie than that, isn’t there? So I did a little more digging as to how I could supplement this data set.
While IMdb is the authoritative source on all movie information, they don’t have a public API and I was going to run into trouble searching for the exact movie that I should collect even if I had relied on web scraping. I turned to two open databases on movies, the Open Movie Database (OMdb) and The Movie database (TMdb).
The Movie Database (TMDb) is a community built movie and TV database. Every piece of data has been added by our amazing community dating back to 2008. TMDb’s strong international focus and breadth of data is largely unmatched and something we’re incredibly proud of.
So that’s the provenance of TMdb. From OMdb’s website:
The OMDb API is a RESTful web service to obtain movie information, all content and images on the site are contributed and maintained by our users.
Although they were respectively well maintained, they each had their pros and cons, and I realised that I’d probably have to utilise both to extract the right blend of information for my feature set. Both databases did not log the iTunes ID in their database for each movie so I was going to have to rely on a title search to match each movie from the iTunes data to the movies database.
Features. OMdb had a lot more useful information than that I couldn’t find in the TMdb dataset. For instance, OMdb contained information on the movie ratings, eg. Rotten Tomatoes ratings, Metacritic score and IMdb ratings. OMdb data also recorded box office performance of the movie, which I felt would be a key feature to predict how much an iTunes movie could list for.
API capabilities. In general, the TMdb API was more capable and offered more methods to look for information. In OMdb, you could only look up movies by a keyword search or by IMdb ID. Also, doing a keyword search on the TMdb API generally yielded better results than the OMdb API.
API rate limits. OMdb had a pretty tight API rate limit of 1,000 calls per day while the TMdb API had a much higher rate limit of 4 requests per second (or ~340,000 calls per day).
So the OMdb API had more information that I needed, but the TMdb API was generally better to work with. I eventually decided that I needed to make very targeted and specific calls to the OMdb API to get movie information through a lookup by ID. OMdb indexed their movie records using the IMdb ID, so I was going to have to find the IMdb ID for every movie that was in my iTunes dataset via keyword search, which as I had mentioned, was better done through the TMdb API.
Though somewhat convoluted, I settled on the following strategy:
Parse the movie title to a search string to pass through the TMdb API, looking for an exact match. Include year of production to narrow down the search further.
Parse the movie title to a search string to pass through the TMdb API, looking for an exact match. Include year of production to narrow down the search further.
2. For exact matches, look for the IMdb ID using a lookup by TMdb ID. For results with no matches, relax the search criteria by dropping the year of production. For movie titles with mutliple matches or results, extract the IMdb from the first result from a Google search in the form of “Movie title + Year of Production + IMdb”
3. Match each movie to an IMdb ID, and lookup OMdb API for movie information for each movie.
Following this procedure, I managed to uniquely identify a rather healthy looking dataset of 603 unique movies from the OMdb API , which matched ~1,500 out of the 1,800 movies I collected on the iTunes store so a decent return overall, without delving into the messy world of entity resolution. There’s a pretty lengthy chunk of code for this, but you can also follow it at my Github repo.
I was pretty happy with the data that I had until I did some initial exploration on the OMdb data. I noticed a large chunk of missing values for box office information in my data.
I could have lived with missing ratings scores, but box office performance would have been a good feature to incorporate into my model.
And it seemed plainly obvious that I could get the information rather easily through public data sources. So I turned to Box Office Mojo, using the ever reliable requests and BeautifulSoup combo.
I found ~7,000 movies, filling all of my missing values and supplementing it with the breakdown of world-wide box office gross versus US box office gross. Now it was just a matter of putting it all together.
After all that wrangling, I have a dataset of ~1,500 movies with data merged from 4 different sources, joined by different keys and methods. They exist primarily in two csv files found in my repository, and I’ve included a broad schema for the database below.
That wraps up my post about collecting data on movies. I’ll be posting updates on the modelling and data exploration here over the next week or so. In the meanwhile, ping me if you have any feedback, suggestions or improvements to my data collection process!