Exam Time for a Binge Watching AI
By: Daniel Shapiro
Tags: Deep Learning, Star Trek, Machine Learning, Artificial Intelligence, Data Science

There are plenty of examples out there showing how an AI can recognize data in video. For example, emotion recognition. Let’s do something cool and different, ignoring the technical implementation details for the sake of brevity. Imagine this: You turn on the TV and then try to figure out what show you are watching, using the style and content of the video as your guide. Can we get an AI to do the same job? How well can it do this job? Well, let’s find out!

Step 1: Force our A.I. to binge watch every episode of 4 popular TV shows:

StarTrek TNG — 7 Seasons (TNG)

Battlestar Gallactica — 4 Seasons (BSG)

The West Wing — 7 Seasons (WW)

The Office — 9 Seasons (OF) Note: this is the US version, not the UK one

Step 2: Ask the AI some questions to figure out what it learned from watching all of that video.

So, the AI has binge watched all the shows. It took a few days.

Let’s ask our AI some questions. After spending several days watching these shows, and ostensibly learning by watching, we should get some decent answers. Let’s show the AI some images and video clips from the internet to discover what show it thinks these images/videos belong to.

Results and Analysis

Let’s do data science!!

Can the AI tell the difference between images from two space shows (TNG and BSG)?

Images for BSG (left) and TNG (right) that were not in the training videos

Have a look at the images in the grid above this paragraph. For the BSG images on the left, the AI guessed with 25% confidence that these are BSG. Not encouraging, but among the 4 options the AI always ranked BSG as the first or second choice for these images. Keep in mind that these images are from google searches, not the training videos.

For the TNG images on the right, the AI decided with 78% confidence that these are TNG images. This is a higher level of certainty, and a much better result than the BSG result. Among the 4 options the AI always ranked TNG as the first or second choice for these images.

Can it tell the difference between two office-based shows (WW and OF)?

Images for The Office (top) and The West Wing (bottom)

In the image grid above, “The Office” images on the top of the grid led the AI to guess with 84% confidence that these are from the show “The Office”. Among the 4 options, the AI always ranked The Office as the first or second choice for these images.

For The West Wing images on the bottom of the grid, the AI found the correct label with 35% confidence. Among the 4 options, the AI always ranked WW as the first or second choice for these images. The cast pictures in the testing data were not included in the training set. The AI associated some key actors in each show to the right TV show. For example, the picture of Joshua Malina and Mary McCormack has a camera guy in it. That picture worked out with higher confidence than any of the other WW images in the testing set. Back in the TNG images, the AI recognized Q, even though he was only in a few episodes. Occasional guest actors did seem to work, but it is more likely that the costumes and scenery were the features that helped it make a decision rather than the actor’s face.

When ordering the images according to the likelihood it belongs to a certain show (across these 4 TV shows that it binge watched), the AI always recognized the image, putting the correct label in first or second place.

Link to interactive chart All of the testing images for all 4 TV shows were ranked as the first or second result into the correct category

Now let’s dig a bit deeper. In the chart below, each column represents a test picture for The Office, and the more yellow we see, the more confident the AI was. So far looks good.

This next chart is the same thing as above, but for test images for The West Wing. The more green we see, the better. The results were not super awesome. We can see that the AI is confusing the two shows that look like office settings. It is smart enough to see that the test images are not from a space show, but struggled with the distinction between The West Wing and The Office. As we will see later on, this is less of an issue when the AI watches a video, as the video can average out mistakes made on one image frame.

We can see in the chart below, for TNG, that the AI has a very good handle on what Star Trek looks like. In this case the more red in each column, the better. The image that did not go well is the one where Captain Picard is turned into a Borg. Perhaps the AI thinks that Borg Picard looks like a Cylon and so classified this image as more BSG than TNG.

Speaking of BSG, the results in the chart below indicate that the AI did not do well discriminating between TNG and BSG. In the bar on the right, the AI thought the BSG ship was a TNG ship by about a 2:1 ratio. Also, the big orange bar in the second from the left is telling us that the fight scene between Kera Thrace and Saul Tigh looks like a scene from The Office. You need to laugh at this, because BSG is so serious, and The Office is so… not.

Does the AI recognize videos related to these TV shows?

Pushing video into a neural network is already challenging, but let’s not make it easy. Let’s show the AI a YouTube fan trailer for The Office, and cast interviews for The West Wing; one with Bradley Whitford and another with Janel Moloney (Josh and Donna).

Frames from the 3 test videos used to evaluate the AI

The results below tell us that the AI can see that the “Josh” cast interview is about The West Wing, even though this footage was not seen by the AI when it binge-watched the TV show. Even if a few frames were misunderstood (as we saw in the image by image example up above), the averaging of the confidence over many frames of video improves the performance of the AI. Strangely, the AI saw more BSG than The Office in the interview with Bradley Whitford. Perhaps this is because the background was pretty dark, like the set of a BSG episode.

Classification results for the Bradley Whitford (Josh) interview

Classification results for the Janel Moloney (Donna Moss) interview

Finally, let’s see how the AI did on a YouTube “The Office” trailer. Not bad. It was not sure if the video was The West Wing or The Office, but generally got the idea that this video is one of those “in an office” shows.

Classification results for The Office fan trailer

Conclusion

That was super fun! The work to create this article was about 40 hours of my time, and a whole lot of GPU compute time. The AI performed even better on understanding video than it did on understanding images. It would be fun to follow up and see if the AI is detecting styles or content or both. We could use style transfer to apply the style of one show to an image/video from another show and see if we can trick the AI into guessing wrong. Perhaps with dedicated facial recognition the classification would improve. If you like these ideas then contact me, and I can talk with you about collaborating on the project. If you like the article please give it some love (💗), and share it with a friend.

-Daniel

daniel@lemay.ai ← Say hi.

Lemay.ai

1(855)LEMAY-AI

Other articles you may enjoy: