Expect to Watch TV Shows From Your Childhood in 4K Thanks to AI
By: Martin Racko
Tags: 4k Video, Transformers, AI, Artificial Neural Network, TV Shows

Expect to Watch TV Shows From Your Childhood in 4K Thanks to AI

Are you rewatching your favorite childhood TV series with your kids and asking yourself — wouldn’t it be great if I could watch them in beautiful 4K? That’s a very legitimate question because the old TV shows you’re watching are mostly in SD quality. This translates into a resolution of 720x576 pixels at best, which is a measly 5% of the pixel count of a 4K screen. In practice, this means that your TV is enlarging each frame 20 times in order to fill the entire 4K screen. No wonder the picture looks blurry.

Comparison SD vs. Full HD vs. 4K UHD

Traditional remastering

A lot of older TV shows and movies like Star Trek TOS/TNG, Aladdin or the original Lion King have already been remastered into higher definition and are available on popular streaming platforms. However, the chance that your favorite childhood TV show or movie is available in at least full HD is rather slim. The reason for this is that remastering into higher definition is often a tedious and costly undertaking. First, you need to get the original 16mm or 8mm film and hope that it’s still in good shape. The film then has to be scanned digitally frame by frame. Once the film is scanned, manual color and image correction is performed. This all requires a lot of time and money, which can rarely be justified from a business perspective. You can read more about film restoration here and here.

AI to the rescue

AI, artificial neural networks and machine learning have all been strong buzzwords in the past couple of years. While often times overhyped, the advancements that artificial neural networks (ANNs) have brought to areas such as voice and image recognition or machine translation can’t be overlooked. They are in principle powerful pattern recognition algorithms that can automatically learn how to transform an input into a desired output based on an existing data set. For example, you can feed an ANN 100,000 pictures of cats and another 100,000 pictures of dogs and it will automatically learn how to distinguish those two. Writing a traditional program to distinguish cats and dogs in pictures would take months of coding and probably wouldn’t be as good as a simple ANN, which can be set up in a few days.

Since ANNs can be trained on a variety of tasks, why not use them for 4K remastering? Give an ANN enough examples of SD and 4K version of the same images or videos and it will learn how to remaster a low-quality SD input into a detailed 4K video. Remember that when converting an SD image into 4K, you’re technically turning one pixel into 20. When you watch SD content on a 4K TV, every pixel is basically enlarged 20 times and blurred to reduce jagged edges. An ANN, on the other hand, will artificially add realistic detail to the 19 newly generated pixels for each pixel of the original video. The final 4K result should be crisp with a lot of detail which was simply missing in the SD video.

You could build and train your own ANN model for 4K upscaling since all of the major neural network frameworks are open-source, but fortunately, solutions for this already exist. One such tool is Topaz Video Enhance AI by Topaz Labs. This tool lets you upscale your low-quality videos to 4K or even 8K with a couple of clicks by utilizing pretrained ANNs.

Testing 4K upscaling using AI

Topaz Video Enhance AI was tested on the DVD release of the original Transformers G1 Animated Series from 1984. This animated series has never been remastered in HD, let alone 4K, which makes it a perfect test subject (apart from the fact that it’s a great show). A mid-range Nvidia GTX 1060 graphics card took about 14 hours to upscale a 22-minute-long episode from SD to 4K, but the wait was well worth it. Below are some comparison shots from the introduction theme of season one:

Left — original SD version upscaled to 4K without AI | Right — 4K version using AI (native 4K section)

Left — original SD version upscaled to 4K without AI | Right — 4K version using AI (native 4K section)

Left — original SD version upscaled to 4K without AI | Right — 4K version using AI (native 4K section)

Left — original SD version upscaled to 4K without AI | Right — 4K version using AI (native 4K section)

The difference between the original and 4K version of the Transformers is hard to miss. The image is sharper and cleaner — something that a regular upscaler is incapable of doing. You will still see the occasional film noise and the AI model can’t improve the animation frame rate, but it certainly gives the 36-year-old Transformers a much-needed polish.

What to expect

The results of 4K upscaling using ANNs are impressive. Is it as good as traditional remastering? No, but given how simple it is to use, it’s an incredible achievement. Currently, 4K upscaling using ANNs is being utilized mostly by enthusiasts and hobbyists. The typical consumer will not see or use such tools because they are relatively slow. With the exception of the 2019 Nvidia Shield, there simply isn’t a consumer product capable of upscaling videos into 4K in real-time using sophisticated ANNs. The question therefore is, when can we expect to see such AI become mainstream? There are two possible solutions:

1. High-End Dedicated Consumer Products

We can expect to see high-end dedicated consumer products that utilize ANNs for real-time 4K and 8K upscaling in the next couple of years. These may take the shape of set-top boxes or consoles and will have to be plugged into the TV. One such already existing product is the 2019 Nvidia Shield. Later, we can expect to have AI upscaling hardware built directly into TVs. This will be especially helpful in selling 8K TVs because it practically neutralizes the lack of available 8K content.

2. Streaming Platforms Begin Utilizing ANNs

The other possibility of ANN upscaling becoming mainstream is for streaming platforms to begin utilizing this technology. The streaming platforms would have to upscale their SD and HD content into 4K or even 8K using ANNs similarly to how it’s done with existing tools. Once finished, the upscaled 4K or 8K content would be made available for everyone to stream. This process will probably require some licensing clarifications with the movie studios, but from a technical point of view, there isn’t anything which would hinder the streaming platforms from using this technology right now.

When 4K remastering using sophisticated ANNs becomes mainstream is anyone’s guess. But the results of what such technology can achieve makes us hope for a sooner rather a than later date.

Disclaimer: TRANSFORMERS and all its related characters are trademarks of Hasbro and I do not claim any right over them. I’m not related to Topaz Labs or any of its employees. Copyright Disclaimer under section 107 of the Copyright Act of 1976, allowance is made for “fair use” for purposes such as criticism, comment, news reporting, teaching, scholarship, education and research. Fair use is a use permitted by copyright statute that might otherwise be infringing.