Title: Sharpen Your Vision: Super-Resolution of CCTV Images Using Hugging Face Diffusers - PyImageSearch

URL Source: https://pyimagesearch.com/2024/06/03/sharpen-your-vision-super-resolution-of-cctv-images-using-hugging-face-diffusers/

Published Time: 2024-06-03T13:00:00+00:00

Markdown Content:
* * *

**Table of Contents**
---------------------

*   [Sharpen Your Vision: Super-Resolution of CCTV Images Using Hugging Face Diffusers](#h1-SR-CCTV-Using-Hugging-Face-Diffusers)
*   [Configuring Your Development Environment](#h2-Configuring-Dev-Env)
*   [Problem Statement](#h2-Problem-Statement)
*   [How Does Super-Resolution Solve This?](#h2-Solve-This)
*   [State-of-the-Art Approaches](#h2-SOA-Approaches)
    *   [Generative Adversarial Networks (GANs)](#h3-GANs)
    *   [Diffusion Models](#h3-Diffusion-Models)
*   [Implementing Diffusion-Based Upscaler Using Hugging Face 🤗 Diffusers](#h2-Implementing-Diffusion-Based-Upscaler)
    *   [Loading Our Data Using Roboflow and Hugging Face Datasets](#h3-Loading-Data-Roboflow-Hugging-Face-Datasets)
    *   [Defining Our Model and Inference Pipeline](#h3-Defining-Model-Inference-Pipeline)
    *   [Plotting Our Results](#h3-Plotting-Our-Results)
*   [The Way Forward](#h2-Way-Forward)
*   [Summary](#h2-Summary)
    *   [Citation Information](#h3-Citation)

* * *

Hello! Welcome to the 2nd part of the series on Image Super-Resolution. In the [previous tutorial](https://pyimagesearch.com/2024/04/29/unlocking-image-clarity-a-comprehensive-guide-to-super-resolution-techniques/), we took a deep-dive into traditional ways in which we covered a few concepts around interpolation, the different methods of how to perform interpolation, and how one can go about doing them with Python. We also saw some interesting applications where this can be used to bring about results in the real world.

[![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20940%20780'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2024/06/2024-06-03-featured-1.png)

This tutorial will dive into one of those applications, specifically around solving for improving the clarity of real-life CCTV images.

In this tutorial, you will learn how you can perform Image Super-resolution on real-life CCTV (Closed-Circuit Television) images using Hugging Face Diffusers.

This lesson is the last of a 2-part series on **Image Super-Resolution**:

1.  _[Unlocking Image Clarity: A Comprehensive Guide to Super-Resolution Techniques](https://pyimg.co/w4kr8)_
2.  _**[Sharpen Your Vision: Super-Resolution of CCTV Images Using Hugging Face Diffusers](https://pyimg.co/eoubf)**_ **(this tutorial)**

**To learn how to perform Image Super-Resolution,** _**just keep reading.**_

* * *

### [**Configuring Your Development Environment**](#TOC-h2-Configuring-Dev-Env)

To follow this guide, you need to have OpenCV and the Hugging Face ecosystem libraries installed on your system.

Luckily, all these libraries are pip-installable:

$ pip install roboflow diffusers accelerate huggingface\_hub peft transformers datasets safetensors scipy bitsandbytes xformers -qqq

**If you need help configuring your development environment for OpenCV, we** _**highly recommend**_ **that you read our** _**[pip install OpenCV](https://pyimagesearch.com/2018/09/19/pip-install-opencv/)**_ **[guide](https://pyimagesearch.com/2018/09/19/pip-install-opencv/)** — it will have you up and running in minutes.

* * *

### Need Help Configuring Your Development Environment?

[![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20334'%3E%3C/svg%3E)](https://pyimagesearch.com/pyimagesearch-university/)

Having trouble configuring your development environment? Want access to pre-configured Jupyter Notebooks running on Google Colab? Be sure to join [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/) — you will be up and running with this tutorial in a matter of minutes.

All that said, are you:

*   Short on time?
*   Learning on your employer’s administratively locked system?
*   Wanting to skip the hassle of fighting with the command line, package managers, and virtual environments?
*   **Ready to run the code immediately on your Windows, macOS, or Linux system?**

Then join [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/) today!

**Gain access to Jupyter Notebooks for this tutorial and other PyImageSearch guides pre-configured to run on Google Colab’s ecosystem right in your web browser!** No installation required.

And best of all, these Jupyter Notebooks will run on Windows, macOS, and Linux!

* * *

[**Problem Statement**](#TOC-h2-Problem-Statement)
--------------------------------------------------

CCTV Cameras are ubiquitous around the world. They are important tools for surveillance and for maintaining safety in large public spaces or appropriate settings. But more often than not, these devices have a low resolution that fails to capture enough features of a face, vehicle, or object of interest. This can lead to those objects not being distinguishable enough for recognition or other downstream applications.

[](https://www.getscw.com/images/faq/moisture-infrared.jpg)Another important reason is the factor of cost. Laying down CCTV cameras across an entire city or a large area comes with huge installation and maintenance costs. For that reason, image quality is often compromised.

Hence, our problem statement would include capturing the still image outputs of these CCTV cameras and performing super-resolution on them directly. We are going to take a look at how this can be accomplished using some current State-of-the-art techniques, and then we will implement one of them.

* * *

[**How Does Super-Resolution Solve This?**](#TOC-h2-Solve-This)
---------------------------------------------------------------

Using state-of-the-art super-resolution techniques, CCTV still images can be post-processed to retain enough information that can be utilized for downstream tasks. Interpolation may not be the best choice in such real-life applications, as the lack of information within the original image only compounds the problem and makes the image worse with each change/iteration.

Current super-resolution techniques draw from their knowledge of certain scenes and their general world knowledge to synthesize new information while keeping the available information center-stage. They add synthetic information that is in line with already-existing image information so as not to distort the image or make it worse off due to the inadvertent introduction of noise through this process.

* * *

[**State-of-the-Art Approaches**](#TOC-h2-SOA-Approaches)
---------------------------------------------------------

Several state-of-the-art approaches exist to Image Super-Resolution, but two of the strongest current candidates for the same are [Generative Adversarial Networks](https://arxiv.org/abs/1406.2661) and [Diffusion Models](https://arxiv.org/abs/1503.03585). Let’s take a good look at both of them.

* * *

### [**Generative Adversarial Networks (GANs)**](#TOC-h3-GANs)

Generative Adversarial Networks, introduced first by Ian Goodfellow, are among the most popular generative models available. These models describe the generation of any information as a minimax problem, wherein a “generator” tends to create data or information that is then passed to a “discriminator” that discerns whether the data it saw was generated synthetically or is part of the real data.

In this setup, the job of the Generator is to make sure that it ends up “fooling” the Discriminator, wherein it would reduce its own loss by increasing the Discriminator’s loss. On the other hand, the Discriminator’s job was to make sure it was certain that each sample it tested was perfectly classified into real or generated with 100% certainty.

[](https://upload.wikimedia.org/wikipedia/commons/8/8b/Generative_Adversarial_Network_illustration.svg)For this reason, GANs are said to be trained in a way different from how normal neural networks are trained. This “competition” setting means that the ideal case for a GAN’s training is to have a 50% certainty on the Discriminator and a 50% certainty on the Generator. This suggests that the Discriminator is absolutely confused about which samples are real or generated, while the Generator is not strong enough to confuse the Discriminator at all times completely.

Keeping this in mind is important because even though loss values can become favorable, GANs are notoriously difficult to get good qualitative outputs from. Since the loss values of the Generator and Discriminator are essentially fighting against each other, this also means that the learning capabilities (from backpropagation of loss values through the network).

The process of performing Super-Resolution using GANs involves using the Generator to simply upscale the image with Transpose Convolutions or by using any network that produces an upscaled version, compared to the original input.

Current super-resolution GAN techniques include the SRGAN, ESRGAN, and its variants. To learn more about these techniques and their varied uses, take a look at our curated blogs on the [GAN architecture](https://pyimagesearch.com/category/gans/). If you’re interested only in Super-Resolution, though, you should check out our blogs on the [SRGAN](https://pyimagesearch.com/2022/06/06/super-resolution-generative-adversarial-networks-srgan/) and [ESRGAN](https://pyimagesearch.com/2022/06/13/enhanced-super-resolution-generative-adversarial-networks-esrgan/), featuring from-scratch implementations!

* * *

### [**Diffusion Models**](#TOC-h3-Diffusion-Models)

Diffusion Models are a new class of models that have been developed and popularized because of their strong results for unconditional and conditional generation. The first version of what we call a standard Diffusion model today was the [DDPM (Denoising Diffusion Probabilistic Model](https://arxiv.org/abs/2006.11239)[)](https://arxiv.org/abs/2006.11239), which described a simple process for performing generation.

*   **Add Noise to an Initial Image:** This entails the use of a noise scheduler. The scheduler is a function that adds random Gaussian noise to the image over multiple steps. Over time, the added noise destroys all the information in the image.
*   **Learn How to Remove the Noise:** Using a denoising model such as a UNet, we attempt to predict a denoised version of the image for a given time step. This step is performed for the same number of times as the original noising process. In some iterations of Diffusion models, this step consists of predicting the “noise” that was added for the corresponding timestep and its subsequent subtraction against the random image obtained after adding noise. Whereas, a few iterations try to predict the noise-subtracted image directly, among many other such ideas. Doing this over time should end up yielding the original image.
*   **Perform Inference:** Now that we have this denoising network trained, we can take any image of random noise and continue passing it through the denoising network until we get a realistic image.

In the landscape of Diffusion models, the next generation has been the [Latent Diffusion Model by Rombach et al. (2022) from LMU and Runway ML](https://arxiv.org/abs/2112.10752). The model we are going to utilize for our problem today is also based on the LDM approach!

The Latent Diffusion Model (LDM) approach was pioneering in how it treated the images and introduced a new technique for controlling the generation using text prompts.

LDMs implement the following approach:

*   The image is encoded into a latent space representation, which is then converted into a Tensor representation. This is done by using an image encoder, which is often a Neural Network that has been previously pre-trained on large-scale image data.
*   The process of adding noise is now performed on the latent space representation instead. This increases the efficiency of the operation as the number of operations per noise-step injection reduces drastically and does not depend on the image size anymore.
*   The Denoising network is trained to reverse this process and finally predict a latent representation that would decode to a clean image instead of directly predicting the noise or the image itself.
*   If we apply “text conditioning” (i.e., using text prompts to control the content of our image), we can pass our prompt through a text encoder, which is subsequently injected as information into the denoising network. For our current use case of super-resolution, we do not utilize this section.
*   After denoising, the final latent representation is then passed through an image decoder, which is usually another Neural network. In our case, this network accepts the latent representation and produces a higher-resolution output compared to the original input.

Now that we have a fair idea of how this works, let’s see how we can implement it using Hugging Face Diffusers!

* * *

[**Implementing Diffusion-Based Upscaler Using Hugging Face 🤗 Diffusers**](#TOC-h2-Implementing-Diffusion-Based-Upscaler)
--------------------------------------------------------------------------------------------------------------------------

This section discusses how to use the Diffusers library to achieve image super-resolution. Currently, we will utilize real-world data taken from Roboflow, but the workflow and pipeline remain fairly standard across different kinds of data.

* * *

### [**Loading Our Data Using Roboflow and Hugging Face Datasets**](#TOC-h3-Loading-Data-Roboflow-Hugging-Face-Datasets)

As always, we first get started with making all our imports and getting them in place. We specifically import the following libraries:

*   🤗Diffusers for model loading and inference
*   Roboflow for Dataset loading
*   Torch for Tensor manipulation
*   Matplotlib for Image plotting
*   Datasets for Dataset registration and manipulation
*   Numpy for Array manipulation

import os
import torch
import numpy as np
from roboflow import Roboflow
import matplotlib.pyplot as plt
from datasets import load\_dataset
from diffusers import LDMSuperResolutionPipeline

from mpl\_toolkits.axes\_grid1.inset\_locator import mark\_inset, zoomed\_inset\_axes

Now, let’s also define some constants as per our future requirements. We define the `device` constant for automatically choosing the GPU as required. We also define the model ID, which references the unique model available on Hugging Face Hub. We also define a `num_samples` variable that helps us control the number of samples we’d like to generate.

device = "cuda" if torch.cuda.is\_available() else "cpu"
model\_id = "CompVis/ldm-super-resolution-4x-openimages"
num\_samples = 3

We download our data from Roboflow. This can be done very quickly using their own loading script, which uses the Roboflow API (which requires a free API key) to download the dataset to your local machine.

We then use the `datasets` library to load the dataset in a Hugging Face-compatible dataset representation (that is also compatible with the `torch.utils.data.Dataset` APIs).

rf = Roboflow(api\_key="")
project = rf.workspace("combine-iixr2").project("retailshop-1gysu")
version = project.version(4)
dataset = version.download("yolov5")

dataset = load\_dataset(
"imagefolder", 
data\_dir="/content/RetailShop-4"
).shuffle(seed=42)

* * *

### [**Defining Our Model and Inference Pipeline**](#TOC-h3-Defining-Model-Inference-Pipeline)

Once our data has been shuffled and kept ready, we define our `pipeline` by importing it from Diffusers and by specifying the model ID, the data type to use for storing the weights, and the device we’d like to utilize for inference.

If you have access to an Ampere GPU or above, you can utilize the `torch.bfloat16` datatype to make your model more efficient.

ldm\_pipeline = LDMSuperResolutionPipeline.from\_pretrained(
    model\_id, torch\_dtype=torch.float16
    # if using an Ampere-class GPU or above, use torch.bfloat16 for benefits
)
ldm\_pipeline = ldm\_pipeline.to(device)

We now define a runner function called `pipe` that basically wraps the original pipeline inference call with our own controls and only returns the images from it.

def pipe(image):
    return ldm\_pipeline(image, num\_inference\_steps=100, eta=1).images

Now that we have our data and our model, we quickly run a loop to generate the higher-resolution images. We append them to a list for quick plotting.

res\_originals = \[\]
res\_outputs = \[\]

for idx in range(0, num\_samples):
    item = dataset\["train"\]\[idx\]
    res\_image = pipe(item\["image"\])

    res\_originals.append(item\["image"\])
    res\_outputs.append(res\_image)

* * *

### [**Plotting Our Results**](#TOC-h3-Plotting-Our-Results)

A good plotting function is always important to understand the effects of our results. Taking some inspiration from this awesome tutorial on [Keras.io](https://keras.io/examples/vision/super_resolution_sub_pixel/), we define our own plotting function that accepts the original and higher-resolution image and performs some zooming and nifty tricks to get an awesome visualization experience!

plt.rcParams\["figure.figsize"\] = (num\_samples\*5, 5)  # Adjust as needed
plt.rcParams\["figure.dpi"\] = 300

def plot\_image\_with\_zoom(original\_img, res\_img):
    original\_np\_img = np.asarray(original\_img)
    res\_np\_img = np.asarray(res\_img).squeeze(0)

    original\_np\_img = original\_np\_img.astype("float32") / 255.0
    res\_np\_img = res\_np\_img.astype("float32") / 255.0

    fig, ax = plt.subplots(1, 2, figsize=(num\_samples\*5, 5), dpi=200)

    ax\[0\].imshow(original\_np\_img)
    ax\[0\].set\_title("Original Image")

    ax\[1\].imshow(res\_np\_img)
    ax\[1\].set\_title("Superresolution Result")

    original\_axins = zoomed\_inset\_axes(ax\[0\], 2, loc=2)
    original\_axins.imshow(original\_np\_img\[::-1\], origin="lower")

    res\_axins = zoomed\_inset\_axes(ax\[1\], 2, loc=2)
    res\_axins.imshow(res\_np\_img\[::-1\], origin="lower")

    x1, x2, y1, y2 = 280, 360, 280, 360
    original\_axins.set\_xlim(x1, x2)
    original\_axins.set\_ylim(y1, y2)

    x1, x2, y1, y2 = 1120, 1440, 1120, 1440
    res\_axins.set\_xlim(x1, x2)
    res\_axins.set\_ylim(y1, y2)

    original\_axins.set\_xticks(\[\])
    original\_axins.set\_yticks(\[\])

    res\_axins.set\_xticks(\[\])
    res\_axins.set\_yticks(\[\])

    mark\_inset(ax\[0\], original\_axins, loc1=1, loc2=3, fc="none", ec="blue")
    mark\_inset(ax\[1\], res\_axins, loc1=1, loc2=3, fc="none", ec="blue")

    plt.show()

There we go! Let’s quickly use our function to plot some samples of our own.

plot\_image\_with\_zoom(res\_originals\[0\], res\_outputs\[0\])
# Iterate over each of the samples to get the result

[![Image 3](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%200%200'%3E%3C/svg%3E)](https://lh7-us.googleusercontent.com/zDh6nyMnxDWxrdnrsFcKIY_rxqNojV8a9jDfJfeiz4gIfNi5SCVqO0t201sV63CG3mrLSIFmJr8-q4Fk03tgUnlPNcYWnmmbW7pLNiLZ0Et5bPVka5fAOpAreKfCl4Z4pEehCJ7wYQJDpms-4dLI2Zk)

[![Image 4](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%200%200'%3E%3C/svg%3E)](https://lh7-us.googleusercontent.com/Kml8F4wI3o5BETD04jIdnkBN7wBQAE6M4HEV24h_Do2zRUiKIHffxMQ479zdiivVAN-Ei62MUVLTunLl94s6_UK_9whjVt40PZ0N3i7CrmLHdfWxmas1abGzfq9SvaRrbbcHnwSR6THiCyYP1-u4asU)

[![Image 5](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%200%200'%3E%3C/svg%3E)](https://lh7-us.googleusercontent.com/XNQldorWJzExCIjWwbiZevpL-0XNTrh7cYKXevceTk4zo4gDxVXUtl0TnrjxF8sR399sZ2aj9X3s2xIxkEXQTAR0AUGNG3tgRJiPczGOCW_oVuqeX25nCjjYRTNTt5nvOCvrxYOHDTPDmFNcce_i69c)

**Figure 5:** Output of Super-Resolution Model (source: image by the author).

* * *

[**The Way Forward**](#TOC-h2-Way-Forward)
------------------------------------------

Super-resolution is an exciting field. It is one of the most practical tasks available in the field of Computer Vision, which has tons of real-world applications. Advances in this field come every day, and it’s important to keep up. Some of the best resources and spaces to keep an eye on include:

*   **Controlled Image Super-Resolution with Diffusion Models:** This process includes using text prompts while performing super-resolution. It gives a stronger signal for the model to perform the task, while allowing the user to control certain details or ideas that need to be focused on by the model.
*   **Approaches Involving Transformers:** Transformers have shown the ability to solve long-range dependencies, meaning they are better at extrapolating information over large distances in order to make the final result coherent. For our task, this is a useful trait since the models would be able to recreate non-local features with ease.
*   **Fast Fourier Transform-Based Convolutions:** Current techniques have also proposed using Fast Fourier Transforms as a method for feature engineering and extraction to help bring out stronger information signals from the image inputs and make the model more receptive to understanding the semantics of a scene.
*   **SWIN Transformer-Based Techniques:** These techniques talk about the use of the Shifted-Windows approach popularized by the SWIN Transformer and how they can be utilized for sharing information over different receptive fields, which can, in turn, help build a more efficient and robust model that can make strong predictions.

* * *

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

* * *

[**Summary**](#TOC-h2-Summary)
------------------------------

That’s it! We just went over quite a few things together at the same time. To quickly look back, we covered the following:

*   [A quick overview of a problem statement involving the upscaling of CCTV images simply because of how difficult they are to interpret](#h2-Problem-Statement)
*   [Current state-of-the-art approaches through GANs and Diffusion Models](#h2-SOA-Approaches)
*   [Implementation of a pipeline using Hugging Face Diffusers that allows us to perform super-resolution efficiently](#h2-Implementing-Diffusion-Based-Upscaler)
*   [Plotting our results with a strong emphasis on details within the image](#h3-Plotting-Our-Results)
*   [How to go ahead from here into further deep-dives in depth](#h2-Way-Forward)

Thank you for following along, and see you soon for the next tutorials!

* * *

### [**Citation Information**](#TOC-h3-Citation)

**Mukherjee, S.** “Sharpen Your Vision: Super-Resolution of CCTV Images Using Hugging Face Diffusers,” _PyImageSearch_, P. Chugh, A. R. Gosthipaty, S. Huot, K. Kidriavsteva, and R. Raha, eds., 2024, [https://pyimg.co/eoubf](https://pyimg.co/eoubf)

@incollection{Mukherjee\_2024\_SR-CCTV,
  author = {Suvaditya Mukherjee},
  title = {Sharpen Your Vision: Super-Resolution of CCTV Images Using Hugging Face Diffusers},
  booktitle = {PyImageSearch},
  editor = {Puneet Chugh and Aritra Roy Gosthipaty and Susan Huot and Kseniia Kidriavsteva and Ritwik Raha},
  year = {2024},
  url = {https://pyimg.co/eoubf},
}

* * *

![Image 6](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Join the PyImageSearch Newsletter and Grab My FREE 17-page Resource Guide PDF

Enter your email address below to **join the PyImageSearch Newsletter** and **download my FREE 17-page Resource Guide PDF** on Computer Vision, OpenCV, and Deep Learning.