Title: Torch Hub Series #5: MiDaS — Model on Depth Estimation - PyImageSearch

URL Source: https://pyimagesearch.com/2022/01/17/torch-hub-series-5-midas-model-on-depth-estimation/

Published Time: 2022-01-17T14:00:00+00:00

Markdown Content:
In the previous part of this series, we discussed some state-of-the-art object detection models; YOLOv5 and SSD. In today’s tutorial, we will discuss MiDaS, an ingenious attempt to aid the depth estimation of images.

![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20393'%3E%3C/svg%3E)

With this tutorial, we will create a basic intuition about the idea behind MiDaS and learn how to use it as a depth estimation inference tool.

This lesson is part 5 of a 6-part series on Torch Hub:

1.  [_Torch Hub Series #1: Introduction to Torch Hub_](https://pyimagesearch.com/2021/12/20/torch-hub-series-1-introduction-to-torch-hub/)
2.  [_Torch Hub Series #2: VGG and ResNet_](https://pyimagesearch.com/2021/12/27/torch-hub-series-2-vgg-and-resnet/)
3.  [_Torch Hub Series #3: YOLO v5 and SSD_ — _Models on Object Detection_](https://pyimagesearch.com/2022/01/03/torch-hub-series-3-yolov5-and-ssd-models-on-object-detection/)
4.  [_Torch Hub Series #4: PGAN_ — _Model on GAN_](https://pyimagesearch.com/2022/01/10/torch-hub-series-4-pgan-model-on-gan/)
5.  _Torch Hub Series #5: MiDaS_ — _Model on Depth Estimation_ (this tutorial)
6.  _Torch Hub Series #6: Image Segmentation_

**To learn how to use MiDaS on your custom data,** **_just keep reading._**

![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20128%20128'%3E%3C/svg%3E)

#### Looking for the source code to this post?

[Jump Right To The Downloads Section](#download-the-code)

### **Introduction**

First, let us understand what depth estimation is or why it is important. Depth estimation of an image predicts the order of objects (if the image was expanded in a 3D format) from the 2D image itself. It is an unequivocally difficult task since getting annotated data and datasets specializing in this area was a mammoth task itself. The use of depth estimation is far and wide, most noticeably in the domain of self-driving cars, where estimating the distance of objects around a car helps in navigation (**Figure 1**).

![Image 3](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20202'%3E%3C/svg%3E)

**Figure 1:** Depth maps ([source](https://medium.com/@Giscle/depth-map-depth-calculation-ce4d914c6afd)).

The researchers behind MiDaS have explained their motives in a very simple manner. They firmly assert that training models on a single dataset will not be robust when dealing with problem statements encompassing real-life issues. When models used in real-time are created, they should be robust enough to deal with as many situations and outliers as possible.

Keeping that in mind, the creators of MiDaS decided to train their model on **multiple** datasets. This includes datasets with different types of labels and objective functions. To achieve this, they devised a method to carry out computations in an appropriate output space compatible with all ground-truth representations.

The idea is very ingenious on paper, but the authors had to carefully devise loss functions and consider the challenges accompanying the choice of using multiple datasets. Since these datasets had different representations of depth estimation to varying degrees, an inherent scale ambiguity, and shift ambiguity come up, as the paper’s authors addressed.

Now, since the datasets in all probability would follow distributions different from each other. Hence the issues are pretty much expected. However, the authors propose solutions to each of the challenges. The end product was a robust depth estimator, which was as efficient as accurate. In **Figure 2**, we see some results shown in the paper.

The idea of cross dataset learning isn’t new, but the complications that arise from getting the ground truths to a common output space are extremely tough to get by. However, the paper explains each step extensively, starting with the intuition right to the mathematical definition of the losses used.

Let’s see how we can use the MiDaS model to find the inverse depth of our custom images.

### **Configuring **Your Development Environment****

To follow this guide, you need to have the OpenCV library installed on your system.

Luckily, OpenCV is pip-installable:

$ pip install opencv-contrib-python

**If you need help configuring your development environment for OpenCV, I _highly recommend_ that you read my** [**_pip install OpenCV_ guide**](https://pyimagesearch.com/2018/09/19/pip-install-opencv/) — it will have you up and running in a matter of minutes.

### **Having **Problems Configuring Your Development Environment?****

[![Image 4](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20334'%3E%3C/svg%3E)](https://pyimagesearch.com/pyimagesearch-university/)

**Figure 3:** Having trouble configuring your dev environment? Want access to pre-configured Jupyter Notebooks running on Google Colab? Be sure to join [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/) — you’ll be up and running with this tutorial in a matter of minutes.

All that said, are you:

*   Short on time?
*   Learning on your employer’s administratively locked system?
*   Wanting to skip the hassle of fighting with the command line, package managers, and virtual environments?
*   **Ready to run the code** **_right now_** **on your Windows, macOS, or Linux system?**

Then join [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/) today!

**Gain access to Jupyter Notebooks for this tutorial and other PyImageSearch guides that are** **_pre-configured_** **to run on Google Colab’s ecosystem right in your web browser!** No installation required.

And best of all, these Jupyter Notebooks will run on Windows, macOS, and Linux!

### **Project **Structure****

We first need to review our project directory structure.

Start by accessing the **_“Downloads”_** section of this tutorial to retrieve the source code and example images.

From there, take a look at the directory structure:

!tree .
.
├── midas\_inference.py
├── output
│   └── midas\_output
│       └── output.png
└── pyimagesearch
    ├── config.py
    └── data\_utils.py

Inside the `pyimagesearch` directory, we have 2 scripts:

*   `config.py`: Contains an end to end configuration pipeline for the project
*   `data_utils.py`: Houses the two data utility functions we’ll be using in the project

In the parent directory, we have one single script:

*   `midas_inference.py`: To infer from the pretrained MiDaS model

Finally, we have the `output` directory, which will house the result plots obtained from running the script.

### **Downloading the Dataset**

Owing to its compactness, we’ll be using the [Dogs & Cats Images](https://www.kaggle.com/chetankv/dogs-cats-images) dataset from Kaggle again.

$ mkdir ~/.kaggle
$ cp <path to your kaggle.json> ~/.kaggle/
$ chmod 600 ~/.kaggle/kaggle.json
$ kaggle datasets download -d chetankv/dogs-cats-images
$ unzip -qq dogs-cats-images.zip
$ rm -rf "/content/dog vs cat"

As explained in previous posts of this series, you’ll need your own unique `kaggle.json` file to connect to the Kaggle API (**Line 2**). The `chmod 600` command on **Line 3** will allow your script complete access to read and write files.

The following `kaggle datasets download` command (**Line 4**) allows you to download any dataset hosted on their website. Finally, we have the unzip command and an auxiliary delete command for the unnecessary additions (**Lines 5 and 6**).

Let’s move on to the configuration pipeline.

### **Configuring the Prerequisites**

Inside the `pyimagesearch` directory, you’ll find a script called `config.py`. This script will house the complete end-to-end configuration pipeline of our project.

\# import the necessary packages
import torch
import os

# define the root directory followed by the test dataset paths
BASE\_PATH = "dataset"
TEST\_PATH = os.path.join(BASE\_PATH, "test\_set")

# specify image size and batch size
IMAGE\_SIZE = 384
PRED\_BATCH\_SIZE = 4

# determine the device type 
DEVICE = torch.device("cuda") if torch.cuda.is\_available() else "cpu"

# define paths to save output 
OUTPUT\_PATH = "output"
MIDAS\_OUTPUT = os.path.join(OUTPUT\_PATH, "midas\_output")

First, we have the `BASE_PATH` variable as a pointer to the dataset directory (**Line 6**). We’re not doing any additional tinkering to our models, so we’ll only use the test set (**Line 7**).

On **Line 10**, we have a variable named `IMAGE_SIZE`, set to `384`, as a mandate for our MiDaS model input. The prediction batch size is set to `4` (**Line 11**), but readers are encouraged to play around with different sizes.

It’s advisable that you have a CUDA-compatible device for today’s project (**Line 14**), but since we’re not going for any heavy training, CPUs should work fine too.

Lastly, we have created paths to save the outputs obtained from the model inferences (**Lines 17 and 18**).

We’ll only be using a single helper function to aid our pipeline in today’s task. For that, we’ll move onto the second script in the `pyimagesearch` directory, `data_utils.py`.

\# import the necessary packages
from torch.utils.data import DataLoader

def get\_dataloader(dataset, batchSize, shuffle=True):
	# create a dataloader and return it
	dataLoader= DataLoader(dataset, batch\_size=batchSize,
		shuffle=shuffle)
	return dataLoader

On **Line 4**, we have the `get_dataloader` function, which takes in the dataset, batch size, and shuffle variables as its arguments. This function returns a generator like the PyTorch Dataloader instance which will help us deal with huge data (**Line 6**).

That concludes our utilities. Let’s move on to the inference script.

### **Finding Inverse Depth Estimation Using MiDaS**

At this time, a very logical question might pop up in your mind; Why are we going to such lengths to get inference on a handful of images?

The path we have chosen here is a full-proof way of dealing with huge datasets, and the pipeline will be useful even if you choose to train the model for fine-tuning later down the road. We have also considered preparing the data as best we can without invoking the premade functions of the MiDaS repository.

\# import necessary packages
from pyimagesearch.data\_utils import get\_dataloader
from pyimagesearch import config
from torchvision.transforms import Compose, ToTensor, Resize
from torchvision.datasets import ImageFolder
import matplotlib.pyplot as plt
import torch
import os

# create the test dataset with a test transform pipeline and
# initialize the test data loader
testTransform = Compose(\[
	Resize((config.IMAGE\_SIZE, config.IMAGE\_SIZE)), ToTensor()\])
testDataset = ImageFolder(config.TEST\_PATH, testTransform)
testLoader = get\_dataloader(testDataset, config.PRED\_BATCH\_SIZE)

As mentioned earlier, since we will be only using the test set, we have created a PyTorch test transform instance where we are reshaping the images and converting them to tensors (**Lines 12 and 13**).

If your dataset is in the format as the one we are using for our project (i.e., images under the folder named as labels), then we can use the `ImageFolder` function to create a PyTorch Dataset instance (**Line 14**). Lastly, we use the previously defined `get_dataloader` function to generate a Dataloader instance (**Line 15**).

\# initialize the midas model using torch hub
modelType = "DPT\_Large" 
midas = torch.hub.load("intel-isl/MiDaS", modelType)

# flash the model to the device and set it to eval mode
midas.to(device)
midas.eval()

Next, we use the `torch.hub.load` function to load the MiDaS model in our local runtime (**Lines 18 and 19**). Again, several available choices can be called here, all of which can be found [here](https://github.com/isl-org/MiDaS). Finally, we load the model to our device and set it to evaluation mode (**Lines 22 and 23**).

\# initialize iterable variable
sweeper = iter(testLoader)

# grab a batch of test data send the images to the device
print("\[INFO\] getting the test data...")
batch = next(sweeper)
(images, \_) = (batch\[0\], batch\[1\])
images = images.to(config.DEVICE) 
 
# turn off auto grad
with torch.no\_grad():
	# get predictions from input
	prediction = midas(images)

	# unsqueeze the predictions batchwise
	prediction = torch.nn.functional.interpolate(
		prediction.unsqueeze(1), size=\[384,384\], mode="bicubic",
		align\_corners=False).squeeze()

# store the predictions in a numpy array
output = prediction.cpu().numpy()

The `sweeper` variable on **Line 26** will act as the iterable variable of the `testLoader`. Each time we run the command on **Line 30**, we’ll get a new batch of data from the `testLoader`. On **Line 31**, we unpack the batch into images and labels, keeping only the images.

After loading the images to our device (**Line 32**), we turn off automatic gradients and pass the images through the model (**Lines 35-37**). On **Line 40**, we use a beautiful utility function called `torch.nn.functional.interpolate` to unpack our predictions into a valid 3-channel image format.

Finally, we store the reformatted predictions to an output variable in numpy format (**Line 45**).

\# define row and column variables
rows = config.PRED\_BATCH\_SIZE
cols = 2

# define axes for subplots
axes = \[\]
fig=plt.figure(figsize=(10, 20))

# loop over the rows and columns
for totalRange in range(rows\*cols):
	axes.append(fig.add\_subplot(rows, cols, totalRange+1))

	# set up conditions for side by side plotting 
	# of ground truth and predictions
	if totalRange % 2 == 0:
		plt.imshow(images\[totalRange//2\]
			.permute((1, 2, 0)).cpu().detach().numpy())
	else :
		plt.imshow(output\[totalRange//2\])
fig.tight\_layout()

# build the midas output directory if not already present
if not os.path.exists(config.MIDAS\_OUTPUT):
	os.makedirs(config.MIDAS\_OUTPUT)

# save plots to output directory
print("\[INFO\] saving the inference...")
outputFileName = os.path.join(config.MIDAS\_OUTPUT, "output.png")
plt.savefig(outputFileName)

To plot our results, we first define the rows and column variables to define the grid format (**Lines 48 and 49**). On **Lines 52 and 53**, we define the subplot list and figure size.

Looping over the rows and columns, we define a methodology where the inverse depth estimation and the ground truth images get plotted side by side (**Lines 56-66**).

Lastly, we save the figure to our desired path (**Lines 69 and 75**).

With our inference script done, let’s check out some of the results.

### **MiDaS Inference Results**

Most of the images in our dataset will have either a cat or dog in the forefront. There might not be enough images with background, but the MiDaS model should give us an output that depicts the cat or dog in the foreground decisively. That is exactly what has happened in our inference images ( **Figures 4-7**).

![Image 5](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20483%20250'%3E%3C/svg%3E)

****Figure 4:**** Inverse depth estimation of an image of a cat closeup.

![Image 6](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20488%20248'%3E%3C/svg%3E)

**Figure 5:** Inverse depth estimation of an image of a dog in a field.

![Image 7](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20476%20242'%3E%3C/svg%3E)

**Figure 6:** Inverse depth estimation of an image of an indoor cat.

![Image 8](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20481%20243'%3E%3C/svg%3E)

**Figure 7:** Inverse depth estimation of an image of a cat in a box.

Although the spectacular capabilities of MiDaS can be seen in all the inference images, we can look into each of them in-depth and draw out some more observations.

In **Figure 4**, not only is the cat depicted in the foreground, but its head is closer to the camera than its body (shown by the change in color). In **Figure 5**, since the field mainly covers the image, there is a clear distinction between the dog and the field. In both **Figure 6** and **Figure 7**, the cat’s head is depicted closer than its body.

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

**Summary**
-----------

Grasping the importance of MiDaS in today’s world is an important step for us. Imagine how much help a perfect depth estimator would be for autonomous vehicles. Since autonomous cars almost completely depend on utilities like LiDAR (light detection and ranging), cameras, SONAR (sound navigation and ranging), etc., having a full-proof system of depth estimation of its surroundings will make travel safer and take the load off the other sensor systems.

The second most important thing to note about MiDaS is the usage of cross dataset mixing. While it has recently gained momentum, executing it to perfection takes considerable time and planning. Moreover, MiDaS does this on a domain that significantly impacts a real-world issue.

I hope this tutorial served as a gateway to pique your interest regarding all things about autonomous systems and depth estimation. Feel free to try this out with your custom datasets and share the results.

### **Citation** Information

**Chakraborty, D.** “Torch Hub Series #5: MiDaS — Model on Depth Estimation,” _PyImageSearch_, 2022, [https://pyimagesearch.com/2022/01/17/torch-hub-series-5-midas-model-on-depth-estimation/](https://pyimagesearch.com/2022/01/17/torch-hub-series-5-midas-model-on-depth-estimation/)

@article{Chakraborty\_2022\_THS5,
  author = {Devjyoti Chakraborty},
  title = {Torch Hub Series \\#5: {MiDaS} — Model on Depth Estimation},
  journal = {PyImageSearch},
  year = {2022},
  note = {https://pyimagesearch.com/2022/01/17/torch-hub-series-5-midas-model-on-depth-estimation/},
}

![Image 9: Featured Image](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20276'%3E%3C/svg%3E)

### Unleash the potential of computer vision with Roboflow - Free!

*   Step into the realm of the future by [signing up or logging into your Roboflow account](https://roboflow.com/?ref=pyimagesearch). Unlock a wealth of innovative dataset libraries and revolutionize your computer vision operations.
*   Jumpstart your journey by choosing from our broad array of datasets, or benefit from PyimageSearch’s comprehensive library, crafted to cater to a wide range of requirements.
*   Transfer your data to Roboflow in any of the 40+ compatible formats. Leverage cutting-edge model architectures for training, and deploy seamlessly across diverse platforms, including API, NVIDIA, browser, iOS, and beyond. Integrate our platform effortlessly with your applications or your favorite third-party tools.
*   Equip yourself with the ability to train a potent computer vision model in a mere afternoon. With a few images, you can import data from any source via API, annotate images using our superior cloud-hosted tool, kickstart model training with a single click, and deploy the model via a hosted API endpoint. Tailor your process by opting for a code-centric approach, leveraging our intuitive, cloud-based UI, or combining both to fit your unique needs.
*   Embark on your journey today with absolutely no credit card required. Step into the future with Roboflow.

[Join Roboflow Now](https://roboflow.com/?ref=pyimagesearch)

**To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), _simply enter your email address in the form below!_**

![Image 10](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a **FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning.** Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!