Title: Computer Graphics and Deep Learning with NeRF using TensorFlow and Keras: Part 1 - PyImageSearch

URL Source: https://pyimagesearch.com/2021/11/10/computer-graphics-and-deep-learning-with-nerf-using-tensorflow-and-keras-part-1/

Published Time: 2021-11-10T15:00:00+00:00

Markdown Content:
**Table of Contents**
---------------------

*   **[Computer Graphics and Deep Learning with NeRF using TensorFlow and Keras: Part 1](#blog_title)**

*   [World Coordinate Frame](#World_Coord_Frame)
*   [Camera Coordinate Frame](#Camera_Coord_Frame)
*   [Coordinate Transformation](#Coord_Transform)
*   [Projective Transformation](#Proj_Transform)
*   [Dataset](#Dataset)

*   **[Summary](#Summary)**

*   [Citation Information](#Citation_Info)

A few days back, while going through my photo gallery, I came across three pictures shown in **Figure 1**.

![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20190'%3E%3C/svg%3E)

**Figure 1:** Three views of a staircase.

We can see that the pictures are of a staircase but from different viewpoints. I took three pictures because I wasn’t sure I could capture this beautiful scene with just one photo. I was worried I’d miss the right perspective.

![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20393'%3E%3C/svg%3E)

**This got me thinking, “what if there was a way to capture the entire 3D scene just from these pictures?”**

That way, you (my audience) will be able to view exactly what I saw that day.

In [NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis, Mildenhall et al. (2020)](https://arxiv.org/abs/2003.08934) proposed a method that turned out to be just what I needed.

Let’s look at what we have achieved by reproducing the paper’s method, as shown in **Figure 2**. For example, giving the algorithm some pictures of a plate of hotdogs from different viewpoints (_top_) could generate the entire 3D scenery (_bottom_) precisely.

![Image 3](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20500'%3E%3C/svg%3E)

![Image 4](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20200%20200'%3E%3C/svg%3E)

**Figure 2:** _Top_: Various views of the scenery (input). _Bottom_: Generated 3D scenery (output).

Neural Radiance Fields (NeRF) bring together **Deep Learning** and **Computer Graphics**. While we at PyImageSearch have written a lot about Deep Learning, this will be the first time we talk about Computer Graphics. This series will be structured in a way that will best suit an absolute beginner. **_We expect no prior knowledge of Computer Graphics_**.

**_Note:_** _A simpler implementation of NeRF won us the_ [_TensorFlow Community Spotlight Award_](https://twitter.com/TensorFlow/status/1466150113814929413)_._

**To learn about Computer Graphics and image rendering, _just keep reading._**

* * *

Computer Graphics has been one of the wonders of modern technology. The applications of rendering realistic 3D scenes range from movies, space navigation, to medical science.

This lesson is part 1 of a 3-part series on Computer Graphics and Deep Learning with NeRF using TensorFlow and Keras:

*   _Computer Graphics and Deep Learning with NeRF using TensorFlow and Keras: Part 1_ (this tutorial)
*   _Computer Graphics and Deep Learning with NeRF using TensorFlow and Keras: Part 2_ (next week’s tutorial)
*   _Computer Graphics and Deep Learning with NeRF using TensorFlow and Keras: Part 3_

In this tutorial, we will cover the workings of a camera in the world of computer graphics. We will also introduce you to the dataset where we will work.

We have divided this tutorial into the following subsections:

*   **World Coordinate Frame:** representing the physical 3D world
*   **Camera Coordinate Frame:** representing the virtual 3D camera world
*   **Coordinate Transformation:** mapping from one coordinate frame to another
*   **Projection Transformation:** forming an image on a 2D plane (the camera sensor)
*   **Dataset:** understanding the dataset for NeRF

Imagine this. You are out with your camera and spot a beautiful flower. You think about the way you want to capture it. Now it is time to orient the camera, calibrate the settings, and click the picture.

This entire process of transforming the world scene into an image is encapsulated in a mathematical model commonly called the **forward imaging model.** We can visualize the model in **Figure 3**.

![Image 5](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20730%20228'%3E%3C/svg%3E)

****Figure 3:**** Forward Imaging Model.

The forward imaging model starts from a point in the **world coordinate frame**. We then transform this to the **camera coordinate frame** using **coordinate transformation**. After that, we use **projection transformation** to transform the camera coordinates onto the **image plane**.

* * *

### **[World Coordinate Frame](#TOC)**

All shapes and objects that we see in the real world exist in a 3D frame of reference. We call this frame of reference the world coordinate frame. Using this frame, we can locate any point or object in the 3D space quite easily.

Let’s take the point ![Image 6: P](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2013%2011'%3E%3C/svg%3E) in the 3D space as shown in **Figure 4**.

![Image 7](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20345%20212'%3E%3C/svg%3E)

**Figure 4:** World Coordinate Frame.

Here, ![Image 8: \hat{x}_{w}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2018%2016'%3E%3C/svg%3E), ![Image 9: \hat{y}_{w}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2017%2016'%3E%3C/svg%3E), and ![Image 10: \hat{z}_w](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2017%2016'%3E%3C/svg%3E) represent the three axes in the world coordinate frame. The location of the point ![Image 11: P](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2013%2011'%3E%3C/svg%3E) is expressed through the vector ![Image 12: X_{w} ](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2022%2014'%3E%3C/svg%3E).

![Image 13: X_{w} = \begin{bmatrix} x_w \\ y_w \\ z_w \\ \end{bmatrix} ](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2080%2060'%3E%3C/svg%3E)

* * *

### **[Camera Coordinate Frame](#TOC)**

Like the world coordinate frame, we have another frame of reference called the camera coordinate frame, as shown in **Figure 5**.

![Image 14](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20221%20202'%3E%3C/svg%3E)

**Figure 5:** Camera Coordinate Frame.

**_This frame sits at the center of a camera_**. Unlike the world coordinate frame, this is not a static frame of reference. We can move this frame as we are moving the camera around to take a picture.

The same point ![Image 15: P](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2013%2011'%3E%3C/svg%3E) from **Figure 4** can now be located with both frames of reference, as shown in **Figure 6**.

![Image 16](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20509%20253'%3E%3C/svg%3E)

**Figure 6:** World-to-camera coordinate transformation.

While in the world coordinate frame, the point is located by the ![Image 17: X_{w} ](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2022%2014'%3E%3C/svg%3E) vector, in the camera coordinate frame, it is located by the ![Image 18: X_{c} ](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2019%2014'%3E%3C/svg%3E) vector as shown in **Figure 6**.

![Image 19: X_{c} = \begin{bmatrix} x_c \\ y_c \\ z_c \\ \end{bmatrix}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2073%2060'%3E%3C/svg%3E)

**_Note:_** _The location of point ![Image 20: P](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2013%2011'%3E%3C/svg%3E) does not change. Only the way of looking at the point changes with change in the frame of reference._

* * *

### **[Coordinate Transformation](#TOC)**

We have established two coordinate frames: the world and the camera. Now let us define a mapping between the two.

Let’s take the point ![Image 21: P ](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2013%2011'%3E%3C/svg%3E) from **Figure 6**. Our goal is to build a bridge between the camera coordinates ![Image 22: X_{c}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2019%2014'%3E%3C/svg%3E) and world coordinates ![Image 23: X_{w}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2022%2014'%3E%3C/svg%3E).

From **Figure 5**, we can say that

![Image 24: X_{c} = R \times (X_{w}-C_{w})](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20150%2018'%3E%3C/svg%3E)

where

We can expand the above equation as follows

![Image 25: X_{c} = R\times (X_{w}-C_{w}) \\ \Rightarrow X_{c} = R \times X_w - R \times C_w \\ \Rightarrow X_{c} = R \times X_w + t ](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20192%2057'%3E%3C/svg%3E)

![Image 26: x_0 = \displaystyle\frac{\tilde{x}_0}{w}, \space x_1 = \displaystyle\frac{\tilde{x}_1}{w} ,\dots ,\space x_n = \displaystyle\frac{\tilde{x}_n}{w} ](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20216%2034'%3E%3C/svg%3E)

Using the homogeneous coordinate system we can transform ![Image 27: X_w](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2022%2014'%3E%3C/svg%3E) (3D) to ![Image 28: \tilde{X}_w](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2022%2019'%3E%3C/svg%3E) (4D).

![Image 29: X_w \equiv \begin{bmatrix} x\\ y\\ z\\ 1\\ \end{bmatrix} \equiv \begin{bmatrix} wx_w\\ wy_w\\ wz_w\\ w\\ \end{bmatrix} \equiv \begin{bmatrix} \tilde{x}\\ \tilde{y}\\ \tilde{z}\\ w\\ \end{bmatrix} \equiv \tilde{X}_w](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20252%2080'%3E%3C/svg%3E)

With the homogeneous coordinates at hand, we can compress the equation … to just matrix multiplication.

![Image 30: \tilde{X}_c = \begin{bmatrix} x_c\\ y_c\\ z_c\\ 1\\ \end{bmatrix} = \begin{bmatrix} R & t \\ 0 & 1 \\ \end{bmatrix} \times \tilde{X}_w = \begin{bmatrix} r_{11} & r_{12} & r_{13} & t_x\\ r_{21} & r_{22} & r_{23} & t_y\\ r_{31} & r_{32} & r_{33} & t_z\\ 0 & 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} x_w\\ y_w\\ z_w\\ 1\\ \end{bmatrix}  \Rightarrow \boxed{ \tilde{X}_c = C_{ex} \times \tilde{X}_w}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20540%2080'%3E%3C/svg%3E)

where ![Image 31: C_{ex}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2024%2014'%3E%3C/svg%3E) is the matrix that holds the orientation and position of the camera coordinate frame. We can call this matrix the **Camera Extrinsic** since it represents values like rotation and translation, both of which are external properties of the camera.

![Image 32: C_{ex} = \begin{bmatrix} r_{11} & r_{12} & r_{13} & t_x\\ r_{21} & r_{22} & r_{23} & t_y\\ r_{31} & r_{32} & r_{33} & t_z\\ 0 & 0 & 0 & 1 \end{bmatrix} ](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20177%2080'%3E%3C/svg%3E)

* * *

### **[Projective Transformation](#TOC)**

![Image 33](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20637%20261'%3E%3C/svg%3E)

**Figure 7:** Projecting a point on the image plane.

To understand projective transformation, the only thing we need is similar triangles. Let’s do a primer on similar triangles.

We have visualized similar triangles in **Figures 8 and 9.** With similar triangles

![Image 34: \displaystyle\frac{AB}{A' B'} = \displaystyle\frac{CB}{C' B'}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2092%2034'%3E%3C/svg%3E)

Yes, you guessed it correctly, ![Image 35: AB'C'](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2046%2013'%3E%3C/svg%3E) and ![Image 36: ABP](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2038%2011'%3E%3C/svg%3E) are similar triangles in **Figure 10**.

![Image 37](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20367'%3E%3C/svg%3E)

****Figure 10:**** Can you spot the similar triangles?

From the properties of similar triangles, we can derive that

![Image 38: \displaystyle\frac{x_i}{f} = \displaystyle\frac{x_c}{z_c}; \space \displaystyle\frac{y_i}{f} = \displaystyle\frac{y_c}{z_c}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20116%2033'%3E%3C/svg%3E)

Therefore it follows:

![Image 39: x_i = f\displaystyle\frac{x_c}{z_c}\\ \space \\ y_i = f\displaystyle\frac{y_c}{z_c}\\ ](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2063%2072'%3E%3C/svg%3E)

Now it is important to remember that the actual image plane is not a virtual plane but rather an array of image sensors. The 3D scene falls on this sensor which leads to the formation of the image. Thus ![Image 40: x_i](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2013%2010'%3E%3C/svg%3E) and ![Image 41: y_i](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2012%2010'%3E%3C/svg%3E) in the image plane can be substituted with pixel values ![Image 42: u,v](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2024%2010'%3E%3C/svg%3E).

![Image 43: u = f\displaystyle\frac{x_c}{z_c} \\ \space \\ v = f\displaystyle\frac{y_c}{z_c} \\ ](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2059%2072'%3E%3C/svg%3E)

A pixel in an image plane starts from the upper left-hand corner `(0, 0)`, so it is also required to shift the pixels with respect to the center of the image plane.

![Image 44: u = f\displaystyle\frac{x_c}{z_c} + o_x \\ \space \\ v = f\displaystyle\frac{y_c}{z_c} + o_y](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2094%2072'%3E%3C/svg%3E)

Here, ![Image 45: o_x](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2015%2010'%3E%3C/svg%3E) and ![Image 46: o_y](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2014%2012'%3E%3C/svg%3E) are the center points of the image plane.

Now we have a point from the 3D camera space represented in terms of ![Image 47: u,v](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2024%2010'%3E%3C/svg%3E) in the image plane. Again to make matrices agree, we have to express the pixel values using homogeneous representation.

Homogeneous representation of ![Image 48: u,v](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2024%2010'%3E%3C/svg%3E), where ![Image 49: u = \displaystyle\frac{\tilde{u}}{\tilde{w}}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2046%2034'%3E%3C/svg%3E) and ![Image 50: v = \displaystyle\frac{\tilde{v}}{\tilde{w}}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2045%2034'%3E%3C/svg%3E)

![Image 51: \begin{bmatrix} u\\ v\\ 1\\ \end{bmatrix} \equiv \begin{bmatrix} \tilde{u}\\ \tilde{v}\\ \tilde{w}\\ \end{bmatrix} \equiv \begin{bmatrix} z_cu\\ z_cv\\ z_c\\ \end{bmatrix} ](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20145%2060'%3E%3C/svg%3E)

This can be further expressed as:

![Image 52: \begin{bmatrix} z_cu\\ z_cv\\ z_c\\ \end{bmatrix}= \begin{bmatrix} fx_c +z_co_x\\ fy_c +y_co_y\\ z_c\\ \end{bmatrix} = \begin{bmatrix} f & 0 & o_x & 0\\ 0 & f & o_y & 0\\ 0 & 0 & 1 & 0\\ \end{bmatrix} \begin{bmatrix} x_c\\ y_c\\ z_c\\ 1\\ \end{bmatrix}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20320%2080'%3E%3C/svg%3E)

Finally, we have:

![Image 53: \begin{bmatrix} \tilde{u}\\ \tilde{v}\\ \tilde{w}\\ \end{bmatrix} = \begin{bmatrix} f & 0 & o_x & 0\\ 0 & f & o_y & 0\\ 0 & 0 & 1 & 0\\ \end{bmatrix} \begin{bmatrix} x_c\\ y_c\\ z_c\\ 1\\ \end{bmatrix} ](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20192%2080'%3E%3C/svg%3E)

which can be expressed simply as

![Image 54: \tilde{u} = C_{in} \times \tilde{x}_c](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2090%2015'%3E%3C/svg%3E)

where ![Image 55: \tilde{x}_c](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2015%2015'%3E%3C/svg%3E) is the set of vectors containing the location of the point in camera coordinate space and ![Image 56: \hat{u}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%209%2013'%3E%3C/svg%3E) is the set of values containing the location of the point on the image plane. Respectively, ![Image 57: C_{in}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2023%2014'%3E%3C/svg%3E) represents the set of values needed to map a point from the 3D camera space to the 2D space.

![Image 58: C_{in} = \begin{bmatrix} f & 0 & o_x & 0\\ 0 & f & o_y & 0\\ 0 & 0 & 1 & 0\\ \end{bmatrix} ](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20147%2060'%3E%3C/svg%3E)

We can call ![Image 59: C_{in}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2023%2014'%3E%3C/svg%3E) the **camera intrinsic** since it represents values like focal length and center of the image plane along ![Image 60: x](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%209%207'%3E%3C/svg%3E) and ![Image 61: y](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%208%2010'%3E%3C/svg%3E) axes, both of which are internal properties of the camera.

* * *

### **[Dataset](#TOC)**

Enough theory! Show me some code.

In this section, we will talk about the data with which we are going to work. The authors have open-sourced their dataset, and you can find it [here](https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1). The link to the dataset was published in the official repository of [NeRF](https://github.com/bmild/nerf). The dataset is structured as shown in **Figure 11**.

![Image 62](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20500'%3E%3C/svg%3E)

Figure 11: The drive folder containing the dataset.

There are two folders, `nerf_synthetic` and `nerf_llff_data`. Moving ahead, we will be using the synthetic dataset for this series.

Let’s see what is in the `nerf_synthetic` folder. The data in the `nerf_synthetic` folder is shown in **Figure 12**.

![Image 63](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20500'%3E%3C/svg%3E)

**Figure 12:** The synthetic data folder.  

There are a lot of synthetic objects here. Let’s download one of them and see what is inside. We have chosen the “ship” dataset, but feel free to download any one of them.

After unzipping the dataset, you will find three folders containing images:

*   `train`
*   `val`
*   `test`

and three files containing the orientation and position of the camera.

*   `transforms_train.json`
*   `transforms_val.json`
*   `transforms_test.json`

To better understand the json files, we can open a [blank Colab notebook](https://research.google.com/colaboratory/) and upload the `transforms_train.json`. We can now perform exploratory data analysis on it.

\# import the necessary packages
import json
import numpy as np

# define the json training file
jsonTrainFile = "transforms\_train.json"

# open the file and read the contents of the file
with open(jsonTrainFile, "r") as fp:
    jsonTrainData = json.load(fp)

# print the content of the json file
print(f"\[INFO\] Focal length train: {jsonTrainData\['camera\_angle\_x'\]}")
print(f"\[INFO\] Number of frames train: {len(jsonTrainData\['frames'\])}")

# OUTPUT
# \[INFO\] Focal length train: 0.6911112070083618
# \[INFO\] Number of frames train: 100

We begin with importing the necessary packages `json` and `numpy` on **Lines 2 and 3**.

Then we the load json and read its values on **Lines 6-10**.

The json file has two parent keys called `camera_angle_x` and `frames`. We see that `camera_angle_x` corresponds to the camera’s field of view, and `frames` are a collection of metadata for each image (frame).

On **Lines 13 and 14,** we print the values of the json keys. **Lines 17 and 18** show the output.

Let’s investigate `frames` a little further.

\# grab the first frame
firstFrame = jsonTrainData\["frames"\]\[0\]

# grab the transform matrix and file name
tMat = np.array(firstFrame\["transform\_matrix"\])
fName = firstFrame\["file\_path"\]

# print the data
print(tMat)
print(fName)

# OUTPUT
# array(\[\[-0.92501402,  0.27488998, -0.26226836, -1.05723763\],
#       \[-0.37993318, -0.66926789,  0.63853836,  2.5740304 \],
#       \[ 0.        ,  0.6903013 ,  0.72352195,  2.91661024\],
#       \[ 0.        ,  0.        ,  0.        ,  1.        \]\])
# ./train/r\_0

We grab the first frame on **Line 20**. Each frame is a dictionary containing two keys, `transform_matrix` and `file_path`, as shown on **Lines 23 and 24**. The `file_path` is the path to the image (frame) under consideration, and the `transform_matrix` is the camera-to-world matrix for that image.

On **Lines 27 and 28,** we print the `transform_matrix` and `file_path`. **Lines 31-35** show the output.

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

* * *

**[Summary](#TOC)**
-------------------

In this tutorial, we looked at some fundamental topics across Computer Graphics. This was essential to understand NeRF. While this is basic, it is still an important step to move forward.

We can recall what we have learned in **three simple steps**:

1.  The forward imaging model (taking a picture)
2.  World-to-Camera (3D to 3D) transformation
3.  Camera-to-Image (3D to 2D) transformation

At this point, we are also familiar with the dataset needed. This covers all of the prerequisites.

Next week we will look at the various underlying concepts of the paper: [NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis](https://arxiv.org/abs/2003.08934). We will also learn how to implement these concepts using TensorFlow and Python.

We hope you enjoyed this tutorial, and be sure to download the dataset and give it a try.

### **[Citation Information](#TOC)**

**Gosthipaty, A. R., and Raha, **R.**** “Computer Graphics and Deep Learning with NeRF using TensorFlow and Keras: Part 1,” _PyImageSearch_, 2021, [https://pyimagesearch.com/2021/11/10/computer-graphics-and-deep-learning-with-nerf-using-tensorflow-and-keras-part-1/](https://pyimagesearch.com/2021/11/10/computer-graphics-and-deep-learning-with-nerf-using-tensorflow-and-keras-part-1/)

@article{Gosthipaty\_Raha\_2021\_pt1,
    author = {Aritra Roy Gosthipaty and Ritwik Raha},
    title = {Computer Graphics and Deep Learning with {NeRF} using {TensorFlow} and {Keras}: Part 1},
    journal = {PyImageSearch},
    year = {2021},
    note = {https://pyimagesearch.com/2021/11/10/computer-graphics-and-deep-learning-with-nerf-using-tensorflow-and-keras-part-1/},   
}

**To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), _simply enter your email address in the form below!_**

![Image 64](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Join the PyImageSearch Newsletter and Grab My FREE 17-page Resource Guide PDF

Enter your email address below to **join the PyImageSearch Newsletter** and **download my FREE 17-page Resource Guide PDF** on Computer Vision, OpenCV, and Deep Learning.