Making GA-Net more Efficient
By: Rutger Dirks
Tags: Deep Learning, Stereo Vision, Efficiency, Ga Net, Computer Vision

Recap: Epipolar geometry

For a monocular camera, since there is only one optical center, the positions of two points through the same ray on the image are the same, so the depth volume between the two points cannot be judged. Therefore, the basis of stereo vision is the picture of the binocular cameras.

We can see from the figure that for binocular imaging, the projection positions of the same point object are different on two image planes, thus we can compare and calculate the hidden depth information. In this case, epipolar geometry is a commonly used description model.

Epipolar Geometry [2].

In figure 2, OL and OR are the optical centers of the left and right cameras respectively. eL and eR is the projection of the optical center on a camera’s image plane, called epipolars. Considering the easiest situation, where two cameras are coplanar. For the specific object X, together with two optical centers, we can define a plane that passes the baseline of the cameras, this plane is called epipolar plane. For a single point on the left image plane, it’s actually the projection of a bunch of points that lay in the same ray in the space. Thus, we can always find a corresponding line projection of these points on the right image plane. This projected line is called the epipolar line. For the purpose of obtaining the depth of the object, we need to pick a point in the left image plane, then search along the epipolar line, and the location of the best matched point on the line can be computed to obtain depth information using the disparity formula. This is how the whole system works, so after searching for every pixel on the image plane, finally the depth map can be generated which is the vision in 3D.

Recap: Traditional algorithms for cost evaluation

Cost is used to decide the best matching points when performing stereo matching. There are a lot of algorithms for calculating the cost for stereo vision tasks. Here we just introduce two related algorithms as the background knowledge which are mentioned in the original paper we built our research upon.

Local Cost Aggregation

This algorithm considers the similarity of the neighboring pixels for the cost computation. The method is to put a window and calculate the sum of pixel differences of the points. However, this kind of algorithm based on pixel value will be very sensitive to light and distortion and will produce certain errors.

Here C(q,d) represents the cost between the target pixel and the neighbor pixel q. w is the aggregation weight which is provided by the pre-defined filter framework. For every candidate disparity d the aggregated cost CA(p,d) for pixel p will be calculated. The point belonging to the disparity that obtains the minimal cost will be selected as the best point of which its depth information is included into a depth map.

Semi-global matching (SGM)

When enforcing global aggregation, the matching cost and the smoothness constraints are formulated into one energy function E(D), and the task becomes to find the best disparity map D that minimizes the energy function to obtain the disparity as a whole. The cost from different directions are aggregated to satisfy the energy function.

The energy function E(D) can be calculated by the above formula. The first term where the sum of matching costs at all pixel locations p for disparity map D. The second term is a constant penalty P1 for locations q in the neighborhood of p if they have small disparity discontinuities in the disparity map. The last term adds a larger constant penalty P2, for all larger disparity changes.

Inspired by the global aggregation, Hirschmuller proposed semi-global matching which takes the aggregated cost of a pixel along various directions.

In this formula, r is the unit direction which can be divided into 8–16 directions uniformly. The formula also takes into account neighboring depth costs and average depth costs. P are the offsets that can be tuned.

Process of Semi-global matching, costs are calculated for 8 directions and aggregated to compute the cost, the root image of the Tsukuba derived from [4].

Architecture of GA-Net

GA-Net is proposed for end-to-end stereo matching using guided aggregation. The main target can be concluded as formulating traditional geometry and optimization of stereo vision into deep neural networks. This figure illustrates the basic architecture of GA-Net, which is composed of 4 parts: a feature extraction block, a cost aggregation for the cost volume, a guidance subnet and a disparity regression.

The GA-Net architecture [1].

Firstly, the left and right images are fed into the same densely stacked hourglass CNN for feature extraction. This network shares weights for different layers.

Then, the extracted features are used to build a 4D matching cost volume with the size of H x W x Dmax x F, where H and W represents height and width respectively, with Dmax being the max disparity and F meaning the feature size. Since the cost volume is 4D, 3D convolution layers are necessary for cost aggregation. GA-Net replaces some of those 3D layers with GA (guided aggregation) layers for better performance and higher calculation efficiency. SGA (semi-global GA) can aggregate the matching costs over the whole image and LGA (Local GA) is useful for refining thin structures and recovering the accuracy loss caused by down-sampling.

The weight matrices for the guided cost aggregation are generated by the guidance subnet (the green network above the main network in the architecture). This guidance subnet is composed of several 2D layers and the outputs are reshaped and normalized into weight matrices for GA layers.

Finally, the disparity regression is implemented after the LGA layer. This part contains a softmax layer, whose output is the disparity estimation representing the depth information.

Related work

Apart from the traditional method (Epipolar geometry), there are many other methods for stereo matching using deep neural networks. Some methods took advantage of the traditional cost aggregation. For example, SGM-Nets [9] predicted the penalty-parameters for SGM using a neural net, whereas Schonberger et al. [10] learned to fuse proposals by optimization in stereo matching and Yang et al. proposed to aggregate costs using a minimum spanning tree [11]. On top of these methods, end-to-end deep neural networks also have a place in the field of stereo matching. DispNet was raised by Mayer et al. to create a large synthetic dataset to train an end-to-end deep neural network for disparity estimation [12]. Pang et al. [13] built a two-stage convolutional neural network to first estimate and then refine the disparity maps. GCNet [13] incorporated the feature extraction, matching cost aggregation and disparity estimation into a single end-to-end deep neural model to get state-of-the-art accuracy on several benchmarks [14].

Since our experiments were focused on three efficiency methods, we also searched several previous works related to efficiency improvement. Lutz illustrated the basic rule of early stopping in [15]. About weight pruning, Frankle and Carbin came up with a method of doing so, by stating the lottery ticket hypothesis [16].

Methodology

Original setup

To train the original GA-Net deep (or GA-Net 22), the following settings were used:

A batch size of 8 on four P40 GPUs with a crop size of 240x624.

Train the model on SceneFlow dataset for 10 epochs and finetune the model with KITTI 2015 and 2012 for 640 epochs.

Evaluation and comparison on SceneFlow Dataset [1].

Our aim is to make the training and testing process more efficient so that it is possible to implement this GA-Net on one Tesla P100-PCIE-16GB GPU for less than 12 hours without much accuracy reduction. For 640 epochs of training it would mean that the network should spend on average less than 67.5 seconds per epoch. The current GA-Net deep network would take an estimated 128.5 seconds per epoch to train on the designated Tesla P100 GPU and a batch size of 16. The training time should therefore at least be halved or the number of epochs needs to be halved. Furthermore, the inference time of the original network is on average 3.87 seconds. It is desired to also reduce this time as much as possible because if this network were used to determine the depth of a vehicle’s surroundings it should be able to handle 60 fps.

To achieve these results, four methods to make a network more efficient are researched. The implementation of these methods is described in the Experiments section.

Results to obtain

The following results will provide the data we need to conclude whether the methods make the GA-Net more efficient. It also provides a measure for the performance of the networks, so a good trade-off between efficiency and performance can be made. The results of the experiments will be shown in both qualitative (generated images) and quantitative (Training time, number of parameters, number of MACs, EPE error, error rate) form. Evaluation metrics are explained as follows.

Qualitative results

The qualitative results are disparity images that are generated with the network. The network outputs gray scale images (white for high disparity, black for low disparity). To increase the clarity of the images, they are converted to colored images in which dark blue stands for low disparity and red for high disparity. By comparing the disparity images with the camera and benchmark images, the performance of the networks can be assessed qualitatively.

Quantitative results

The quantitative results that are computed from the experiments are the EPE error, the error rate, the training time, the number of parameters, and the number of MACs.

EPE Error: The mean of the absolute difference between the predicted disparity values and the ground truth (or benchmark) disparity values of an image. This error is also used as the loss function in the GA-Net network.

Error rate: The sum of the number of occurrences at which the absolute difference between the predicted disparity values and the ground truth (or benchmark) disparity values of an image are higher than a certain threshold, divided by the total number of disparity values.

Training time: The average time it takes to train one epoch. It is computed so that the influence of the experiments on the training time (time efficiency) can be analysed.

Number of parameters: The number of parameters (weights and biases) that the network consists of. It is computed to measure the potential number of solutions that the network may output.

Number of MACs: The number of Multiply-accumulate operations (a product of two numbers that is added to an accumulator) within the network. The number of MACs is computed to measure how the experiments affect the computational efficiency.

Experiments

Dataset and validation

KITTI dataset [8] is jointly founded by Karlsruhe Institute of Technology in Germany and Toyota’s American Institute of Technology. It is currently the world’s largest evaluation dataset for computer vision algorithms in autonomous driving scenarios. The dataset is used to evaluate the performance of computer vision technologies such as stereo vision, optical flow, visual odometry, 3D object detection and 3D tracking in a car environment.

Since our code is built in Google Colab which can provide only one GPU, it is timesaving and space-saving to use a relatively small dataset to train our network. For this project we use a stereo subset from KITTI 2015 benchmark suite, which contains 200 training scenes and 200 test scenes. Original images from both left and right cameras can be found in the “image_2&3” folders of both the training and testset. Images in “disp_occ0” are used as ground truth images where the depth information is reflected in grayscale intensity. Normally, the trained model can be submitted to the KITTI leaderboard to get the model’s performance on the testset so there is no ground truth image provided in the testset. However, since we have to do multiple experiments for this project the submission way is not the solution. To solve the problem, we generate depth images from the original GA-Net pre-trained model as our ground truth images for the testset and evaluate performance based on that.

Another problem of the KITTI dataset is that it does not have an extra validation set, making it difficult to do operations related to the validation error like early stopping. The solution we chose is the hold-out method which splits the training set with a given ratio (1 to 4) into a validation set and a training set randomly. This method is a simple validation method.

Batch size

Batch size is a commonly tuned hyper-parameter that can influence the performance of a network. Increasing the batch size can improve space efficiency and reduce the number of iterations in one epoch. Also, the direction of the gradient can be more accurate and less fluctuated. However, a too large batch size will need more epochs and more time to get a similar accuracy compared with small batch size. In our experiments, we implemented the batch size of 16, due to space and time restrictions.

Effects of the efficiency methods

The methods that are researched are explained as follows.

Crop size

Obviously, a large crop size contains more information resulting in more accurate predictions but it also is more time and space consuming. Thus, we would like to do a trade off between the performance and the consumed time by tuning the crop size. In our experiments we researched the crop sizes 48x48, 48x96 and 48x144 pixels. This is compared to the original network that uses a crop size of 240x624 pixels.

The number of epochs

The original paper trains the GA-Net 10 epochs on the SenseFlow dataset and 640 epochs on the KITTI 2012 and 2015 datasets. Because of space limitations, in our experiments the networks are solely trained on the KITTI 2015 dataset. During the experiments, we train the networks for 640 epochs and save a checkpoint for every 100 epochs to measure the performance. Furthermore, we investigate the effects of early stopping on the network. Early stopping can prevent overfitting, which may improve the test results, and it reduces the training time.

The number of 3D convolution layers

The main contribution of GA-Net is to replace 3D convolutional layers with GA layers because 3D layers have more parameters to be learned while GA layers are more efficient. To make this GA-Net more efficient, we decided to test how the number of 3D convolution layers influence the performance. It is expected that more 3D layers can make the results more precise but cost more time and space, so our method is to keep the number of GA layers stable and evaluate the relationship between the decreased 3D layers and decreased accuracy. In our experiments we researched networks with 18, 20, 22, and 26 3D convolutions. The results of these experiments are compared with the original network that has 22 3D convolutions (and uses a larger image crop size).

Weight pruning sparsity

After a network is trained, weight pruning can reduce the number of computations significantly. This can result in reduced inference times. On the GA-Net model, weight pruning for the sparsities of 0, 25, 50, 60, 70, 80, 90, 95, 97, and 99 percent is researched. The pruning is performed on the 3D convolution layers of the model.

Results

In this section we will present the results of our experiments. For every category, first the qualitative results will be analyzed and discussed followed by the quantitative results. In the next section these results will be concluded.

Ground truth and Benchmark

The following images are the ground truth images from the KITTI 2015 stereo dataset [3]. Besides the ground truth images, we also compare our results to the Benchmark generated with the original GA-Net deep network [1].

Validation Images