Road Feature Detection & GeoTagging with Deep Learning
By: David Yu
Tags: Ocr, Arcgis, GIS, Machine Learning, Object Detection

Finally, we must also take into account auxiliary information such as the bearing of the camera, which is made up of the bearing of the vehicle and camera yaw. We will for now, not consider other parameters such as camera pitch and roll and how they would contribute to changes in inferred distance.

More accurate distance inference with Deep Learning

The above approach to distance inference gave us a good starting point to improve upon, but is by no means foolproof. Apart from the issue of non-deterministic physical sizes on select classes of signs, we also have the issue of perspective distortion should a sign get really close to the camera. What we need to figure out then, is whether we can bake an additional prior understanding of size into the model itself. Where there’s a will, there is a way involving deep learning.

The idea of applying deep learning to solve distance inference is itself not very novel, but the approach taken by the monodepth library is a very refreshing one: Rather than train a convolutional neural net to minimize the loss between an inferred depth-map and some kind of ground-truth map informed by LiDAR data, monodepth uses a dual-camera set up to capture data for training. Back propagation in this case involves adjusting the network’s weights so that it is able to reproduce a right-lens image from a single left-lens input (and vice versa). The corollary is that the model builds up an intrinsic understanding of depth over time, and this is all achieved without any expensive data collection equipment.

monodepth output on one of our MMS images. The street sign is clearly visible, and has a uniform gradient, meaning monodepth has assigned it a fixed depth. There are some artifacts on the left because the model hasn’t been optimized on highways in the US.

To figure out which pixel value corresponds to which distance is then a simple regression problem and can be done automatically on the testing set where signs have a known location.

The relationship between brightness and distance appears to be a linear one. A more accurate regression line can be obtained via transfer learning with local images captured from a dual-lens setup.

After integrating this ML solution for distance inference to our programmatic method, we are now able to precisely determine an object’s geocoordinates from a geotagged image. Neat!

Metrics, Metrics, Metrics

Of the five design aspects that define a machine learning workflow (data, model architecture, loss function, optimization method and performance measure), it is performance measure that is the most commonly overlooked yet most crucial facet. Throwing together a ML model without designing appropriate metrics would be like building a car and calling it a finished product without ever turning the key to see if it runs. The issue is that most developers are too lazy to test their models against a set of standard benchmarks, or to come up with custom metrics from the outset where standard performance measures are clearly insufficient.

To do this properly, I have created three separate metrics for each of the three outputs of the model: lat/long inference, OCR and detection/classification. But before any of that, we would first need to designate a set of testing data against which future benchmark tests would be carried out. There are two advantages to keeping a fixed test set: First, we can be assured that when performing model comparison that all non-systematic errors are eliminated. Second, depending on where we wish to deploy the model, we could manually incorporate images from that testing space to assess the model’s ability to generalize to foreign environments. The only downside to a fixed testing set is that lat/long of PoIs would have to be manually assigned, which we have painstakingly done over a period of two hours. As it stands, the current testing set consists of 43 images which does not sound like a lot but is plenty to illustrate the current level of model performance.

Lat/Long Inference

Two pieces of data are collected to measure the accuracy of lat/long inference. First, the L-2 standard deviation of inferred locations. This piece of information lets us understand the spread of inferred coordinates, and is something we wish to reduce over time. The other is the probability distribution of bearing of line AB where A is the ground-truth location and B is the inferred location. Ideally, the bearing should have an even distribution, indicating a lack of systematic errors which might otherwise manifest as clusters around certain bearing values.

OCR

Two metrics are used to assess the OCR accuracy: Number of words detected (each associated with the inferred signage distance) and the percentage of correctly identified words for all detections (based on a similarity index produced by Python’s SequenceMatcher). The ground-truth data in this case are manually entered into a python dict. The implicit rule I used when writing these strings is that I must not spend more than a second to interpret the text from a sign, and that I should certainly write out an empty ground-truth string if I deem the text to be indecipherable. Nonetheless, in the case of OCR, the model did not produce very accurate outputs for the few signs that contained any text at all.

Some particularly difficult OCR metric images. If it takes you more than a second to figure out what’s written, it’s probably too difficult for current methods to solve.

Detection/Classification

This is one of the few cases where a standard metric does the job better than any metric you can concoct. Since the detection model consists of Mask R-CNN or YOLOv3, the recommended performance measure is mean average precision (mAP). A detailed explanation of what this entails is given here.

Traditional metrics for VOC-style detection tasks vs. new metrics for current use case. Of course everyone is free to come up with whatever performance measure they see fit, so long as these metrics make sense in the context of the problem.

Integrating into the ArcGIS framework

The model in its current iteration is capable of consuming mass amounts of raw data and converting those into useful features. It is however not a full-fledged solution because there are no mechanisms in place for an end-user to easily consume these features and hence cannot derive actionable insights. This is where an ArcGIS end-to-end solution offers the user a decisive advantage through offerings such as real-time status monitoring using Operations Dashboard, post-processing techniques such as pattern analysis, spatial relationship modelling and clustering methods exposed through various spatial statistics tool sets, as well as ground crew management and coordination platforms like Workforce or Collector.

On the topic of post-processing, we realized that a pitfall with using MMS data is that unique objects tend to get identified multiple times from different camera frames as the MMS vehicle approaches the PoI. This would not be an issue if lat/long inference or object classification can be carried out without error, however this is typically not the case.