# Automatic Building Height Estimation: Machine Learning Models for Urban Image Analysis
## 5 Artificial Intelligence Applied to Building Height Estimation
### Facade Detection: Semantic Segmentation Model  
The first step in any tool for data collection from images is to detect the spatial and categorical information of a certain object in the input images. In our case, the input images are street view images from the point of view of a car driver, from which we extract spatial information of buildings, cars, pavement and other objects. This is conducted with a semantic segmentation machine learning model that assigns a class to each pixel of the input image. We developed a fully convolutional neural network based on the HRNet encoder [25], which maintains a high-resolution representation of the input image during the whole model representation. The output labelled image has half the resolution of the input to enable faster runtime. The HRNet encoder Figure 4 splits the input image into two branches, one with half the original resolution and the other one with one sixteenth of the original resolution. The second branch acts as the bottleneck, while the first one acts as a high-resolution representation of the image. The downscaling process of the input image takes place gradually through the encoder, halving the resolution in the second branch in each encoder layer. Both branches are correlated together through cross-convolutions (Figure 5).  
Figure 4: Model encoder.  
In the encoder, ResNext (Figure 6) convolutions [23] are used and, subsequently, an atrous spatial pyramid (Figure 7) convolution [29] is applied. ResNext is a recently proposed model that uses depth separable convolutions [28] with an inverse bottleneck and achieves state-of-the-art results using less training parameters. DenseNet layers are layers that increase the feature dimension by concatenating outputs from convolutions to the input and by repeating this process multiple times [38].  
In the decoder (Figure 8), simple depthwise convolutions without a bottleneck are used. After applying an attention mechanism (Figure 6), skip connections coming from layers 5, 4 and 3 are fused with the decoder layers 1, 2 and 3, respectively, concatenating over the feature dimension of each branch. In every decoder layer, the second branch is upsampled until the size of the first branch is achieved in the last decoder layer, just before both branches are fused together using an attention mechanism. The attention mechanism assigns an attention, which establishes how important each pixel is and provides a criterion for fusing two images together.  
Figure 5: Encoder layer. The first cross-convolution is not applied, and mean pool is applied on branch 1 in the encoder layer 1. \(\times\) denotes array dimension and \({}^{*}\) denotes multiplication.  
Figure 6: Basic layers of the model. \(\times\) denotes array dimension and \({}^{*}\) denotes multiplication.  
The model is programmed in the Julia programming language [39] using Zygote [40] and Flux [41] libraries and cross-entropy [42] as the loss function for classification tasks. Optimisation is done with supervised learning using the ADAM optimizer [34]. Training is performed with a A100 Nvidia GPU on a Magerit supercomputer at Cesvima UPM. For model parameter initialization, a glorot uniform distribution [43] is used.