This is my research. I want a detailed results and discussion section

Subsurface Utility Detection and Augmented Reality Visualization Using GPR and Deep Learning


Abstract
Recent urban revitalisation requires advanced utility management and innovative technology to achieve high-precision utility management. This research addresses the challenge of subsurface utilities detection and visualization by leveraging modern technologies specifically Ground Penetrating Radar (GPR), deep learning, and Augmented Reality (AR) to provide an automated means for accurate subsurface detection and visualization by developing a methodology consisting of four stages. The first stage involves GPR data collection using a multisensory GPR device, preprocessing for raw data decryption and fusion, noise reduction and signal enhancement, and zero-time correction for accurate signal synchronization with the depth and location of the subsurface targets. The second stage focuses on developing a Mask R-CNN deep learning model by preparing a dataset and training the model while integrating transfer learning and data augmentation. The third stage involves inference on case study data, refining results through profile alignment and Non-Maximum Suppression to increase accuracy to extract features as point cloud representation of pixel-wise location of detected features. The final stage implements AR for visualizing the extracted features by developing an AR application that incorporates spatial mapping and anchoring for precise model alignment and tracking. The developed system demonstrates promising results, providing an efficient utility detection and visualization solution.
Keywords: Ground Penetrating Radar; Mask R-CNN; Augmented Reality; Utilities Mapping and 
Visualization
	Introduction
Management of subsurface utilities presents a significant challenge where water, sewage, electrical, and communication networks spanning hundreds of thousands of kilometres highlight the complexity and scale of these networks. The maintenance and operation are vital for sustaining urban infrastructure and ensuring the serviceability and reliability of subsurface utilities. While specific figures may vary across counties and regions, the need for advanced technologies and efficient management strategies to perform maintenance or expansion of existing utilities remains a common concern to support sustainable development and ensure uninterrupted services to communities, minimizing risks to the workers and public safety. Therefore, utilizing a proper method for subsurface utilities detection and visualization is crucial to minimize risks associated with construction and maintenance.
Typically, subsurface utilities imaging and mapping are performed based on technological instruments implementing various geophysical theories which describe and explain the physical properties and processes occurring within the Earth [1] where current technologies for utilities detection using such theories. 
 GPR has emerged as a valuable technology in utility management by offering a non-destructive and efficient subsurface utilities investigation by accurately identifying subsurface utilities of various materials and at various depths. The fundamental operating principle of GPR involves the transmission of electromagnetic pulses into the ground and the reception of the reflected signals which have penetrated the subsurface and reflected with an amplitude that varies based on the dielectric properties of different materials that caused the reflections. By processing the recorded signals, a detailed cross-sectional image of the subsurface, also known as radargrams, can be generated [2], [3]. The process of operating principle is completed using: a transmission antenna that emits radar pulses, a recipient antenna to receive reflected signals, a control unit that manages the timing of pulse generation and processing reflected signals, and a data recording device to store the collected information for further analysis [4].
The main objective of this research is to address the challenge of subsurface utilities detection and visualization in the context of urbanization and infrastructure planning. The research aims to leverage modern technologies, specifically deep learning and Augmented Reality (AR), to automate the process of Ground Penetrating Radar (GPR) data interpretation and visualization of the obtained results. To achieving this objective, the research provides a comprehensive framework for utility detection and visualization to contribute to a more accurate and reliable outcomes. This includes acquiring raw data, employing innovative signal fusion algorithm for multiple data from different antennas, and processing fused data to enhance the signal quality and reduce the signal-to-noise ratio. It implements a state-of-the-art deep learning techniques to automate the interpretation of GPR data by exploring and implementing deep learning models to choose the most fit model for the research framework based on previous literature. Also, the research develops an augmented reality application for results visualization using an interactive and user-friendly AR application that enable users to visualize the acquired results as a point cloud representing pixels of detected objects while providing intuitive tools for filtering and spatial location annotations.
	Literature Review
Implementing multi-sensor data fusion and visualizing the information using the AR platform is an advanced field with substantial potential. However, the existing literature on this subject is limited and previous research related to GPR utilities detection mainly focus on the integration of GPR data with other technologies or on reconstructing 3D objects from GPR data. For instance, Ayala-Cabrera et. al. [5] introduced a framework to analyse GPR data by clustering perpendicular slices to build a trace. The framework includes data pre-processing to clean the sensor noise of the images and generate contours representing features of interest which undergo a classification process to generate the 3D model. The framework was evaluated in a laboratory-controlled environment using a water pipe in a sand-filled concealed wooden tank that revealed that the pipe visualization exaggerated the actual pipe diameter due to the GPR’s reflections. The proposed framework can be useful for workers with limited experience in GPR. However, the research was conducted on a single element with short length with no insight into the framework performance on complex buried utilities which reflects the real-world application.
Pereira et. al. [6] addressed the limitation of the previous research by introducing a framework that processes raw 3D multi-static GPR data as input and generates a 3D model using an enhancement filter tailored for tubular structures which are used in medical research that reduces background noise and to highlight features of interest. The research was conducted in a laboratory-controlled environment using various diameters and shapes of PVC pipes. The result showed that due to the ability to perform parallel computation with over 130,000 image points being computed in under five seconds which can be reduced further using cloud cluster computation. The framework algorithms along with the filtering algorithm provided a decent quality 3D model which generated remarkable results for U-shaped pipes with a narrow gap that could be mistaken by the algorithm as a single entity from the raw data.
	GPR Data Fusion and Interpretation 
Utility detection is vital for successful operation and sustainable service provision. During most tasks related to underground utilities that might include repair, replacement, or installation, the most common method of excavation is the open-cut method. This preference is most desirable because of the prominent level of uncertainties subjected to data provided by the utility owner, especially in old urban areas where records of existing utilities are mostly lost or not categorized and tracked properly.
Uninformed open-cut excavation method can damage other utilities and the traffic which in turn can disrupt serviceability and having social impact prompting countries such as the UK and Australia to map underground utilities [7]. For instance, The UK project named “Mapping the Underworld” focused on locating, mapping, and developing reliable records of underground utilities using a developed multisensory survey that includes GPR, Vibro-Acoustics (VA), low-frequency electromagnetic fields, and passive magnetic fields while yielding utility companies’ record data [7]. Furthermore, Dou et. al. [8] used an algorithm namely “Marching-Cross-Section Algorithm” to fuse data from GPR, Passive Magnetic Fields, Magnetic Gradiometer, Low-Frequency Electromagnetic Fields and VA while developing a novel formulation of the Extended Kalman Filter, an extension of the Kalman Filter for non-linear systems to estimate the state of a system subjected to non-linear transformation, to build a framework that generates virtual scan lines based on hypothesized detections when sensors do not provide common scan lines. The system performance is evaluated based on real data where it showed high precision of the system.
 Silvia et. al. [5] presented research that focused on leak identification and extracting features from GPR raw images to create 3D models for visualization by using subsequent interpretation while applying a variance filter to highlight anomalies.
Another framework presented by Dou et. al. [9] utilized probability distribution to formulate the mapping problem as an Integer Linear Programming problem where the final map represents the sum of the highest probability. Sensor measurements were categorized into hypothesized detections that reflect the presence of certain utilities and hypothesized detection-free measurements where a measurement was performed without extraction of hypothesized detections. The research found that the framework was efficient. However, the framework requires the detection of a single utility with a predetermined start and end points for improved accuracy.
	Deep Learning in GPR Data Interpretation
Deep learning, an innovative field within artificial intelligence, has revolutionized the landscape of neural network research. The initial studies comparing shallow and deep sum-product neural networks [10] revealed that deep networks requires exponentially more neurons in a two-layer neural network. And to better simulate the human neural system, the memory task had to be addressed which was addressed initially with a Self-Organizing Map (SOM) that involves utilizing unsupervised ANNs that organize high-dimensional data into low-dimensional representation while preserving the data properties of the input space which is a good tool for identifying clusters. However, SOMs may struggle with complex data distributions and need parameter tuning [11].
Neural networks advanced into utilizing a bidirectional networks that stores patterns and allowing retrieval from partial or noisy inputs [12]. Moreover, the Boltzmann Machine was introduced for memory tasks and modelling joint probability distributions over binary data and learns to represent complex patterns. However, it faced slow convergence and computational inefficiency [13] that was mitigated by the Restricted Boltzmann Machine by utilizing a bipartite structure without connections between hidden neurons making it suitable for large-scale applications and feature learning tasks [14]. And by utilizing it in staked layers, Deep Belief Nets emerged among other unsupervised learning models making them capable of learning hierarchical representations of the data and excelling in feature learning and hierarchical abstraction [13].
All these concepts and techniques have influenced the development of Convolutional Neural Networks (CNNs) by incorporating feature learning, unsupervised pre-training, and probabilistic modelling making them capable of learning complex patterns from raw data. Leading to improving performance in various tasks involving image classification, object detection, and natural language processing. This development led to CNN variants such as AlexNet [15], VGGNet [17], and ResNet which features residual blocks with skip connection [19].
Variants of CNN have driven Mask R-CNN architecture development through the advancement of object detection and instance segmentation techniques which started with a region-based Convolutional Neural Network (R-CNN) for objects localization using a region proposal network (RPN) followed by a classifier. Development of Fast R-CNN and Faster R-CNN improved both speed and accuracy but they lacked pixel-level segmentation [20], [21], [22] which is addressed by Fully Convolutional Neural Networks (FCNs) by enabling end-to-end pixel-wise detection. Leading the creation of Mask R-CNN which integrates FCNs with Faster R-CNN for instance segmentation and object detection [23]. Mask R-CNN extends Faster R-CNN by not only providing two outputs for each detected object, a class label and a bounding box, but providing a third output which is an object mask that requires extraction of a much finer spatial layout of an object through pixel-to-pixel alignment [23], making Mask R-CNN a multiple architecture model that involves 1) a convolutional network acting as backbone architecture for feature extraction, 2) a network head for bounding box classification and regression, and 3) mask detection applied separately to each Region of Interest (RoI). By employing ResNet, Mask R-CNN benefits the residual learning using a building block.
Transfer learning (TL) is an essential technique in deep learning for situations with limited data or where computational resources are constrained. TL involves leveraging a pre-trained model that has been trained on a large-scale dataset such as ImageNet and then utilized as feature extractors to capture high-level representations of images that can be transferred to the target task. Thus, TL requires fine-tuning a pre-trained model on a related task and dataset and then adapting the model to the specific target task. One of the most utilized datasets in training and evaluating CNN models is Microsoft Common Objects in Context (MS COCO) dataset includes a vast collection of annotated images of diverse and complex data which have been used to benchmark and evaluate the robustness and generalization capability of CNN-based architectures in various computer vision tasks [21], [23], [24], [25].
The application of deep learning has shown a promising result in automating and enhancing the GPR data interpretation. This development is illustrated in a study by Kang et. al. [26] that proposed a system based on CNNs for autonomous underground cavity detection beneath urban roads using GPR data where the system was capable of distinguishing between different underground objects such as cavities, pipes and manholes yet found difficulty in identifying the cavities from underground rocks or gravels. 
Another study by Hai et. al. [27] focused on the application of deep learning in GPR data using You Only Look Once Version 3 (YOLOv3) which provided a bounding box around the detected hyperbolic features of existing subsurface utilities. Nevertheless, YOLOv3 is a black box model which in return limits model configuration. Additionally, the provided bounding box would require expensive computational costs and further analysis to extract the information to be used in 3-D model visualization. Another framework proposed by Hou et. al. [28] used Mask R-CNN for pixel-level detection, showing promising results in detecting hyperbolic features of concrete bridge deck’s rebars which requires vast amounts of real data for training and focuses on rebar detection and localization. Therefore, the study does not involve the real-world applications of utility detection using GPR data which poses the existence of various elements and noise that affect the interpretation of obtained data compared to data from concrete segments which provide a uniform and obvious hyperbolic signature. Previous research conducted the last five years mostly utilized CNN variants and other deep learning models to GPR data for buried objects identification such as rebar [29], wires [30], and diameter estimation [31], subsurface structure detection such as underground cavities [32] and tree roots detection [33], pavement and soil analysis such as asphalt.
	Augmented Reality in Subsurface Utilities Visualization
Accessibility to reliable subsurface utilities information is crucial for effective planning that limits the damages and service disruption to utility operations. Augmented Reality (AR) provides a real-time means of visualization that results in informed decision-making. This transformative technology offers interactive and immersive solutions for utility professionals and addresses challenges associated with the traditional methods.
AR applications for subsurface utilities depends on precise coordinate systems to accurately align of virtual information with the physical environment. The coordinate system minimizes deviations and misalignments by integrating key components including: 1) Integration of geographic datasets for consistent spatial reference, 2) Reference datum for the origin point, orientation, and scale of the coordinate system to ensure a consistent reference for measurement, and 3) Measurement units for quantifying distances, angles, and geometric characteristics.
The coordinate system is crucial aspect of integrating AR into applications where various types of coordinate systems are employed in AR geospatial applications such as Geographic Coordinate Systems (GCS) that serve as a reference framework to specify locations on the space’s surface by utilizing latitude and longitude coordinates to define objects positions relative to the Earth’s equator and prime meridian. Studies highlight the importance of utilizing GCS in AR applications for outdoor navigation and location-based services to provide spatial accuracy. Additionally, GCS incorporates other components such as an angular unit of measurement, Greenwich as a prime reference and a spheroid datum base. All these components serve the establishment of a grid system of parallels and meridians for accurate spatial reference [34], [35]. Conversely, Projected Coordinate Systems (PCS) transforms GCS coordinates into a two-dimensional surface or a planar system. The location of PCS is defined by (X, Y) coordinates on a grid related to the origin (0, 0) [36], [37], [38]. These coordinate systems differentiate AR types which can be classified into two categories. The first one is Marker-based AR which relies heavily on PCS with  markers or visuals to trigger and position virtual models whilst determining the model centre, orientation, and range [39], [40], [41], [42]. The second type is Markerless-based AR which relies on GCS while some cases use PCS. Thus, this type consists of optical-mechanical, gyroscopes, accelerometers, magnetometers, or GPS tracking systems that recognise the end-user location and movement [43], [44], [45], [46], [47], [48]. Figure 1 illustrates different types of AR localization.
 
Figure 1: Augmented Reality Types
AR might be used in complex environment, such as construction sites, with high risk that must be minimized [49]. Optical See-Through Head-Mounted Displays (OST-HMDs) are recommended as they offer more safe solution where they provide higher transparency to the virtual model and better visibility of the real world, preventing occlusion caused by handheld devices that causes distraction and making end-user partially distant from the surrounding environment. Previous AR visualization research focused on problems related to depth perception, parallax effect, and positional accuracy. For instance, Becher et. al. [50] evaluated the usage of Naïve Display, Projection Display, Grid Display, and Projection Grid Display to evaluate depth perception where they used Microsoft 1st generation HoloLens by showing six simple grey cubes at different distances and depths and randomly rotated to avoid comparison bias between each cube with the others and participants had to classify the cubes by altitude and then order the cubes by distance. The results then were analysed statistically and showed that Projection Display and Projection Grid Display had a better user experience regarding depth perception.
Stylianidis et. al. [51] overcame the depth perception issue by visualizing buried infrastructure in a topographic view where all buried pipe mains are projected on the pavement level regardless of the depth of the utility and retrieving models from a geodatabase. The framework evaluated the accuracy of the GIS database and GNSS integration which participants agreed upon with 14% of participants feeling below average. On the other hand, Piroozfar et. al. [52] developed a mobile application using a unity game engine integrated Application Programming Interfaces for AR to provide 3-D AR functionality then integration with GPS functionality was implemented into the application. However, the application had issues with the lost performance and overlaying multiple models at the same time which was addressed by applying modifications to the user interface to place hotspots indicating user location based on GPS. Hansen et. al. [53] used Reality Capture to reconstruct 3-D of the excavated area during the construction of utilities using Close-range Photogrammetry to create 3-D reconstruction of the excavated utilities areas for AR visualization. They performed video recording of the exposed utilities during construction to generate point clouds. Afterwards, the geo-reference of the point cloud was set to ensure that the location and scale were aligned with the actual location of the area.
	Research Methodology
The proposed framework aims to address challenges related to the complexity of interpreting GPR data and interpretation visualization. This proposed framework combines the strengths of GPR, deep learning, and AR to provide a comprehensive solution for utility detection and visualization. The proposed framework starts with collecting the GPR raw data representing the reflected signals from subsurface objects and leverages the unique ability of GPR to detect both metallic and non-metallic utilities. Furthermore, the raw data is introduced to modules for data decryption and further processing and filtering to obtain a dataset that can be utilized in the subsequent stages. 
Due to the complexity of GPR data interpretation, deep learning techniques are utilized to automate the interpretation by using a supervised learning and segmentation model effective in pattern recognition tasks. To achieve this target, the framework utilizes Mask R-CNN, a state-of-the-art deep learning model, for pixel-level detection and segmentation of hyperbolic and attention change features in GPR data indicating the presence of buried objects. The detected objects are then visualized using AR technology that merges real and virtual worlds to produce an environment where physical and digital objects co-exist and enable user interaction in real time. 
	Framework Architecture 
Allowing an immersive visualization and enhancing the understanding and interpretation of the results. Hence, the purpose of the proposed framework is to automate subsurface utilities detection and visualization by targeting two main objectives: 1) Identify, in an automatic technique, buried utilities using GPR data, and 2) Visualize processed detections obtained from the trained model using Augmented Reality. The first objective is achieved through executing the following stages: 1) GPR data collection and processing, 2) Mask R-CNN model training and inference, and 3) Object Detection and post-processing. The second objective is achieved through implementing AR visualization which involves project setup, scene design, spatial mapping, and interaction, anchoring and tracking, User Interface (UI) and visual effects, and HoloLens features integration. Figure 1 illustrates the proposed framework stages and processes.
The data collection and processing stage within the proposed framework serves as the foundational stage upon which the subsequent stages’ analysis and interpretations are built. It involves a series of processes to extract the most insights from the GPR data. This stage involves the collection and processing of GPR data which presents subsurface objects, changes in material properties, and voids based on the reflected signals that are being detected based on attenuation of the generated signals by performing a set of processes. The process begins with collecting GPR data using a GPR device to scan areas where buried utility locations are known by cross-referencing scans with the as-built data. However, to maintain a comprehensive data acquisition, it is crucial to consider antenna frequency selection and time window determination for optimizing depth penetration and resolution for efficient subsurface mapping during the scanning.
The following process involves raw data decryption which is a critical process due to GPR device proprietary format that requires decryption before it can be processed. Data decryption can be performed through reverse engineering the raw data and tailoring a decryption algorithm based on the raw data investigation. Data investigation can be performed using HEX editor to understand its format and determine the raw data header and the raw data. The decryption is achieved by assigning a pointer to seek and read characters from the raw data file and convert them in a bin-reading manner in trials of different bit configurations and convert the bin readings to ASCII data to obtain the raw data from each sensor. To enhance the data, the framework integrates a sophisticated data fusion mechanism to harmonize information collected from multiple GPR sensors, which is achieved by merging the data of each sensor data using a spectral forecast algorithm to merge separate datasets into a unified data set which requires signal resampling if collected data have different sampling rates to ensure uniformity in temporal domains across datasets. This process allows datasets fusion while mitigating noise.

 
Figure 1: Proposed Framework Flowchart
After data fusion, the data undergoes preprocessing to improve the quality and reliability of the signal even further by performing advanced noise filters and data correction while utilizing wavelet transforms and A-scan signal analyses including signal gains to minimize distortions or anomalies in the data and zero-time corrections. Lastly, data processing is utilized to filter out datasets that do not match the Signal-to-Noise Ratio (SNR) threshold and retain only datasets that meet the predefined SNR thresholds for further analysis and to generate reliable B-Scan images which can be used in the subsequent stages. The entire stage’s input and output for each process is illustrated in Table 1.
Table 1: Data Collection and Processing Processes Input and Output List
Stage Process	Input	Output
Data Collection Process	Parameters Definition.	Profiles Raw Data Files.
Raw Data Process	Profiles Raw Data Files.	1- Scan Parameters,
2- Arrays of Scan Data.
Data Fusion	Arrays of Scan Data.	Fused datasets for each profile.
Data Preprocessing		Scan Parameters,
	Arrays of Scan Data.		Modified Scan Parameters,
2- Arrays of different filtering configurations with the same sampling rate.
Data Processing	Fused dataset.	B-scan images of the filtered-out dataset.
The second stage of the proposed framework involves training a Mask R-CNN model. This stage provides a trained model that can be used in detection for further projects and involves a set of processes (see Figure 2). The first process in the stage involves creating a database of the training, validation, and testing datasets. The database contains labels for masks indicating the region of interest in each image, representing object signatures in the B-Scan, in a COCO JSON format [23]. The generated JSON file contains signatures mask coordinates, labels category, and bounding box coordinates. The annotated dataset is then split into training and validation sets where the training set is utilized for the model training whilst the validation set is utilized to evaluate the model’s performance. Before the commencement of model training the following processes are considered: 1) Model selection: based on the dataset and the complexity of the problem definition, the base architecture proposed for the model is ResNet-50 for its performance compared to ResNet-101 and 2) Model configuration: the model is configured with four classes representing utility, other objects, noise, and background along with other parameters.
Model training involves incorporating methods that include: 1) Transfer learning by using pre-trained weights from a trained model. The weights serve as an initialisation for the model for faster convergence and generalization improvement for new tasks, 2) Data augmentation which involves cropping, scaling, brightness and contrast adjustment, and colour manipulation to mitigate overfitting and to increase the dataset, and 3) Training metrics to monitor Total Loss and accuracy training metrics. Based on the best-trained model, acquiring detected objects is performed on obtained data and undergoes postprocessing to remove any duplicated masks located within the same region using Non-Maximum Suppression (NMS).
  
Figure 2: Mask R-CNN Training and Inference
The third stage involves enhancing the detected objects which is crucial as it translates the model’s detections into actionable insights that will be visualized to the end user as a set of spatial point clouds acquired by performing a set of processes illustrated in . Since the inference for object detection is performed on multiple filtering configurations for each profile, the first step in this stage is obtaining common features for each profile and translating them into one image. This is performed by checking each feature location for all configurations generating a binary instance matrix based on their occurrence and removing instances that do not meet the minimum occurrence threshold. The following step is to align all profiles so that all profiles share the same start or end points based on Common Feature Extraction to determine shifting value and direction for each profile based on the referenced profile. After aligning all profiles, a further post-processing step is performed to remove any masks that do not meet a threshold to minimize negative positives by comparing the occurrence rate of each mask across all profiles. Lastly, the filtered masks are translated into a binary matrix where each pixel location is converted into coordinates while taking into consideration 1) depth calculations based on wave travel time and 2) profile spacing. The binary matrix is translated into a point cloud which is a set of data points in a 3-D coordinate system that represents the detected utilities. Following the last post-processing step, model evaluation is utilized using the mean Intersection over Union method (mIoU).
The last stage aims to visualize the output of the object detection process through AR by setting up the environment by performing several processes. The first process is to set up the project by choosing the appropriate software and hardware tools for the task and ensuring their compatibility to prevent any further issues during development or model build. The hardware utilized is MS HoloLens [54] along with Microsoft’s Mixed Reality Toolkit (MRTK) for Unity [55]. This combination provides a set of components and features to accelerate cross-platform AR app development in Unity whilst maintaining smooth integration between core functions.
The next process is designing the scene by setting a wireframe of the scene and creating a virtual environment where the AR content is displayed using Unity Scene View to select and position the model, cameras, light source and other types of objects. The third process is spatial mapping which is a key feature of AR that allows the hardware, the HoloLens, to understand and map the physical environment through embedded cameras and sensors to create a 3D model of the real environment which is used to enable interactions between the virtual objects and the real world. Afterwards, visual effects and user interface are used to enhance the realism and immersion of the AR model by including a floating control dock, floating informatic text illustrating the coordinates of object that falls within the user’s gaze point, and information and help bulletin. The final process is integrating the features provided by the HoloLens such as gesture recognition and voice commands which can greatly enhance the interactivity and immersion of the AR experience.
	 Framework Implementation
To demonstrate the practicality and effectiveness of the framework and to prove that it can be utilized to achieve desirable outcomes in similar situations, the proposed framework is applied in a real-life case study. The case study is performed in Mostakbal City, New Cairo, Egypt (see Figure 3) which is planned over an area of 5,200 acres consisting of three phases. The first phase rolls out on 1,490 acres while the second and the third phases are 1,607 and 1,875 acres respectively serving residential areas and regional services centres. Due to construction progress at the time of model implementation, the case study focuses on the first phase of Mostakbal City since it had most of the utility’s construction completed. The model implementation was performed based on the framework stages where each of these stages is performed utilizing the appropriate tools and software.
 
Figure 3: Mostakbal City Location
	Data Collection and Processing
	GPR System Selection and Configuration
The data collection process for the model implementation involved the use of an IDS RIS MF Hi-MOD GPR system with two antennas. The first one is 400MHz to provide high penetration depth while the second one is 2,000MHz to provide high resolution for shallow depths. These selections were based on the utilities as-built located in the scanned area and are used as a reference for data annotation. The scanned utilities consist of potable water and irrigation main at depth range of 1.50 up to 2.50 meters, gas lines within 1.50 meters, and electric duct banks within 1.20 meters. While some other sub-main utilities are located within 1 meter.

	GPR Data Collection
The GPR system is utilized to scan various sites within phase one of Mostakbal City. The performed scans are serving two purposes to: 1) obtain training data and validation data for Mask R-CNN model training and 2) obtain unseen data for the framework performance measurement and to be utilized in the AR visualization. The total scanned profiles are 114 profiles including two sets. The first set has 54 profiles with an average length of 30 meters at zones with congested utilities serving as training and validation data. The second set has 60 profiles with an average length of 20 meters with two duct banks serving as framework performance measurements and complete framework implementation. Figure 4 illustrates a schematic representation of buried utilities at the case study location.
 
Figure 4: Schematic Representation of Buried Utilities at Case Study Location
The scanning process involves: 1) Determining grid type (uni-directional or bi-directional grid) which is set to be a bi-directional grid for zones with utilities at both longitudinal and transverse directions and uni-directional scan for zones with utilities at the traverse direction of the main carriageway of the road, 2) Determining spacing between scanned profiles which vary from less than 25 cm to 50 cm and performed either in uni-directional or bi-directional grid based on the utility’s direction. Nevertheless, the line spacing is determined using Equation (1) which lead to line spacing of 37 cm for the 400MHz antenna and 10 cm for the 2000MHz antenna. However, due to the utilized system being a combined system, an average of 20 cm is applied, 3) System configuration (time window, sampling interval, sampling rate per trace, step size, and stacking) is set to the factory base configuration for each antenna, 4) Locating and marking scanning zones’ corners with a total station for accurate physical location on site (see Figure 5), 5) Defining the origin point of the scanned zone by selecting a corner and setting the scanning perimeter and profile spacings, 6) Preparing the system software for data acquisition and performing system calibration, job name, and profile ID, and 7) Beginning data acquisition while employing string as a guidance system of profile scanning.
Spacing=c/2f 								(1)
Where,
f: Frequency (Hz)
c:   Speed of light (m/sec)
 
Figure 5: Locating Zones of Interest
	Raw Data Decryption
After obtaining the data, the first process is decrypting the raw data. This is performed by investigating the raw data file to determine both the address ranges for the header and data sections. The investigation is performed using a HEX editor to display both memory addresses and the data field (see Figure 6-a). In the memory address, each address uses a base of 16 in a hexadecimal format or a base of 10 in decimal format and each address refers to a specific memory location. On the other hand, the data field contains 16 bytes (base 16) of data, that is located at the corresponding address, and each pair of the hexadecimal characters corresponds to one byte or 8 bits. This is helpful to generate a pointer and assign its location accordingly. Lastly, the investigation concludes the address range of both the header and the data and the marker that indicates the start of the data section (see Figure 6-b). The second step involves reading the raw data using pointers to parse the header information and extract various metadata and start reading the data matrix by iterating through the binary data and storing the metadata until it encounters the data marker and starts storing the GPR data while incorporating data conversion functions (see Figure 6-c). Ultimately, this process results in acquiring the decrypted raw data.
	Data Fusion
The data fusion process combines both the acquired data from the 400MHz and the 2000MHz antennas to create a comprehensive and informative dataset surpassing the single dataset provided by each sensor separately. This is performed by applying a spectral forecast algorithm to merge the data into a unified data set by analysing the spectral characteristics of each sensor data, optimizing its information content while minimizing noise and interferences. The developed function resamples the data and iterates over every trace of both sensor datasets and compute algorithm parameters using Equations 2 and 3 [57] to calculate fused values based on these parameters. The function outputs a list of dictionaries containing signals, along with relevant metadata of each dataset where each dictionary contains the resampled signal array, resampled signal time, and profile metadata. Overall, these functions provide comprehensive functionality for resampling and fusing signals which is then exported as a data array as shown in and the process I/O is illustrated in Table 2.

 
a) Raw Data Displayed Using a HEX Editor
 
b) Raw Data Displayed Using a HEX Editor
 
c) Data Conversion Functions
Figure 6: Raw Data Decryption
Fused Data=MR/(Max_A1 )*T_A1+(Max-MR)/(Max_A2 )*T_A2					(2)
MR=(MR_A1+MR_A2)/(2*BIAS)								(3)
Where,
MR: Middle range of both antenna 1 and antenna 2 resampled trace
Max_A1: Maximum of the absolute resampled trace value from antenna 1
Max_A2: Maximum of the absolute resampled trace value from antenna 2
Max: Maximum of the absolute resampled trace value from antenna 1 and antenna 2
T_A1: Resampled trace value from antenna 1
T_A2: Resampled trace value from antenna 2










Table 2: Data Fusion Process
Input	List of dictionaries of raw data
 
Data from 2000MHz sensor
 
Data from the 400MHz sensor

Process		Performing a paired iteration loop that processes two datasets at each iteration and skips them the following iteration,
	Check data direction based on file structure,
	For each dataset, compute its parameters, and calculate fused data.
Output	The updated list of dictionaries includes: 
Updated parameters: profile direction, profile name, fused data matrix, signal time, and position.
Sorting technique: Sorted in two levels: the first level is profile direction and the second level is profile name.
 
Fused Data

	Data Preprocessing and Data Processing
This process is performed to enhance the quality of the obtained scan data after decryption and fusion to acquire a reliable data for efficient interpretation, this process includes preprocessing steps accounting for data corrections. The process input is the data of each profile and applies filtering configurations based on Table 3. These configurations are obtained through iteration and manual checks for the proper range that does not cause important feature removal. The process starts with performing a low amplitude filtering function based on the iterated threshold by applying a low-pass Butterworth filter to the fused data while calculating the signal-to-noise ratio (SNR) for each filtered signal and retains the ones that maximize the SNR.
The preprocessing process begins with the application of noise filters to remove any unwanted signals or disturbances that might interfere with data interpretation. This is achieved by removing: 1) the system noise between antennas, 2) wave interference, 3) other material direct reflection, and 4) other external EM interference. Wavelet transform and A-Scan (or one dimensional) signal analysis are used to make the removal by performing: 1) Low amplitude filtering: utilizing threshold-based filtering to eliminate signals below or above certain amplitude thresholds and retaining filtered signals based on their deviation from the threshold values based on Equations 4 to 7 and 2) Dewow filtering: by applying a suppress low-frequency noise components and removing long-term trends or drifts in the signal caused by instrumental or other external factors by using Equations 8 to 10. Afterwards, data corrections are performed to adjust for any inconsistencies or errors in the data including performing signal gains to compensate for signal attenuation (decrease in signal strength as the signal travels through the ground). This correction factor amplifies weaker signals yet maintains signal integrity throughout the data and is performed utilizing Equation 11 and zero-time correction by utilizing Equations 12 to 14.
F_S≥2*F_max									(4)
F_nyquist=F_s/2								(5)
ω_c= F_(cut-off)/F_nyquist							(6)
H(s)=(X(s))/(1+(s/ω_c )^2n )								(7)
Where,
F_S: Sampling frequency.
F_max: Highest frequency component of the signal.
ω_c: the cutoff angular frequency.
H(s): Transfer function.
X(s): Input signal.
S: The complex frequency variable.
n: The filter order where 1st order has a roll-off rate of 20db/decade (logarithmic nature of the decibel scale).
x(t)= ∑_(n=1)^N▒〖X_n cos⁡(2πf_n t+∅_n)〗						(8)
Y(f)= {█(X(f).H(f)         if f is within the selected range@X(f)                  if f is outside the seleceted range)┤		(9)
y(t)= F^(-1) (Y(f))								(10)
Where,
X_n: Amplitude of nth component.
f_n: Frequency of the nth component.
∅_n: Phase of the nth component (shift in the starting position.
F^(-1): Inverse Fourier Transform.
Y(f): Dewow-filtered signal.
G(t)= {█(T.F(t)                     if F(t)>T_threshold@F(t)                           Otherwise                  )┤				(11)
Where,
G(t): Gain-enhanced signal.
T: The amplification factor.
F(t): The original signal.

Time=Distance/Velocity								(12)
Distance=(X^2+h^2 )^(1/2)							(13)
Time=(2(x^2+h^2 )^(1/2))/Velocity								(14)
Where,
x: Horizontal distance between surface points from sensor and the object.
h: The depth to the anomalous zone.
Moreover, each function has an embedded callable plotting function to plot the radargram of the data and saves an image file to the configured directory with each file named in the following manner “Scanning Direction / Profile Number – Low Pass Filter Value – Power Gain Value – Dewow Filter Value.PNG” to test the filtered data based on the plotted radargrams. This allows the process to output the filtered data array based on each configuration by utilizing the functions to complete the process which its I/O is illustrated in Table 4.
Table 3: Filters Range
	Min. Range	Max. Range
Low Amplitude (GHz)	0.3	2.1
Dewow (window)	2	10
Signal Gain (Factor)	1.05	1.3
Table 4: Data Pre-Processing and Processing Process
Input	List of dictionaries of fused data
 
Fused Data

Process		Applying multiple filters each with its configuration.
	Performing SNR to filter out datasets.
	Performing Zero-time correction.
Output	B-Scan Images
 
Set of filtered data

	Mask R-CNN Model Training and Inference
After obtaining the filtered dataset and concluding the first stage, the second stage is initiated by performing two main processes. The first process includes obtaining a reliable trained model for future GPR data interpretation whilst the second process involves utilizing the trained model for data interpretation in the case study and evaluating the results. The following subsections will discuss each process and its involved steps by illustrating the implementation of each step and the impact of the subsequent steps and processes.
	Data Preparation
The data preparation process involves several steps such as dataset processing followed by data annotation and training set generation. The first step is considered a redundant step in the current model since the generated dataset does not include any padding or plotted axes or titles. However, it is recommended to remove any of these items while performing data augmentation since it had a negative impact during the training process. The second step is data annotation, in this study data annotation is performed using (makesense.ai) which is an open-source and free-to-use web application designed to assist in image annotation for computer vision and deep learning tasks. This tool has the advantage of supporting multiple label types including bounding boxes, lines, points, and polygons and the annotations can be exported in various formats such as VGG JSON format or a CSV format. 
The data annotation is performed considering the following procedures: 1) Loading the dataset into the web application and selecting “Object Detection” task, 2) Setting labels using three types of labels, a label for high attenuation hyperbola (Hyperbola_T1), a normal hyperbola (Hyperbola_T2), and a third annotation for areas within hyperbolas that have been affected by the applied filters (Object), 3) Starting the annotation using polygons, and the tool automatically generates the corresponding bounding box data that will be used during the training process (see Figure 7), and 4) Exporting the annotation data in various formats including VGG JSON which will be used as input to the Mask R-CNN model. This process I/O is illustrated in Table 4.

 
Figure 7: Drawing Polygons Around Region of Interest
 
Table 4: Data Preparation Process
Input	B-Scan Images
 
Set of filtered data

Process		Setting object labels.
	Annotating the objects.
	Export the annotation in VGG format as a JSON file.
Output	JSON database file including the data annotation in a VGG format.
 
Annotations
 
Exported file

	Model Preparation and Training
Once data annotation is complete, model preparation is initiated by choosing the proper model and applying the suitable configuration. The chosen model is Mask R-CNN for its configurability and the capability to provide pixel-level object detection for instance segmentation by utilizing ResNET-50 as its base architecture due to its balance between computational efficiency and model performance and is configured while considering its compatibility with the machine specifications as illustrated in Table 5 according to configurations shown in Table 6
Table 5: Machine Specifications
Hardware	Specifications
GPU	T4
Clock rate (GHz)	1.59
Number of Cores	40
Memory (GB)	16
Max Memory Bandwidth (GB/sec)	300
RAM (GB)	52


Table 6: Model Hyperparameters
Hyperparameter	Configured Value
GPU Count	1
Images per GPU	4
Steps per epoch	500
Validation steps	15
Backbone	ResNet-50
RPN Anchor Scales	[32, 64, 128, 256, 512]
RPN Anchor Ratios	[0.5, 1, 2]
RPN_NMS_Threshold	0.70
Minimum Image Size	800
Maximum Image Size	1024
Image Channel Count	3
Detection Minimum Confidence	0.80
Detection NMS Threshold	0.3
Learning Rate	0.001
Learning Momentum	0.9
Weight Decay	0.0001
Number of Classes	4 (3 classes + background)
ROI Positive Ratio	0.33
Pool Size	7
Mask Pool Size	14





















Afterwards, the model is prepared by developing functions building the Mask R-CNN model and its architecture. The codebase is an open-source project available on the GitHub platform developed by Matterport which contain scripts written in Python and utilises TensorFlow and Keras deep learning frameworks and includes functions to complete object detection, instance segmentation, and image segmentation tasks. Additionally, the codebase consists of key components such as model architecture, data preprocessing functions, training pipeline, and evaluation metrics which can be modified based on the required configuration with a well-documented codebase and the ability to download pre-trained model weights for transfer learning.  However, during implementation, some bugs have been solved to account for full integration with Google Colab and with the developed inference modules. Thus, functions and methods have been upgraded to match recent Python and package versions.
Additionally, some preparation and changes in the Google Colab Script Kernel have been performed to allow full access and integration with the Google Colab remote machine. This includes configuring the Google Colab Script environment and the remote machine to prevent any combability issues during the model training by configuring the kernel and environment to match the scripts uploaded to the remote machine and to match the remote GPU hardware. This involves downloading and installing NVIDIA GPU-accelerated deep neural network library (cudNN) along with CUDA. Afterwards, it is required to restart the kernel to refresh the installed framework and kernel. Another resolved conflict related to “Protocol Buffers” which is a method for serializing structured data and used for efficient data interchange between different systems and languages and utilized to specify model parameters, training settings, and any related configuration definition for the model by providing a standardized and efficient mechanism for defining data structures, serializing data, and facilitating communication between different components of the system and combability issues are resolved for required packages.
Afterwards, the following steps are performed to prepare for model training and transfer learning: 1) Loading the customed configuration of the model, 2) Determining the initial number of epochs, 3) Configuring the data augmentation parameters, 4) Setting Transfer Learning variables to import the last trained model, 5) Initiating model training, 6) Due to runtime limitations associated with Google Colab, a function is developed to export trained models for further analysis and determining the best epoch of the last training process, and 7) Continue training with transfer learning from the last epochs’ best model.
These processes result in obtaining a trained model that can be further analysed to obtain the best-trained mode. To enhance the recognition and detection performance of the model, a pre-trained model is used to promote the training process on the task problem. Therefore, the training process included a total of 100 epochs with an initial Transfer Learning using a pre-trained model, the second Transfer Learning is performed after epoch number 60, and the third Transfer Learning is performed after epoch number 80. During the learning process loss metrics were recorded (see Figure 8) along with the execution time for each record that has been collated for each stage in Table 7. The aforementioned processes included in this stage are summarized in Table 8 illustrating the input and expected output of the stage.
Table 7: Training Time
TL Stages	Number of epochs	Total Runtime (hr)	Average Runtime per epoch (min.)
Initial Stage	60	23.51	23.51
Second Stage	20	7.11	21.32
Third Stage	20	6.95	20.84
Total	100	37.56	
 
Figure 8: Model Training Loss and Validation Loss
Table 8: Model Preparation and Training Process
Input	B-Scan Images Dataset
Annotation Database
Pre-trained model on the COCO dataset
Process		Determining Mask R-CNN configuration,
	Building Mask R-CNN model.
	Integrating Google Colab Hardware by developing a training script with all the necessary kernel and firmware modification/setup,
	Calling Mask R-CNN python script model,
	Calling images dataset and the annotation database,
	Testing the integration of all performed calls,
	Setting data augmentation parameters, and
	Start training while incorporating transfer learning and data augmentation.



Output	Trained models in *.h5 format.
	Inference and Detections Post-Processing
During the training process, each epoch has two loss functions: training loss function and validation loss function. The epoch run time is recorded based on the processor time difference between the epoch start and its finish. These metrics are then employed to assess the performance and efficiency of each model where the models with the lower validation loss values indicate better performance. Therefore, plotting these metrics is necessary to visualize their convergence and to determine the lowest validation loss without overfitting. Moreover, to prevent spikes generated from TL, the first three epochs of each Transfer Learning stage are removed for proper interpretation. The assessment is then validated by using Equations 14 and 15 which directs to selecting model no.93 as the best model. 
Combined Score=  (Loss+Val_Loss)/2						(15)
Efficiency Metric=Combined Score+1/(Epoch Time)				(16)






Model no. 93 is then utilized to detect utility locations by using a developed function that utilizes the trained weights. This function enables generating masks representing detected objects on several radargram images through the following steps: 1) Reading radargram images with several configurations of each profile, 2) Executing the object detection function to obtain mask instances (see Figure 9), and 3) Exporting detected masks as arrays which are utilized for post-processing.
 
Figure 9: Detected Objects
The post-processing is performed in three steps. The first step involves performing post-processing to keep masks that meet certain thresholds for different filtering configurations for more reliability. The second step is aligning profiles to have a common start and finish. The third step involves performing pixel reduction and generating a point cloud to obtain an optimized performance during visualization. The first process in the stage refines the raw masks by performing a set of sequential steps that involves iterating through all detected object masks of each profile’s filtering configurations and generating a binary instance array. This enables checking mask instance occurrence frequency across all filtered configurations. In case the occurrence frequency does not meet the minimum threshold, the instance is removed. This process is looped on all instances for all configurations to produce one binary instance array for each profile as illustrated in Figure 10. The second process involves aligning profiles so that all profiles are aligned to the same start and endpoints. This is achieved by setting a datum profile and performing feature extraction of the datum profile and other profiles independently then setting a reference point from the datum profile to ensure proper alignment and find shifting values of each profile. Finally, shifting values are performed to each profile which results in all B-scans being aligned to the same reference point. The third process involves performing pixel reduction on each profile instance to improve AR model performance and generating a point cloud accordingly. Moreover, for better understanding, Figure 11 illustrates the involved processes in the stage. 
 
Figure 10: Feature Extraction and Retention of Valid Masks
 
Figure 11: Inference and Detections Post-Processing
				
		
	

	
	
	

	

	
		
	

		
		

	
		
		

	
	
	

	
		
	
	Augmented Reality Visualization
This stage involves the use of several tools and preparation that are crucial for the development of the application which includes: 1) Adobe Illustrator: a vector graphics editor that allows the design of VuMark and deployment in Unity, 2) Visual Studio: an integrated development environment used for C# scripts coding, debugging, and building the AR application into the HoloLens, 3) Unity Hub: a management tool for unity projects, 4) Unity Editor 2018.1.9f2: enables creating interactive 3D content and developing AR applications, 5) Unity Metro support: enables development and building applications for the windows 10 platform including HoloLens, 6) Unity Universal Windows Platform support: leverages native Windows features and APIs, 7) Unity Vuforia AR support: provides tools and libraries for creating marker-based AR applications, 8) Unity IL2CPP support: provides means to improve runtime performance and deployment, 9) Unity standard assets: pre-built assets and scripts accelerate the development of AR applications, 10) Windows 10 SDK: provides tools, libraries, and APIs for developing applications targeting the windows 10 platform, 11) DotNet SDK 5.0.408: provides tools and libraries for developing applications using the .NET framework, 12) HoloLens emulator: allows testing and debugging the AR application without needing to build the application on the HoloLens, 13) VuMark Designer 10.13.03: allows creating custom VuMarks by integrating its scripts into Adobe Illustrator, and 14) Mixed Reality Tool Kit (MRTK): a collection of scripts, components, and prefabs (prefabricated 3D objects) designed to accelerate the development of AR applications.
After installing all required software, features, and plug-ins. The created project utilized Unity Editor 2018.1.9f2 for its combability with HoloLens. The second step includes importing all required resources into the project using the import prompt (Assets > Import Package > Custom Package) by navigating to the MRTK foundation assets directory and importing all items. Similarly, the same steps are performed to import MRTK tools, extensions, examples, Vuforia package. The third step involves configuring the project for an AR application by applying MR project settings, MR scene settings, and UWP Capability. This will create a new Main Camera, representing the user’s head, with the necessary scripts attached to it.
The fourth step involves creating objects that satisfy the application wireframe. this is completed by creating game objects and scripts assigned to them where the game object serves as the front-end of the application and the assigned scripts serve as the back-end for actions and any required feedback analysis. Game object creation is a straightforward process which involves the following main aspects: 1) Creating the object by right-clicking the project Hierarchy and creating a 3D game object or by importing a prefab game object, 2) Identifying object shape from the “Inspector” panel either by using a box and shape collider, 3) Object transformation from the “Inspector” panel that enables applying transformation for the object position, rotation, or scale in any of the three-axis directions, 4) Mesh renderer from the “Inspector” panel that enables assigning material to the game object, 5) Script assigned to the game object which can be applied through a simple drag from the Project Explorer and drop into the assigned object, and 6) Design the VuMark using Adobe Illustrator to generate the VuMark and dragging it into the VuMark game object to be assigned as a material. This enables the creating objects that ultimately will complete the User Interface shown in Figure 11. Afterwards, setting the back end of objects is performed through C# scripts that will be assigned to the objects. Different functions are developed to provide necessary control of the environment (see Appendix A).
 
Figure 11: AR Application Scene Components
The final stage is building and deployment of the AR application into the HoloLens. To build the application, it is required to configure the Unity build settings (File > Build Settings) into the following configurations: 1) Selecting the scene, 2) Setting the application platform to UWP, 3) Setting “Target Device” to HoloLens, 4) Setting “Build Type” to D3D, 5) Setting “Build Configuration” to Release, and 6) Checking “Unity C# Projects”. This produces the application solution, by which the following steps are performed: 1) Opening the *.sln file using Visual Studio, 2) Connecting the HoloLens to the machine, 3) Configuring Visual Studio for the application deployment into Configuration: “Release” and Platform: “Active x86”, and 5) Initiating the application deployment by selecting “Debug > Start Without Debugging”. After deployment and the application is installed in the HoloLens, the application can be initiated and used by the user. The visualized point cloud enables the user to determine their coordinates based on the floating coordinates text that interact with the user's gaze location and head movement around the area (see Figure 12).

 
Figure 12: Visualizing Point Cloud Coordinates Data