In tackling the pressing concern of deteriorating infrastructure, the integration of element-level con- dition ratings and the digitization of the national bridge inventory stands out as a crucial response, not only refining the quality of inspection data but also unlocking the potential for machine learning and statistical methods to forecast future infrastructure conditions. AR emerges as a beacon of promise, touted as a tool capable of elevating the quality, precision, and efficiency of bridge inspections. AR is a technology that blends computer-generated information with the user’s real-world environment in real-time. Unlike virtual reality, which creates a completely artificial environment, augmented reality enhances the existing surroundings by overlaying digital information, such as images, videos, or 3D models, onto the physical world. AR is typically experienced through devices like smartphones, tablets, smart glasses, or AR headsets. It has applications in various fields, including gaming, education, healthcare, manufacturing, and navigation. For example, AR can be used to display additional information about a product when a user points their smartphone camera at it, or it can provide real-time navigation cues overlaid on the street view seen through smart glasses. However, the technology, despite its potential, remains in its infancy within this domain, grappling with various implementation challenges in structural and infrastructure health monitoring. The purpose of this review is to provide an overview of both the state of the practice and state of the art applications of AR and related technologies in the bridge inspection context. Based on this overview, key research gaps are identified and discussed. This study ventures into the exploration of Augmented Reality (AR) and immerses itself in the latest developments and the transformative influence it could wield on the digital aspects of bridge inspection. It addresses many aspects of bridge structural inspec- tion aided by augmented reality, such as (i) process of AR bridge inspection (ii) present stage and future implementation in the industry (iii) software (iv) challenges and (v) future of AR. Keywords: Structural Health Monitoring, Bridge Inspection, Augmented Reality 1 Introduction and Background AR stands for Augmented Reality. It is a technology that blends computer-generated information with the user’s real world environment in real time. Unlike virtual reality, which creates a completely artificial 1 Baishya Journal environment, augmented reality enhances the existing surroundings by overlaying digital information, such as images, videos, or 3D models, onto the physical world. AR is typically experienced through devices like smartphones, tablets, smart glasses, or AR headsets. It has applications in various fields, including gaming, education, healthcare, manufacturing, and navigation. For example, AR can be used to display additional information about a product when a user points their smartphone camera at it, or it can provide real-time navigation cues overlaid on the street view seen through smart glasses. Using Augmented Reality (AR) in structural engineering or specifically in bridge infrastructure inspection involves leveraging AR technologies to enhance various aspects of the field, including design, construction, inspection, and maintenance. Here are several ways AR can be applied in structural engineering. Venturing into the exploration of Augmented Reality (AR), this literature review immerses itself in the latest devel- opments and the transformative influence it could wield on the digital aspects of bridge inspection. AR emerges as a beacon of promise, touted as a tool capable of elevating the quality, precision, and efficiency of bridge inspections. However, technology, despite its potential, remains in its infancy within this domain, grappling with various implementation challenges. The central objective of this review is to furnish a sweep- ing overview of prevailing industry practices and cutting-edge applications of AR and related technologies within the realm of bridge inspection. Within this landscape, the review identifies and deliberates on key research gaps. While some of these gaps revolve around technical intricacies, a noticeable void is discerned in studies that delve into how AR systems impact field performance. At a broader level, the observation is that digitization and automation initiatives often evolve in isolation, bereft of comprehensive testing within the inspector-computer system. Embarking on the review with a succinct justification for the role of digitization and AR in shaping the future of bridge inspection, the narrative progresses to dissect the state of industry practices across various state departments of transportation (DOTs). It covers both general digitization technologies and present and future AR models. Expanding the horizon, the exploration delves into the broader research literature to unravel the state-of-the-art applications of AR. Current biennial inspections involve on-site assessors who document conditions manually, a practice that is later transferred to digital formats, creating room for errors [1], [6] and [8]. This manual entry contributes to data reliability issues, as evidenced by bridge ratings that have unaccountably improved without any record of maintenance [7]. The review also subjects to the process of AR Bridge inspection, present industry stage and implementation, software used, challenges of machine learning and computer vision in the field of bridge health monitoring and future of AR. AR-based bridge inspection and emerging trends within the specialized realm of machine learning. Bringing the review to a close, there is a deliberate emphasis on the changes or improvement for usefulness and efficiency of the technology. Each segment of the review endeavors to cast light on the present stage, future stage and efficiency of AR in bridge inspections. In tackling the pressing concern of deteriorating infrastructure, the integration of element level condition ratings and the digitization of the national bridge inventory stands out as a crucial response, not only refining the quality of inspection data but also unlocking the potential for machine learning and statistical methods to forecast future infrastructure conditions. Nevertheless, the advancements in these realms have introduced a layer of intricacy into the inspection process, creating a noticeable rift between traditional manual note- taking methods and the escalating demands of digital data integration. This underscores an imminent need for a seamless shift toward in-field digital data collection during bridge inspections. AR emerges as a beacon of promise, touted as a tool capable of elevating the quality, precision, and efficiency of bridge inspections. However, technology, despite its potential, remains in its infancy within this domain, grappling with various implementation challenges. Page 2 Baishya Journal 2 Literature Review R aided bridge inspection is a detailed approach that provides a nuanced understanding of bridge deterio- ration and assists in prioritizing maintenance, it also increases the time and effort required from inspectors and can result in less consistent documentation. Current biennial inspections involve on-site assessors who document conditions manually, a practice that is later transferred to digital formats, creating room for errors. This manual entry contributes to data reliability issues, as evidenced by bridge ratings that have unaccount- ably improved without any record of maintenance. These challenges underscore the growing necessity for digital transformation in bridge inspections. One key area for improvement in current bridge inspection practice is the digitization field data. Improving this aspect would not only streamline the inspection pro- cess but also alleviate the quality control issues commonly encountered during the translation of analog field notes into digital databases. The whole process is divided into three parts. The first part is data collection. This is the step in which the data is collected for creating the database for the AR interface. While viewing a bridge, inspectors could be presented relevant information, such as information from previous inspection reports, overlaid over the real scene they are viewing. Gathering National Bridge Inventory Data using the Structural Inventory and Appraisal Sheet. The second step is prototype. A prototype was developed to incorporate the different interface concepts and illustrate how they would work within a virtual reality (VR) environment. The goal was to explore which AR interfaces would be most appropriate for use by bridge inspectors. The prototype allowed for the initial evaluation of the effectiveness of using AR-based interfaces without dealing with the problems of implementing AR technology in outdoor settings. Computer vision (CV) techniques are also an important technique in the field of bridge inspection. CV methods have been developed and modified for tasks such as material segregation, component and element identification, and defect detection. CV machine learning models have been used to detect defects and contextualize them with associated structural details. Fig. 1: Example of Using Computer Vision to Extract Patterns On a Wall [3] Page 3 Baishya Journal Some of the frameworks used in AR are Unity: The authors used the Unity game engine as a cross platform software development package for creating AR applications. Unity allows developers to design scenes and create Game Objects that contain 3D holograms, buttons, input fields, and other interactive elements. C# scripting is used in Unity to write codes, which are then built into a solution file. Visual Studio is used to deploy the solution file to the HMD. ii. Microsoft’s Mixed Reality Tool Kit (MRTK): The authors also utilized MRTK, which is a toolkit provided by Microsoft for developing AR applications. MRTK offers inbuilt attributes like spatial mapping, voice, and gesture recognition, which can be integrated into Unity applications. It provides basic building blocks for Unity applications and supports various devices, including HoloLens. MRTK simplifies the development workflow by providing components for input systems, user interface controls, gaze and gesture control, and spatial awareness. [2] Al-Sabbag Z. et. al. proposes a novel end-to-end system called Human-Machine Collaborative Inspection (HMCI) to enable collaboration between inspectors with Mixed Reality (MR) headsets and a robotic data collection platform (robot) for structural inspections. [3] Park J. et. al. proposed technique utilizing surface textures captured in images, the technique can estimate the scale of the corresponding images. This enables quantitative evaluation and measurement of inspection regions without the need for additional sensor data. The technique can be integrated into existing vision-based inspection algorithms, allowing for a fully automated procedure for damage detection, localization, and quantification using only images. The third step is evaluation. The evaluation was conducted using a heuristic evaluation method, where several evaluators were presented with the interface designs and asked to evaluate them. The evaluators are generally civil engineers or graduate students. The evaluation focused on parameters such as usability, intuitiveness, and effectiveness, and feedback was collected through a questionnaire. One of the glimpses of the evaluation process can be seen in [4] Lorraine Lin et.al. paper where they performed this procedure with twenty participants (12 male, 8 females; ages between 18 and 40) who volunteered for the study. Participants consisted mainly of undergraduate and graduate students recruited from Clemson University. Nineteen of our participants were right-handed and one was left-handed. One participant wore our size 2 gloves, four wore size 3, three wore size 4, nine wore size 5, and three wore size 6. We obtained informed consent from all participants before the study following the guidelines set by our Institutional Review Board. Participants received a 5 dollar voucher for their time. Fig. 2: : A Rough Flowchart of The Whole Evaluation Process [4] 3 Practice Review There was also a session with Dr. Rodrigo Sarlo from The Charles Edward Via Jr. Department of Civil & Environmental Engineering of Virginia Tech, who gave a better insight into the present and future of AR technology in bridge inspection in terms of research and industry. He mentioned that AR is in its starting stage, and it is aimed to be a tool for visualizing in 3D on site. It is also mainly aimed to reduce the Page 4 Baishya Journal number of inspectors, compare the defect size by comparing it to historical data. He also mentioned that the technology is still in the stages of data acquisition. It is still distinguishing the data as correct or wrong and not working on prediction. Presently drones are most useful for inspection of structures but according to him AR once developed will be a better option for crack detection, predicting lifespan of a structure and predicting the requirement of repairing. Unity, Microsoft’s Mixed Reality Tool Kit (MRTK) and computer vision is used to detect cracks. Packages (YOLO and COCO) are some of the software and CV packages used for detecting cracks. These software and models are mainly aimed at working on semantic segmentation (multiple pixel photo) and object detection mainly for qualifying cracks. In terms of inventory, he mentioned that making an inventory gives us an idea of the present state of infrastructure. Inspect X and Headlight are some of the software used for creating the inventories. Parallel to data acquisition researchers are also working on device limitations and on locating itself when taken to a new area. He also mentioned that the in terms of prototyping it is still in the evaluation stage. There are many prototypes that researchers have come up with but could not reach the minimum viable product of the prototype. One of the issues that was mentioned is that AR has potential, but hardware is not ready, like heating processing power etc. Also mentioned that it becomes evident that there is a wealth of technologies for digitizing the bridge inspection workflow, ranging from highly practical data entry systems to sophisticated machine learning models. In this landscape, augmented reality stands out as a platform for centralizing this wide array of digital tools while also capturing spatial context and integrating easily into existing inspection workflow. AR implementation in bridge inspection is still in infancy, with the most recent examples related to either visualization or virtual measurements. These are far from realizing the full potential associated with creating a complete AR workflow. Aca- demic research has come closer to this goal by incorporating computer vision (CV) into AR as a means to streamline the documentation process, yet to our knowledge all research has been performed in structured lab settings. There is virtually no research on real bridges with real inspectors. Conclusions In conclusion, bridge structural inspection aided by augmented reality (AR) interfaces has the potential to revolutionize the field by enhancing efficiency, safety, and accuracy. In terms of enhancing visualization, AR mitigates this issue by enabling the real-time operation of machine learning algorithms in the field, instantly displaying the results as the inspector captures images. 10 David DL Mascarenas et al. built an AR application for the collection of linear and area measurements where the user would select points in 3D space to trace lines or polygons on a surface. They found that the area measures had less than 2 ±2% error while the linear measurements had around 6% error when compared to ground truth. In terms of efficiency, AR can streamline the inspection process by providing inspectors with digital overlays that highlight areas of concern, reducing the time required for manual inspections. Most studies focus on machine learning (ML) or computer vision (CV) measurement tools, or a combination of data visualization and localization. AR systems introduce an element of real-time automation that traditional methods lack. Typically, machine learning techniques in bridge inspection analyze a dataset of images to identify damaged regions. AR provides real-time, 3D visual overlays of bridge structures, enabling inspectors to see structural elements and potential issues more clearly. This can lead to more accurate assessments. The significant technical challenges to be overcome when it comes to implementing such systems in the field, primarily in relation to 1) outdoor localization and 2) reduced CV performance in unstructured environments. The main challenge, however, is human centric: So far, no work has considered the role Page 5 Baishya Journal of the inspector in the workflow. This applies not only to the interaction with automated processes like CV but also to techniques for leveraging inspectors’ domain knowledge to mitigate some of the technical challenges. The challenges face in AR aided bridge inspection is that a major limitation of studies like these is that VR is not a perfect analog to AR, and VR sidesteps many of the real world difficulties of AR such as glare, limited field of view, limited processing, and heating handling capability [6]. Most research into AR for measurement/identification tasks focuses on some form of computer vision or machine learning. The challenge to these types of studies is the lack of processing power and unique operating systems of these AR devices. Limitations of this study are the small sample size (2 inspectors) for their experiment testing of the interface, a lack of metrics associated with said experiment beyond a noted 50% reduction in task completion time. Course localization, often referred to as initialization of the system, is usually a core tracking step that occurs when a device needs to locate itself in a new area or tracking has been previously lost. Recommendation For the recommendation, the developers or researchers should work on VR headset’s real-world difficulties of supporting AR models such as glare, limited field of view, limited processing, and heating handling capability in outside use. They should work on the processing units of the VR headsets as the processors are not ready to take the load of running the AR models. Another recommendation would be to increase the number of inspectors for testing the device. Lastly, they should work on offline tracking which is defined as the ability of the tracking device to not require constant internet network connection to function and continue localization and tracking. Acknowledgments I would first like to express my gratitude to Dr. Sunil K Sinha and Dr. Rodrigo Sarlo, for their support and guidance throughout my time as a master’s student at Virginia Tech. I would also like to thank them for inspiring me with different research ideas and guiding me throughout my course project. Finally, I would like to thank my parents, family, and friends, for their continued support and encouragement. I would not have been able to do this without them.